=== RUN TestOdhOperator/components/group_2/kueue/Validate_component_removed_to_unmanaged_transition helper_test.go:150: Setting Kueue component (default-kueue) to Removed mode to start with a clean state. helper_test.go:150: Cleaning up any existing Kueue test resources. helper_test.go:150: Cleaning up Kueue cluster-scoped resources helper_test.go:150: Will attempt to delete 2 Kueue resources helper_test.go:150: Attempting to delete ClusterQueue /default helper_test.go:150: Successfully processed deletion of ClusterQueue /default helper_test.go:150: Attempting to delete Kueue /cluster helper_test.go:150: Successfully processed deletion of Kueue /cluster helper_test.go:150: Creating test namespace (test-kueue-managed-d989jnk35hkc74rp4pag) with Kueue management annotation. helper_test.go:150: Creating Kueue ConfigMap (kueue-manager-config) to verify default Kueue resource is created correctly. helper_test.go:150: Creating Kueue ConfigMap (kueue-manager-config) in namespace (redhat-ods-applications). helper_test.go:150: Updating the management state of the component in the DataScienceCluster to Unmanaged (expecting NotReady initially due to missing operator). helper_test.go:150: Ensuring LWS Operator is installed (namespace=openshift-lws-operator, name=leader-worker-set). helper_test.go:150: Ensuring LeaderWorkerSetOperator CR exists (namespace=openshift-lws-operator, name=cluster). helper_test.go:150: Ensuring OCP Kueue Operator is installed (namespace=openshift-kueue-operator, name=kueue-operator). helper_test.go:150: Verifying the component in the DataScienceCluster transitions to Ready state. helper_test.go:150: Verifying Kueue configuration (cluster) is created with all expected frameworks. testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_2/kueue/Validate_component_removed_to_unmanaged_transition (31.14s) === RUN TestOdhOperator/components/group_2/kueue === PAUSE TestOdhOperator/components/group_2/kueue === CONT TestOdhOperator/components/group_2/kueue test_context_test.go:223: Timed out after 10.000s. The function passed to Eventually failed at /workspace/tests/e2e/resource_applier_test.go:114 with: Expected resource 'cluster' of kind 'Kueue' to match condition {(["BatchJob", "Deployment", "JobSet", "LeaderWorkerSet", "MPIJob", "PaddleJob", "Pod", "PyTorchJob", "RayCluster", "RayJob", "StatefulSet", "TFJob", "TrainJob", "XGBoostJob"] - .spec.config.integrations.frameworks) | length == 0 []}. Expected <string>: map[apiVersion:kueue.openshift.io/v1 kind:Kueue metadata:map[annotations:map[platform.opendatahub.io/instance.generation:1 platform.opendatahub.io/instance.name:default-kueue platform.opendatahub.io/instance.uid:11c658a5-8ef3-4488-8a91-f2a180da55db platform.opendatahub.io/type:OpenShift AI Self-Managed platform.opendatahub.io/version:3.5.0-ea.2] creationTimestamp:2026-07-10T06:21:23Z deletionGracePeriodSeconds:0 deletionTimestamp:2026-07-10T07:06:17Z finalizers:[kueue.openshift.io/finalizer] generation:2 labels:map[platform.opendatahub.io/part-of:kueue] managedFields:[map[apiVersion:kueue.openshift.io/v1 fieldsType:FieldsV1 fieldsV1:map[f:status:map[f:conditions:map[k:{"type":"Available"}:map[.:map[] f:lastTransitionTime:map[] f:message:map[] f:reason:map[] f:status:map[] f:type:map[]] k:{"type":"CertManagerAvailable"}:map[.:map[] f:lastTransitionTime:map[] f:message:map[] f:reason:map[] f:status:map[] f:type:map[]] k:{"type":"Degraded"}:map[.:map[] f:lastTransitionTime:map[] f:message:map[] f:reason:map[] f:status:map[] f:type:map[]] k:{"type":"Progressing"}:map[.:map[] f:lastTransitionTime:map[] f:message:map[] f:reason:map[] f:status:map[] f:type:map[]]] f:readyReplicas:map[]]] manager:kueue-operator operation:Apply subresource:status time:2026-07-10T06:22:09Z] map[apiVersion:kueue.openshift.io/v1 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:annotations:map[.:map[] f:platform.opendatahub.io/instance.generation:map[] f:platform.opendatahub.io/instance.name:map[] f:platform.opendatahub.io/instance.uid:map[] f:platform.opendatahub.io/type:map[] f:platform.opendatahub.io/version:map[]] f:labels:map[.:map[] f:platform.opendatahub.io/part-of:map[]]] f:spec:map[.:map[] f:config:map[.:map[] f:integrations:map[.:map[] f:frameworks:map[.:map[] v:"Deployment":map[] v:"Pod":map[] v:"PyTorchJob":map[] v:"RayCluster":map[] v:"RayJob":map[] v:"StatefulSet":map[] v:"TrainJob":map[]]]] f:logLevel:map[] f:managementState:map[] f:operatorLogLevel:map[]]] manager:manager operation:Update time:2026-07-10T06:21:23Z] map[apiVersion:kueue.openshift.io/v1 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:finalizers:map[.:map[] v:"kueue.openshift.io/finalizer":map[]]]] manager:kueue-operator operation:Update time:2026-07-10T07:06:17Z]] name:cluster resourceVersion:23089352 uid:3324c9a4-3a7b-494b-9fb4-07bad0c4c8a4] spec:map[config:map[integrations:map[frameworks:[Deployment Pod PyTorchJob RayCluster RayJob StatefulSet TrainJob]]] logLevel:Normal managementState:Managed operatorLogLevel:Normal] status:map[conditions:[map[lastTransitionTime:2026-07-10T06:22:04Z message:All 2 replicas are ready reason:AllReplicasReady status:True type:Available] map[lastTransitionTime:2026-07-10T06:22:04Z message:Deployment is up to date reason:AsExpected status:False type:Progressing] map[lastTransitionTime:2026-07-10T06:22:04Z message: reason:AsExpected status:False type:Degraded] map[lastTransitionTime:2026-07-10T06:21:48Z message:cert-manager is installed reason:CertManagerInstalled status:True type:CertManagerAvailable]] readyReplicas:2]] to match expression <string>: (["BatchJob", "Deployment", "JobSet", "LeaderWorkerSet", "MPIJob", "PaddleJob", "Pod", "PyTorchJob", "RayCluster", "RayJob", "StatefulSet", "TFJob", "TrainJob", "XGBoostJob"] - .spec.config.integrations.frameworks) | length == 0 test_context_test.go:1106: cleanup: best-effort delete failed for /, Kind= redhat-ods-applications/kueue-manager-config: Object 'Kind' is missing in 'unstructured object has no kind' --- FAIL: TestOdhOperator/components/group_2/kueue (31.36s) === RUN TestOdhOperator/components/group_2 --- FAIL: TestOdhOperator/components/group_2 (0.00s) === RUN TestOdhOperator/components 2026/07/10 07:09:49 TEST FAILURE DETECTED: group 2 2026/07/10 07:09:49 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:group 2) === 2026/07/10 07:09:49 === CLUSTER STATE === 2026/07/10 07:09:49 Node: ip-10-0-1-23.us-east-2.compute.internal 2026/07/10 07:09:49 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/10 07:09:49 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/10 07:09:49 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/10 07:09:49 Condition Ready: True - kubelet is posting ready status 2026/07/10 07:09:49 Resources: 15500m CPU, 63182900Ki memory allocatable (of 16000m CPU, 64333876Ki memory capacity) 2026/07/10 07:09:49 Node: ip-10-0-1-45.us-east-2.compute.internal 2026/07/10 07:09:49 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/10 07:09:49 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/10 07:09:49 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/10 07:09:49 Condition Ready: True - kubelet is posting ready status 2026/07/10 07:09:49 Resources: 15500m CPU, 63871012Ki memory allocatable (of 16000m CPU, 65021988Ki memory capacity) 2026/07/10 07:09:49 Node: ip-10-0-1-76.us-east-2.compute.internal 2026/07/10 07:09:49 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/10 07:09:49 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/10 07:09:49 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/10 07:09:49 Condition Ready: True - kubelet is posting ready status 2026/07/10 07:09:49 Resources: 15500m CPU, 63871016Ki memory allocatable (of 16000m CPU, 65021992Ki memory capacity) 2026/07/10 07:09:49 === DEPLOYMENTS === 2026/07/10 07:09:49 Namespace: redhat-ods-applications 2026/07/10 07:09:49 Deployment maas-api: 1/1 (READY) 2026/07/10 07:09:49 Deployment maas-controller: 1/1 (READY) 2026/07/10 07:09:49 Deployment mlflow-operator-controller-manager: 1/1 (READY) 2026/07/10 07:09:49 Deployment trustyai-service-operator-controller-manager: 1/1 (READY) 2026/07/10 07:09:49 Namespace: kube-system 2026/07/10 07:09:49 === PODS === 2026/07/10 07:09:49 Namespace kube-system: no problematic pods found 2026/07/10 07:09:49 Namespace redhat-ods-applications: no problematic pods found 2026/07/10 07:09:49 === OPERATOR STATUS === 2026/07/10 07:09:49 Operator deployment: 3/3 ready 2026/07/10 07:09:49 === DSCI STATUS === 2026/07/10 07:09:49 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/10 07:09:49 === DSC STATUS === 2026/07/10 07:09:49 Failed to collect DSC data: default-dsc unhealthy conditions: ModelsAsServiceReady=False 2026/07/10 07:09:49 === RECENT EVENTS === 2026/07/10 07:09:49 07:09:41 INFO Pod/model-registry-operator-controller-manager-74db7cdccd-s6h64: Killing - Stopping container manager 2026/07/10 07:09:49 07:09:41 INFO Pod/workload-variant-autoscaler-controller-manager-7f5b9cd77b-fdg2v: Killing - Stopping container manager 2026/07/10 07:09:49 07:09:41 INFO Pod/odh-model-controller-65fdb9c5c-sllgn: Killing - Stopping container manager 2026/07/10 07:09:49 07:09:41 INFO Pod/model-serving-api-5679d6b4f-w5c7j: Killing - Stopping container server 2026/07/10 07:09:49 07:09:40 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/10 07:09:49 07:09:40 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/10 07:09:49 07:09:40 INFO Pod/kserve-controller-manager-dd9f6cf9-nnb86: Killing - Stopping container manager 2026/07/10 07:09:49 07:09:40 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-84r5z: Killing - Stopping container manager 2026/07/10 07:09:49 07:09:39 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/10 07:09:49 07:09:27 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-65fdb9c5c-sllgn_5b69ea1d-6736-4ee2-8754-b6c87b80f375 became leader 2026/07/10 07:09:49 07:09:24 INFO Pod/workload-variant-autoscaler-controller-manager-7f5b9cd77b-fdg2v: Created - Created container: manager 2026/07/10 07:09:49 07:09:24 INFO Pod/workload-variant-autoscaler-controller-manager-7f5b9cd77b-fdg2v: AddedInterface - Add eth0 [10.129.1.153/23] from ovn-kubernetes 2026/07/10 07:09:49 07:09:24 INFO Pod/workload-variant-autoscaler-controller-manager-7f5b9cd77b-fdg2v: Pulled - Container image "quay.io/rhoai/odh-workload-variant-autoscaler-controller-rhel9@sha256:042a83d343bb9733c322eb22eeb445d8991365a2184ace45556acb7b110b6a83" already present on machine 2026/07/10 07:09:49 07:09:24 INFO Pod/workload-variant-autoscaler-controller-manager-7f5b9cd77b-fdg2v: Started - Started container manager 2026/07/10 07:09:49 07:09:24 INFO Lease/72dd1cf1.llm-d.ai: LeaderElection - workload-variant-autoscaler-controller-manager-7f5b9cd77b-fdg2v_c297d0da-c8cc-4d89-b6e3-39237cfdaf2f became leader 2026/07/10 07:09:49 07:09:23 INFO Pod/workload-variant-autoscaler-controller-manager-7f5b9cd77b-fdg2v: Scheduled - Successfully assigned redhat-ods-applications/workload-variant-autoscaler-controller-manager-7f5b9cd77b-fdg2v to ip-10-0-1-45.us-east-2.compute.internal 2026/07/10 07:09:49 07:09:23 INFO ReplicaSet/workload-variant-autoscaler-controller-manager-7f5b9cd77b: SuccessfulCreate - Created pod: workload-variant-autoscaler-controller-manager-7f5b9cd77b-fdg2v 2026/07/10 07:09:49 07:09:23 INFO Pod/workload-variant-autoscaler-controller-manager-7f5b9cd77b-7g8rg: Killing - Stopping container manager 2026/07/10 07:09:49 07:09:23 INFO Pod/workload-variant-autoscaler-controller-manager-7f5b9cd77b-rwtdb: Killing - Stopping container manager 2026/07/10 07:09:49 07:09:23 INFO Deployment/workload-variant-autoscaler-controller-manager: ScalingReplicaSet - Scaled up replica set workload-variant-autoscaler-controller-manager-7f5b9cd77b from 0 to 1 2026/07/10 07:09:49 07:09:09 INFO Pod/odh-model-controller-65fdb9c5c-sllgn: Created - Created container: manager 2026/07/10 07:09:49 07:09:09 INFO Pod/odh-model-controller-65fdb9c5c-sllgn: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/10 07:09:49 07:09:09 INFO Pod/odh-model-controller-65fdb9c5c-sllgn: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 534ms (534ms including waiting). Image size: 238959091 bytes. 2026/07/10 07:09:49 07:09:09 INFO Pod/odh-model-controller-65fdb9c5c-sllgn: Started - Started container manager 2026/07/10 07:09:49 07:09:09 INFO Pod/odh-model-controller-65fdb9c5c-sllgn: AddedInterface - Add eth0 [10.129.1.152/23] from ovn-kubernetes 2026/07/10 07:09:49 07:09:08 INFO Pod/odh-model-controller-65fdb9c5c-sllgn: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-65fdb9c5c-sllgn to ip-10-0-1-45.us-east-2.compute.internal 2026/07/10 07:09:49 07:09:08 INFO Pod/odh-model-controller-65fdb9c5c-7vxg4: Killing - Stopping container manager 2026/07/10 07:09:49 07:09:08 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-65fdb9c5c from 0 to 1 2026/07/10 07:09:49 07:09:08 INFO Pod/odh-model-controller-65fdb9c5c-zr5sc: Killing - Stopping container manager 2026/07/10 07:09:49 07:09:08 INFO ReplicaSet/odh-model-controller-65fdb9c5c: SuccessfulCreate - Created pod: odh-model-controller-65fdb9c5c-sllgn 2026/07/10 07:09:49 07:08:54 INFO Pod/model-serving-api-5679d6b4f-w5c7j: Started - Started container server 2026/07/10 07:09:49 07:08:54 INFO Pod/model-serving-api-5679d6b4f-w5c7j: AddedInterface - Add eth0 [10.129.1.151/23] from ovn-kubernetes 2026/07/10 07:09:49 07:08:54 INFO Pod/model-serving-api-5679d6b4f-w5c7j: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/10 07:09:49 07:08:54 INFO Pod/model-serving-api-5679d6b4f-w5c7j: Created - Created container: server 2026/07/10 07:09:49 07:08:53 INFO Pod/model-serving-api-5679d6b4f-w5c7j: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-5679d6b4f-w5c7j to ip-10-0-1-45.us-east-2.compute.internal 2026/07/10 07:09:49 07:08:53 INFO Pod/model-serving-api-5679d6b4f-j9krp: Killing - Stopping container server 2026/07/10 07:09:49 07:08:53 INFO Pod/model-serving-api-5679d6b4f-shz2x: Killing - Stopping container server 2026/07/10 07:09:49 07:08:53 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-5679d6b4f from 0 to 1 2026/07/10 07:09:49 07:08:53 INFO ReplicaSet/model-serving-api-5679d6b4f: SuccessfulCreate - Created pod: model-serving-api-5679d6b4f-w5c7j 2026/07/10 07:09:49 07:07:53 INFO Pod/workload-variant-autoscaler-controller-manager-7f5b9cd77b-7g8rg: Created - Created container: manager 2026/07/10 07:09:49 07:07:53 INFO Pod/workload-variant-autoscaler-controller-manager-7f5b9cd77b-7g8rg: Pulled - Container image "quay.io/rhoai/odh-workload-variant-autoscaler-controller-rhel9@sha256:042a83d343bb9733c322eb22eeb445d8991365a2184ace45556acb7b110b6a83" already present on machine 2026/07/10 07:09:49 07:07:53 INFO Pod/workload-variant-autoscaler-controller-manager-7f5b9cd77b-7g8rg: Started - Started container manager 2026/07/10 07:09:49 07:07:53 INFO Pod/workload-variant-autoscaler-controller-manager-7f5b9cd77b-7g8rg: AddedInterface - Add eth0 [10.129.1.150/23] from ovn-kubernetes 2026/07/10 07:09:49 07:07:52 INFO Pod/workload-variant-autoscaler-controller-manager-7f5b9cd77b-7g8rg: Scheduled - Successfully assigned redhat-ods-applications/workload-variant-autoscaler-controller-manager-7f5b9cd77b-7g8rg to ip-10-0-1-45.us-east-2.compute.internal 2026/07/10 07:09:49 07:07:52 INFO ReplicaSet/workload-variant-autoscaler-controller-manager-7f5b9cd77b: SuccessfulCreate - Created pod: workload-variant-autoscaler-controller-manager-7f5b9cd77b-7g8rg 2026/07/10 07:09:49 07:07:52 INFO Deployment/workload-variant-autoscaler-controller-manager: ScalingReplicaSet - Scaled up replica set workload-variant-autoscaler-controller-manager-7f5b9cd77b from 1 to 2 2026/07/10 07:09:49 07:07:34 INFO Pod/odh-model-controller-65fdb9c5c-7vxg4: Started - Started container manager 2026/07/10 07:09:49 07:07:33 INFO Pod/odh-model-controller-65fdb9c5c-7vxg4: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/10 07:09:49 07:07:33 INFO Pod/odh-model-controller-65fdb9c5c-7vxg4: AddedInterface - Add eth0 [10.129.1.149/23] from ovn-kubernetes 2026/07/10 07:09:49 07:07:33 INFO Pod/odh-model-controller-65fdb9c5c-7vxg4: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 597ms (597ms including waiting). Image size: 238959091 bytes. 2026/07/10 07:09:49 07:07:33 INFO Pod/odh-model-controller-65fdb9c5c-7vxg4: Created - Created container: manager 2026/07/10 07:09:49 07:07:32 INFO Pod/odh-model-controller-65fdb9c5c-7vxg4: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-65fdb9c5c-7vxg4 to ip-10-0-1-45.us-east-2.compute.internal 2026/07/10 07:09:49 07:07:32 INFO ReplicaSet/odh-model-controller-65fdb9c5c: SuccessfulCreate - Created pod: odh-model-controller-65fdb9c5c-7vxg4 2026/07/10 07:09:49 07:07:32 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-65fdb9c5c from 1 to 2 2026/07/10 07:09:49 07:07:13 INFO Pod/model-serving-api-5679d6b4f-j9krp: Started - Started container server 2026/07/10 07:09:49 07:07:13 INFO Pod/model-serving-api-5679d6b4f-j9krp: Created - Created container: server 2026/07/10 07:09:49 07:07:13 INFO Pod/model-serving-api-5679d6b4f-j9krp: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/10 07:09:49 07:07:13 INFO Pod/model-serving-api-5679d6b4f-j9krp: AddedInterface - Add eth0 [10.129.1.148/23] from ovn-kubernetes 2026/07/10 07:09:49 07:07:12 INFO Pod/model-serving-api-5679d6b4f-j9krp: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-5679d6b4f-j9krp to ip-10-0-1-45.us-east-2.compute.internal 2026/07/10 07:09:49 07:07:12 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-5679d6b4f from 1 to 2 2026/07/10 07:09:49 07:07:12 INFO ReplicaSet/model-serving-api-5679d6b4f: SuccessfulCreate - Created pod: model-serving-api-5679d6b4f-j9krp 2026/07/10 07:09:49 07:07:02 INFO Pod/maas-api-5b7784b768-lhj94: Started - Started container maas-api 2026/07/10 07:09:49 07:07:02 INFO Pod/maas-api-5b7784b768-lhj94: Created - Created container: maas-api 2026/07/10 07:09:49 07:07:02 INFO Pod/maas-api-5b7784b768-lhj94: Pulled - Successfully pulled image "quay.io/rhoai/odh-maas-api-rhel9@sha256:9c87c8243770b9ebbff6ccab0dd663ccb402488cf5ee11f491ec90cf3b4686f6" in 543ms (543ms including waiting). Image size: 175018396 bytes. 2026/07/10 07:09:49 07:07:01 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/10 07:09:49 07:07:01 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/10 07:09:49 07:07:01 INFO Pod/maas-api-5b7784b768-lhj94: Pulling - Pulling image "quay.io/rhoai/odh-maas-api-rhel9@sha256:9c87c8243770b9ebbff6ccab0dd663ccb402488cf5ee11f491ec90cf3b4686f6" 2026/07/10 07:09:49 07:07:01 INFO Pod/maas-api-5b7784b768-lhj94: AddedInterface - Add eth0 [10.130.0.19/23] from ovn-kubernetes 2026/07/10 07:09:49 07:07:00 INFO Pod/maas-api-5b7784b768-lhj94: Scheduled - Successfully assigned redhat-ods-applications/maas-api-5b7784b768-lhj94 to ip-10-0-1-23.us-east-2.compute.internal 2026/07/10 07:09:49 07:07:00 WARN ReplicaSet/maas-api-5b7784b768: FailedCreate - Error creating: pods "maas-api-5b7784b768-" is forbidden: error looking up service account redhat-ods-applications/maas-api: serviceaccount "maas-api" not found 2026/07/10 07:09:49 07:07:00 INFO Deployment/maas-api: ScalingReplicaSet - Scaled up replica set maas-api-5b7784b768 from 0 to 1 2026/07/10 07:09:49 07:07:00 INFO ReplicaSet/maas-api-5b7784b768: SuccessfulCreate - Created pod: maas-api-5b7784b768-lhj94 2026/07/10 07:09:49 07:06:52 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/10 07:09:49 07:06:52 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/10 07:09:49 07:06:45 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-7b46b9fbbb-84r5z_215b5063-c274-424f-ac83-d834a46d3c65 became leader 2026/07/10 07:09:49 07:06:44 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-dd9f6cf9-nnb86_fc3241d5-63a0-4690-b6dd-ed2cbe30215e became leader 2026/07/10 07:09:49 07:06:32 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/10 07:09:49 07:06:32 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/10 07:09:49 07:06:32 INFO Lease/72dd1cf1.llm-d.ai: LeaderElection - workload-variant-autoscaler-controller-manager-7f5b9cd77b-rwtdb_796e70d8-746a-4383-bdf5-dbb5fc89b0fa became leader 2026/07/10 07:09:49 07:06:31 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/10 07:09:49 07:06:31 WARN ServiceMonitor/workload-variant-autoscaler-controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/workload-variant-autoscaler-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: bearerTokenSecret: [REDACTED] to get secret "workload-variant-autoscaler-controller-manager-token": secrets "workload-variant-autoscaler-controller-manager-token" not found 2026/07/10 07:09:49 07:06:31 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/10 07:09:49 07:06:31 INFO Pod/workload-variant-autoscaler-controller-manager-7f5b9cd77b-rwtdb: Pulled - Container image "quay.io/rhoai/odh-workload-variant-autoscaler-controller-rhel9@sha256:042a83d343bb9733c322eb22eeb445d8991365a2184ace45556acb7b110b6a83" already present on machine 2026/07/10 07:09:49 07:06:31 INFO Pod/workload-variant-autoscaler-controller-manager-7f5b9cd77b-rwtdb: Created - Created container: manager 2026/07/10 07:09:49 07:06:31 INFO Pod/workload-variant-autoscaler-controller-manager-7f5b9cd77b-rwtdb: AddedInterface - Add eth0 [10.128.0.185/23] from ovn-kubernetes 2026/07/10 07:09:49 07:06:31 INFO Pod/workload-variant-autoscaler-controller-manager-7f5b9cd77b-rwtdb: Started - Started container manager 2026/07/10 07:09:49 07:06:30 WARN Pod/workload-variant-autoscaler-controller-manager-7f5b9cd77b-rwtdb: FailedMount - MountVolume.SetUp failed for volume "epp-metrics-token" : secret "workload-variant-autoscaler-epp-metrics-token" not found 2026/07/10 07:09:49 07:06:29 INFO Pod/workload-variant-autoscaler-controller-manager-7f5b9cd77b-rwtdb: Scheduled - Successfully assigned redhat-ods-applications/workload-variant-autoscaler-controller-manager-7f5b9cd77b-rwtdb to ip-10-0-1-76.us-east-2.compute.internal 2026/07/10 07:09:49 07:06:29 WARN ReplicaSet/workload-variant-autoscaler-controller-manager-7f5b9cd77b: FailedCreate - Error creating: pods "workload-variant-autoscaler-controller-manager-7f5b9cd77b-" is forbidden: error looking up service account redhat-ods-applications/workload-variant-autoscaler-controller-manager: serviceaccount "workload-variant-autoscaler-controller-manager" not found 2026/07/10 07:09:49 07:06:29 INFO ReplicaSet/workload-variant-autoscaler-controller-manager-7f5b9cd77b: SuccessfulCreate - Created pod: workload-variant-autoscaler-controller-manager-7f5b9cd77b-rwtdb 2026/07/10 07:09:49 07:06:29 INFO Lease/maas-controller.models-as-a-service.opendatahub.io: LeaderElection - maas-controller-75bbfd978-skltg_8f012a44-9e9c-4013-9e1d-9c721ef6e320 became leader 2026/07/10 07:09:49 07:06:29 INFO Deployment/workload-variant-autoscaler-controller-manager: ScalingReplicaSet - Scaled up replica set workload-variant-autoscaler-controller-manager-7f5b9cd77b from 0 to 1 2026/07/10 07:09:49 07:06:28 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/10 07:09:49 07:06:28 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/10 07:09:49 07:06:28 INFO Lease/85f368d1.opendatahub.io: LeaderElection - model-registry-operator-controller-manager-74db7cdccd-s6h64_b890f386-7f74-4e27-ad62-f198cb581514 became leader 2026/07/10 07:09:49 07:06:26 INFO Pod/kserve-controller-manager-dd9f6cf9-nnb86: Created - Created container: manager 2026/07/10 07:09:49 07:06:26 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-65fdb9c5c-zr5sc_2f2e7a5e-f81f-4494-9277-11c8577fc66b became leader 2026/07/10 07:09:49 07:06:26 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-84r5z: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" 2026/07/10 07:09:49 07:06:26 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-84r5z: Started - Started container manager 2026/07/10 07:09:49 07:06:26 INFO Pod/kserve-controller-manager-dd9f6cf9-nnb86: Started - Started container manager 2026/07/10 07:09:49 07:06:26 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-84r5z: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" in 654ms (654ms including waiting). Image size: 236366599 bytes. 2026/07/10 07:09:49 07:06:26 INFO Pod/kserve-controller-manager-dd9f6cf9-nnb86: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" in 593ms (593ms including waiting). Image size: 238435613 bytes. 2026/07/10 07:09:49 07:06:26 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-84r5z: Created - Created container: manager 2026/07/10 07:09:49 07:06:25 WARN Pod/llmisvc-controller-manager-7b46b9fbbb-84r5z: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "llmisvc-webhook-server-cert" not found 2026/07/10 07:09:49 07:06:25 INFO Pod/kserve-controller-manager-dd9f6cf9-nnb86: AddedInterface - Add eth0 [10.130.0.16/23] from ovn-kubernetes 2026/07/10 07:09:49 07:06:25 INFO Pod/kserve-controller-manager-dd9f6cf9-nnb86: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" 2026/07/10 07:09:49 07:06:25 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-84r5z: AddedInterface - Add eth0 [10.128.0.183/23] from ovn-kubernetes 2026/07/10 07:09:49 07:06:24 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-84r5z: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-7b46b9fbbb-84r5z to ip-10-0-1-76.us-east-2.compute.internal 2026/07/10 07:09:49 07:06:24 INFO Pod/kserve-controller-manager-dd9f6cf9-nnb86: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-dd9f6cf9-nnb86 to ip-10-0-1-23.us-east-2.compute.internal 2026/07/10 07:09:49 07:06:24 INFO ReplicaSet/kserve-controller-manager-dd9f6cf9: SuccessfulCreate - Created pod: kserve-controller-manager-dd9f6cf9-nnb86 2026/07/10 07:09:49 07:06:24 INFO ReplicaSet/llmisvc-controller-manager-7b46b9fbbb: SuccessfulCreate - Created pod: llmisvc-controller-manager-7b46b9fbbb-84r5z 2026/07/10 07:09:49 07:06:24 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-7b46b9fbbb from 0 to 1 2026/07/10 07:09:49 07:06:24 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-dd9f6cf9 from 0 to 1 2026/07/10 07:09:49 07:06:11 INFO Pod/model-serving-api-5679d6b4f-shz2x: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-5679d6b4f-shz2x to ip-10-0-1-23.us-east-2.compute.internal 2026/07/10 07:09:49 07:06:11 INFO ReplicaSet/model-serving-api-5679d6b4f: SuccessfulCreate - Created pod: model-serving-api-5679d6b4f-shz2x 2026/07/10 07:09:49 07:06:11 INFO Pod/maas-controller-75bbfd978-skltg: Started - Started container manager 2026/07/10 07:09:49 07:06:11 INFO Pod/maas-controller-75bbfd978-skltg: Created - Created container: manager 2026/07/10 07:09:49 07:06:11 INFO Pod/maas-controller-75bbfd978-skltg: Pulled - Successfully pulled image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" in 717ms (717ms including waiting). Image size: 212866077 bytes. 2026/07/10 07:09:49 07:06:11 INFO Pod/model-serving-api-5679d6b4f-shz2x: Started - Started container server 2026/07/10 07:09:49 07:06:11 INFO Pod/model-serving-api-5679d6b4f-shz2x: AddedInterface - Add eth0 [10.130.0.14/23] from ovn-kubernetes 2026/07/10 07:09:49 07:06:11 INFO Pod/model-serving-api-5679d6b4f-shz2x: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/10 07:09:49 07:06:11 INFO Pod/model-serving-api-5679d6b4f-shz2x: Created - Created container: server 2026/07/10 07:09:49 07:06:10 INFO Pod/maas-controller-75bbfd978-skltg: AddedInterface - Add eth0 [10.130.0.7/23] from ovn-kubernetes 2026/07/10 07:09:49 07:06:10 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-5679d6b4f from 0 to 1 2026/07/10 07:09:49 07:06:10 INFO Pod/model-registry-operator-controller-manager-74db7cdccd-s6h64: Started - Started container manager 2026/07/10 07:09:49 07:06:10 WARN ReplicaSet/model-serving-api-5679d6b4f: FailedCreate - Error creating: pods "model-serving-api-5679d6b4f-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/10 07:09:49 07:06:10 INFO Pod/model-registry-operator-controller-manager-74db7cdccd-s6h64: Pulled - Container image "quay.io/rhoai/odh-model-registry-operator-rhel9@sha256:ebcec5c43468283ed3cbab50530f939166f8f3e8c9d306e91a8107d05d7fa9d4" already present on machine 2026/07/10 07:09:49 07:06:10 INFO Pod/model-registry-operator-controller-manager-74db7cdccd-s6h64: AddedInterface - Add eth0 [10.130.0.5/23] from ovn-kubernetes 2026/07/10 07:09:49 07:06:10 INFO Pod/maas-controller-75bbfd978-skltg: Pulling - Pulling image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" 2026/07/10 07:09:49 07:06:10 INFO Pod/odh-model-controller-65fdb9c5c-zr5sc: Started - Started container manager 2026/07/10 07:09:49 07:06:10 INFO Pod/odh-model-controller-65fdb9c5c-zr5sc: Created - Created container: manager 2026/07/10 07:09:49 07:06:10 INFO Pod/odh-model-controller-65fdb9c5c-zr5sc: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 596ms (596ms including waiting). Image size: 238959091 bytes. 2026/07/10 07:09:49 07:06:10 INFO Pod/model-registry-operator-controller-manager-74db7cdccd-s6h64: Created - Created container: manager 2026/07/10 07:09:49 07:06:09 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/10 07:09:49 07:06:09 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/10 07:09:49 07:06:09 INFO Pod/maas-controller-75bbfd978-skltg: Scheduled - Successfully assigned redhat-ods-applications/maas-controller-75bbfd978-skltg to ip-10-0-1-23.us-east-2.compute.internal 2026/07/10 07:09:49 07:06:09 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/10 07:09:49 07:06:09 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/10 07:09:49 07:06:09 INFO Pod/model-registry-operator-controller-manager-74db7cdccd-s6h64: Scheduled - Successfully assigned redhat-ods-applications/model-registry-operator-controller-manager-74db7cdccd-s6h64 to ip-10-0-1-23.us-east-2.compute.internal 2026/07/10 07:09:49 07:06:09 INFO Pod/odh-model-controller-65fdb9c5c-zr5sc: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/10 07:09:49 07:06:09 WARN Pod/maas-controller-75bbfd978-skltg: FailedMount - MountVolume.SetUp failed for volume "webhook-cert" : secret "maas-controller-webhook-cert" not found 2026/07/10 07:09:49 07:06:09 INFO Pod/odh-model-controller-65fdb9c5c-zr5sc: AddedInterface - Add eth0 [10.130.0.4/23] from ovn-kubernetes 2026/07/10 07:09:49 07:06:09 INFO ReplicaSet/maas-controller-75bbfd978: SuccessfulCreate - Created pod: maas-controller-75bbfd978-skltg 2026/07/10 07:09:49 07:06:09 INFO Deployment/maas-controller: ScalingReplicaSet - Scaled up replica set maas-controller-75bbfd978 from 0 to 1 2026/07/10 07:09:49 07:06:09 INFO ReplicaSet/model-registry-operator-controller-manager-74db7cdccd: SuccessfulCreate - Created pod: model-registry-operator-controller-manager-74db7cdccd-s6h64 2026/07/10 07:09:49 07:06:09 WARN Pod/model-registry-operator-controller-manager-74db7cdccd-s6h64: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "model-registry-operator-webhook-service" not found 2026/07/10 07:09:49 07:06:09 INFO Deployment/model-registry-operator-controller-manager: ScalingReplicaSet - Scaled up replica set model-registry-operator-controller-manager-74db7cdccd from 0 to 1 2026/07/10 07:09:49 07:06:08 INFO Pod/odh-model-controller-65fdb9c5c-zr5sc: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-65fdb9c5c-zr5sc to ip-10-0-1-23.us-east-2.compute.internal 2026/07/10 07:09:49 07:06:08 INFO ReplicaSet/odh-model-controller-65fdb9c5c: SuccessfulCreate - Created pod: odh-model-controller-65fdb9c5c-zr5sc 2026/07/10 07:09:49 07:06:08 WARN Pod/odh-model-controller-65fdb9c5c-zr5sc: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/10 07:09:49 07:06:08 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-65fdb9c5c from 0 to 1 2026/07/10 07:09:49 07:05:58 INFO Pod/dashboard-redirect-84fcf4d86-hdcwb: Killing - Stopping container nginx 2026/07/10 07:09:49 07:05:58 INFO Pod/dashboard-redirect-84fcf4d86-gtr7f: Killing - Stopping container nginx 2026/07/10 07:09:49 07:05:17 INFO Pod/rhods-dashboard-84cdfcd5d8-qj4zg: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:137ad2806fdab58f8611063a64b6cbaa3f1cf160b6d40adea8e58eb1c5d1e548" in 441ms (441ms including waiting). Image size: 185092464 bytes. 2026/07/10 07:09:49 07:05:17 INFO Pod/rhods-dashboard-84cdfcd5d8-2h24c: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:137ad2806fdab58f8611063a64b6cbaa3f1cf160b6d40adea8e58eb1c5d1e548" in 577ms (577ms including waiting). Image size: 185092464 bytes. 2026/07/10 07:09:49 07:05:17 INFO Pod/rhods-dashboard-84cdfcd5d8-qj4zg: Created - Created container: mlflow-ui 2026/07/10 07:09:49 07:05:17 INFO Pod/rhods-dashboard-84cdfcd5d8-qj4zg: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:137ad2806fdab58f8611063a64b6cbaa3f1cf160b6d40adea8e58eb1c5d1e548" 2026/07/10 07:09:49 07:05:17 INFO Pod/rhods-dashboard-84cdfcd5d8-qj4zg: Started - Started container mlflow-ui 2026/07/10 07:09:49 07:05:16 INFO Pod/rhods-dashboard-84cdfcd5d8-2h24c: Started - Started container mlflow-ui 2026/07/10 07:09:49 07:05:16 INFO Pod/rhods-dashboard-84cdfcd5d8-2h24c: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-maas-rhel9@sha256:c66d4116ea3689b73ea85fa75e451b643d3e18f5c52124c18e0d56ab368bf8ef" in 485ms (485ms including waiting). Image size: 187802205 bytes. 2026/07/10 07:09:49 07:05:16 INFO Pod/rhods-dashboard-84cdfcd5d8-2h24c: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:83f85b7a7d0b4bb9d2c1c2bd6c546ab3f9237cfc8c6820395d7d0b7f24f95410" 2026/07/10 07:09:49 07:05:16 INFO Pod/rhods-dashboard-84cdfcd5d8-2h24c: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:83f85b7a7d0b4bb9d2c1c2bd6c546ab3f9237cfc8c6820395d7d0b7f24f95410" in 672ms (672ms including waiting). Image size: 177808591 bytes. 2026/07/10 07:09:49 07:05:16 INFO Pod/rhods-dashboard-84cdfcd5d8-2h24c: Created - Created container: mlflow-ui 2026/07/10 07:09:49 07:05:16 INFO Pod/rhods-dashboard-84cdfcd5d8-qj4zg: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:83f85b7a7d0b4bb9d2c1c2bd6c546ab3f9237cfc8c6820395d7d0b7f24f95410" in 609ms (609ms including waiting). Image size: 177808591 bytes. 2026/07/10 07:09:49 07:05:16 INFO Pod/rhods-dashboard-84cdfcd5d8-2h24c: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:137ad2806fdab58f8611063a64b6cbaa3f1cf160b6d40adea8e58eb1c5d1e548" 2026/07/10 07:09:49 07:05:16 INFO Pod/rhods-dashboard-84cdfcd5d8-2h24c: Created - Created container: maas-ui 2026/07/10 07:09:49 07:05:16 INFO Pod/rhods-dashboard-84cdfcd5d8-qj4zg: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:83f85b7a7d0b4bb9d2c1c2bd6c546ab3f9237cfc8c6820395d7d0b7f24f95410" 2026/07/10 07:09:49 07:05:16 INFO Pod/rhods-dashboard-84cdfcd5d8-qj4zg: Started - Started container maas-ui 2026/07/10 07:09:49 07:05:16 INFO Pod/rhods-dashboard-84cdfcd5d8-qj4zg: Created - Created container: maas-ui 2026/07/10 07:09:49 07:05:16 INFO Pod/rhods-dashboard-84cdfcd5d8-qj4zg: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-maas-rhel9@sha256:c66d4116ea3689b73ea85fa75e451b643d3e18f5c52124c18e0d56ab368bf8ef" in 527ms (527ms including waiting). Image size: 187802205 bytes. 2026/07/10 07:09:49 07:05:16 INFO Pod/rhods-dashboard-84cdfcd5d8-2h24c: Started - Started container maas-ui 2026/07/10 07:09:49 07:05:15 INFO Pod/rhods-dashboard-84cdfcd5d8-qj4zg: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:a32d767a12da1752d119f9901982622f76cf1e681fdbcda9e00d62b91af8538d" in 664ms (664ms including waiting). Image size: 241242949 bytes. 2026/07/10 07:09:49 07:05:15 INFO Pod/rhods-dashboard-84cdfcd5d8-2h24c: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-maas-rhel9@sha256:c66d4116ea3689b73ea85fa75e451b643d3e18f5c52124c18e0d56ab368bf8ef" 2026/07/10 07:09:49 07:05:15 INFO Pod/rhods-dashboard-84cdfcd5d8-2h24c: Started - Started container gen-ai-ui 2026/07/10 07:09:49 07:05:15 INFO Pod/rhods-dashboard-84cdfcd5d8-2h24c: Created - Created container: gen-ai-ui 2026/07/10 07:09:49 07:05:15 INFO Pod/rhods-dashboard-84cdfcd5d8-2h24c: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:a32d767a12da1752d119f9901982622f76cf1e681fdbcda9e00d62b91af8538d" in 987ms (988ms including waiting). Image size: 241242949 bytes. 2026/07/10 07:09:49 07:05:15 INFO Pod/rhods-dashboard-84cdfcd5d8-qj4zg: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-maas-rhel9@sha256:c66d4116ea3689b73ea85fa75e451b643d3e18f5c52124c18e0d56ab368bf8ef" 2026/07/10 07:09:49 07:05:15 INFO Pod/rhods-dashboard-84cdfcd5d8-qj4zg: Started - Started container gen-ai-ui 2026/07/10 07:09:49 07:05:15 INFO Pod/rhods-dashboard-84cdfcd5d8-qj4zg: Created - Created container: gen-ai-ui 2026/07/10 07:09:49 07:05:14 INFO Pod/rhods-dashboard-84cdfcd5d8-2h24c: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:7114e8975a11bc2146cd6b3c0760d9d5b506c94d83d5e1f611a920fd7b15a97c" in 672ms (672ms including waiting). Image size: 182438697 bytes. 2026/07/10 07:09:49 07:05:14 INFO Pod/rhods-dashboard-84cdfcd5d8-qj4zg: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:a32d767a12da1752d119f9901982622f76cf1e681fdbcda9e00d62b91af8538d" 2026/07/10 07:09:49 07:05:14 INFO Pod/rhods-dashboard-84cdfcd5d8-qj4zg: Started - Started container model-registry-ui 2026/07/10 07:09:49 07:05:14 INFO Pod/rhods-dashboard-84cdfcd5d8-qj4zg: Created - Created container: model-registry-ui 2026/07/10 07:09:49 07:05:14 INFO Pod/rhods-dashboard-84cdfcd5d8-qj4zg: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:7114e8975a11bc2146cd6b3c0760d9d5b506c94d83d5e1f611a920fd7b15a97c" in 490ms (490ms including waiting). Image size: 182438697 bytes. 2026/07/10 07:09:49 07:05:14 INFO Pod/rhods-dashboard-84cdfcd5d8-2h24c: Created - Created container: model-registry-ui 2026/07/10 07:09:49 07:05:14 INFO Pod/rhods-dashboard-84cdfcd5d8-2h24c: Started - Started container model-registry-ui 2026/07/10 07:09:49 07:05:14 INFO Pod/rhods-dashboard-84cdfcd5d8-2h24c: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:a32d767a12da1752d119f9901982622f76cf1e681fdbcda9e00d62b91af8538d" 2026/07/10 07:09:49 07:05:14 INFO Pod/rhods-dashboard-84cdfcd5d8-qj4zg: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:7114e8975a11bc2146cd6b3c0760d9d5b506c94d83d5e1f611a920fd7b15a97c" 2026/07/10 07:09:49 07:05:14 INFO Pod/rhods-dashboard-84cdfcd5d8-qj4zg: Started - Started container kube-rbac-proxy 2026/07/10 07:09:49 07:05:14 INFO Pod/rhods-dashboard-84cdfcd5d8-qj4zg: Created - Created container: kube-rbac-proxy 2026/07/10 07:09:49 07:05:13 INFO Pod/rhods-dashboard-84cdfcd5d8-2h24c: Started - Started container kube-rbac-proxy 2026/07/10 07:09:49 07:05:13 INFO Pod/rhods-dashboard-84cdfcd5d8-2h24c: Pulled - Container image "quay.io/rhoai/odh-kube-rbac-proxy-rhel9@sha256:1e614e46799fd020ac18d95c06a463421154ce86e59ccb3fa6007e66fed9910e" already present on machine 2026/07/10 07:09:49 07:05:13 INFO Pod/rhods-dashboard-84cdfcd5d8-qj4zg: Started - Started container rhods-dashboard 2026/07/10 07:09:49 07:05:13 INFO Pod/rhods-dashboard-84cdfcd5d8-2h24c: Pulled - Successfully pulled image "quay.io/rhoai/odh-dashboard-rhel9@sha256:91277c86a8c3cac87e10664137e4d827f09e904d3cdbed0fed837ebe2a9a0022" in 708ms (708ms including waiting). Image size: 799822033 bytes. 2026/07/10 07:09:49 07:05:13 INFO Pod/rhods-dashboard-84cdfcd5d8-2h24c: Created - Created container: rhods-dashboard 2026/07/10 07:09:49 07:05:13 INFO Pod/rhods-dashboard-84cdfcd5d8-2h24c: Started - Started container rhods-dashboard 2026/07/10 07:09:49 07:05:13 INFO Pod/rhods-dashboard-84cdfcd5d8-qj4zg: Pulled - Container image "quay.io/rhoai/odh-kube-rbac-proxy-rhel9@sha256:1e614e46799fd020ac18d95c06a463421154ce86e59ccb3fa6007e66fed9910e" already present on machine 2026/07/10 07:09:49 07:05:13 INFO Pod/rhods-dashboard-84cdfcd5d8-2h24c: Created - Created container: kube-rbac-proxy 2026/07/10 07:09:49 07:05:13 INFO Pod/rhods-dashboard-84cdfcd5d8-2h24c: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:7114e8975a11bc2146cd6b3c0760d9d5b506c94d83d5e1f611a920fd7b15a97c" 2026/07/10 07:09:49 07:05:13 INFO Pod/rhods-dashboard-84cdfcd5d8-qj4zg: Created - Created container: rhods-dashboard 2026/07/10 07:09:49 07:05:13 INFO Pod/rhods-dashboard-84cdfcd5d8-qj4zg: Pulled - Successfully pulled image "quay.io/rhoai/odh-dashboard-rhel9@sha256:91277c86a8c3cac87e10664137e4d827f09e904d3cdbed0fed837ebe2a9a0022" in 777ms (777ms including waiting). Image size: 799822033 bytes. 2026/07/10 07:09:49 07:05:12 INFO Pod/rhods-dashboard-84cdfcd5d8-2h24c: Scheduled - Successfully assigned redhat-ods-applications/rhods-dashboard-84cdfcd5d8-2h24c to ip-10-0-1-76.us-east-2.compute.internal 2026/07/10 07:09:49 07:05:12 INFO Pod/rhods-dashboard-84cdfcd5d8-qj4zg: Scheduled - Successfully assigned redhat-ods-applications/rhods-dashboard-84cdfcd5d8-qj4zg to ip-10-0-1-23.us-east-2.compute.internal 2026/07/10 07:09:49 07:05:12 INFO ReplicaSet/rhods-dashboard-84cdfcd5d8: SuccessfulCreate - Created pod: rhods-dashboard-84cdfcd5d8-qj4zg 2026/07/10 07:09:49 07:05:12 INFO Pod/rhods-dashboard-84cdfcd5d8-qj4zg: Pulling - Pulling image "quay.io/rhoai/odh-dashboard-rhel9@sha256:91277c86a8c3cac87e10664137e4d827f09e904d3cdbed0fed837ebe2a9a0022" 2026/07/10 07:09:49 07:05:12 INFO Pod/rhods-dashboard-84cdfcd5d8-qj4zg: AddedInterface - Add eth0 [10.130.1.254/23] from ovn-kubernetes 2026/07/10 07:09:49 07:05:12 INFO Pod/rhods-dashboard-84cdfcd5d8-2h24c: AddedInterface - Add eth0 [10.128.0.182/23] from ovn-kubernetes 2026/07/10 07:09:49 07:05:12 INFO Pod/rhods-dashboard-84cdfcd5d8-2h24c: Pulling - Pulling image "quay.io/rhoai/odh-dashboard-rhel9@sha256:91277c86a8c3cac87e10664137e4d827f09e904d3cdbed0fed837ebe2a9a0022" 2026/07/10 07:09:49 07:05:12 INFO ReplicaSet/rhods-dashboard-84cdfcd5d8: SuccessfulCreate - Created pod: rhods-dashboard-84cdfcd5d8-2h24c 2026/07/10 07:09:49 07:05:11 INFO Deployment/rhods-dashboard: ScalingReplicaSet - Scaled up replica set rhods-dashboard-84cdfcd5d8 from 0 to 2 2026/07/10 07:09:49 07:05:11 WARN Pod/rhods-dashboard-84cdfcd5d8-xq5fz: Unhealthy - Readiness probe failed: Get "https://10.129.1.145:8444/healthz": dial tcp 10.129.1.145:8444: connect: connection refused 2026/07/10 07:09:49 07:04:57 INFO Pod/maas-controller-75bbfd978-rxdtg: Killing - Stopping container manager 2026/07/10 07:09:49 07:04:57 INFO Pod/maas-api-5b7784b768-slvn7: Killing - Stopping container maas-api 2026/07/10 07:09:49 07:04:57 WARN Pod/maas-controller-75bbfd978-rxdtg: Unhealthy - Readiness probe failed: Get "http://10.128.0.151:8081/readyz": dial tcp 10.128.0.151:8081: connect: connection refused 2026/07/10 07:09:49 07:04:56 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/10 07:09:49 07:04:56 INFO Pod/kserve-controller-manager-dd9f6cf9-45q4z: Killing - Stopping container manager 2026/07/10 07:09:49 07:04:56 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-wnwfc: Killing - Stopping container manager 2026/07/10 07:09:49 07:04:56 WARN Endpoints/model-serving-api: FailedToUpdateEndpoint - Failed to update endpoint redhat-ods-applications/model-serving-api: Operation cannot be fulfilled on endpoints "model-serving-api": StorageError: invalid object, Code: 4, Key: /kubernetes.io/services/endpoints/redhat-ods-applications/model-serving-api, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 31bc12c3-1711-4042-966a-43cd8c6894a3, UID in object meta: 2026/07/10 07:09:49 07:04:56 INFO Pod/odh-model-controller-bd4c5cbcb-kzhxq: Killing - Stopping container manager 2026/07/10 07:09:49 07:04:56 INFO Pod/model-serving-api-5679d6b4f-p2cfh: Killing - Stopping container server 2026/07/10 07:09:49 07:04:55 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/10 07:09:49 07:04:55 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/10 07:09:49 07:04:50 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/10 07:09:49 07:04:50 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/10 07:09:49 07:04:50 INFO Pod/spark-operator-webhook-5f8768dbfd-b27rj: Killing - Stopping container webhook 2026/07/10 07:09:49 07:04:50 INFO Pod/spark-operator-controller-5dd7d48f87-qhbcl: Killing - Stopping container controller 2026/07/10 07:09:49 === RESOURCE QUOTAS === 2026/07/10 07:09:49 No resource quotas found in monitored namespaces 2026/07/10 07:09:49 Classification for group 2: infrastructure/storage (code=1006, confidence=medium) 2026/07/10 07:09:49 Evidence: event Pod/workload-variant-autoscaler-controller-manager-7f5b9cd77b-rwtdb: FailedMount - MountVolume.SetUp failed for volume "epp-metrics-token" : secret "workload-variant-autoscaler-epp-metrics-token" not found FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"storage","error_code":1006,"evidence":["event Pod/workload-variant-autoscaler-controller-manager-7f5b9cd77b-rwtdb: FailedMount - MountVolume.SetUp failed for volume \"epp-metrics-token\" : secret \"workload-variant-autoscaler-epp-metrics-token\" not found"],"confidence":"medium"} 2026/07/10 07:09:49 Diagnostics complete for failed test: group 2 controller_test.go:753: Stopping: group 2 test failed. 2026/07/10 07:09:49 Circuit breaker: failure classified as infrastructure/storage (code=1006) 2026/07/10 07:09:49 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) 2026/07/10 07:13:58 Circuit breaker: success recorded, resetting failure counter (was 1) --- FAIL: TestOdhOperator/components (1016.08s) === RUN TestOdhOperator 2026/07/10 07:00:35 ARTIFACT_DIR not set, skipping periodic metrics collection 2026/07/10 07:17:42 TEST FAILURE DETECTED: components 2026/07/10 07:17:42 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:components) === 2026/07/10 07:17:43 === CLUSTER STATE === 2026/07/10 07:17:43 Node: ip-10-0-1-23.us-east-2.compute.internal 2026/07/10 07:17:43 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/10 07:17:43 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/10 07:17:43 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/10 07:17:43 Condition Ready: True - kubelet is posting ready status 2026/07/10 07:17:43 Resources: 15500m CPU, 63182900Ki memory allocatable (of 16000m CPU, 64333876Ki memory capacity) 2026/07/10 07:17:43 Node: ip-10-0-1-45.us-east-2.compute.internal 2026/07/10 07:17:43 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/10 07:17:43 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/10 07:17:43 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/10 07:17:43 Condition Ready: True - kubelet is posting ready status 2026/07/10 07:17:43 Resources: 15500m CPU, 63871012Ki memory allocatable (of 16000m CPU, 65021988Ki memory capacity) 2026/07/10 07:17:43 Node: ip-10-0-1-76.us-east-2.compute.internal 2026/07/10 07:17:43 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/10 07:17:43 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/10 07:17:43 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/10 07:17:43 Condition Ready: True - kubelet is posting ready status 2026/07/10 07:17:43 Resources: 15500m CPU, 63871016Ki memory allocatable (of 16000m CPU, 65021992Ki memory capacity) 2026/07/10 07:17:43 === DEPLOYMENTS === 2026/07/10 07:17:43 Namespace: redhat-ods-applications 2026/07/10 07:17:43 Deployment kserve-controller-manager: 1/1 (READY) 2026/07/10 07:17:43 Deployment kubeflow-trainer-controller-manager: 1/1 (READY) 2026/07/10 07:17:43 Deployment llmisvc-controller-manager: 1/1 (READY) 2026/07/10 07:17:43 Deployment maas-postgres: 1/1 (READY) 2026/07/10 07:17:43 Deployment model-serving-api: 1/1 (READY) 2026/07/10 07:17:43 Deployment odh-model-controller: 1/1 (READY) 2026/07/10 07:17:43 Namespace: kube-system 2026/07/10 07:17:43 === PODS === 2026/07/10 07:17:43 Namespace redhat-ods-applications: no problematic pods found 2026/07/10 07:17:43 Namespace kube-system: no problematic pods found 2026/07/10 07:17:43 === OPERATOR STATUS === 2026/07/10 07:17:43 Operator deployment: 3/3 ready 2026/07/10 07:17:43 === DSCI STATUS === 2026/07/10 07:17:43 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/10 07:17:43 === DSC STATUS === 2026/07/10 07:17:43 DSC default-dsc: 2026/07/10 07:17:43 Ready: True - 2026/07/10 07:17:43 ProvisioningSucceeded: True - 2026/07/10 07:17:43 ComponentsReady: True - 2026/07/10 07:17:43 AIPipelinesReady: False - Component ManagementState is set to Removed 2026/07/10 07:17:43 DashboardReady: False - Component ManagementState is set to Removed 2026/07/10 07:17:43 FeastOperatorReady: False - Component ManagementState is set to Removed 2026/07/10 07:17:43 KserveLLMInferenceServiceDependencies: True - 2026/07/10 07:17:43 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/10 07:17:43 KserveReady: True - 2026/07/10 07:17:43 KueueReady: True - 2026/07/10 07:17:43 LLM-D-WVADependencies: True - 2026/07/10 07:17:43 MLflowOperatorReady: False - Component ManagementState is set to Removed 2026/07/10 07:17:43 ModelControllerReady: True - 2026/07/10 07:17:43 ModelRegistryReady: False - Component ManagementState is set to Removed 2026/07/10 07:17:43 ModelsAsServiceReady: False - Component ManagementState is set to Removed 2026/07/10 07:17:43 OGXReady: False - Component ManagementState is set to Removed 2026/07/10 07:17:43 RayReady: False - Component ManagementState is set to Removed 2026/07/10 07:17:43 SparkOperatorReady: False - Component ManagementState is set to Removed 2026/07/10 07:17:43 TrainerReady: True - 2026/07/10 07:17:43 TrainingOperatorReady: False - Component ManagementState is set to Removed 2026/07/10 07:17:43 TrustyAIReady: False - Component ManagementState is set to Removed 2026/07/10 07:17:43 WorkbenchesReady: False - Component ManagementState is set to Removed 2026/07/10 07:17:43 === RECENT EVENTS === 2026/07/10 07:17:43 07:17:26 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-7b46b9fbbb-bmtnh_d2ab1611-8fce-4d26-a7a5-40156c2d8b88 became leader 2026/07/10 07:17:43 07:17:23 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-dd9f6cf9-l5vmt_4fa2ab4c-081a-44bd-a084-8b193b81c651 became leader 2026/07/10 07:17:43 07:17:11 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-bd4c5cbcb-5798s_f9a488b9-a7fb-4c65-b425-72b2336ea565 became leader 2026/07/10 07:17:43 07:17:09 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-bmtnh: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" in 545ms (545ms including waiting). Image size: 236366599 bytes. 2026/07/10 07:17:43 07:17:09 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-bmtnh: Started - Started container manager 2026/07/10 07:17:43 07:17:09 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-bmtnh: Created - Created container: manager 2026/07/10 07:17:43 07:17:08 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-bmtnh: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" 2026/07/10 07:17:43 07:17:08 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-bmtnh: AddedInterface - Add eth0 [10.128.0.205/23] from ovn-kubernetes 2026/07/10 07:17:43 07:17:07 INFO Pod/kserve-controller-manager-dd9f6cf9-l5vmt: Started - Started container manager 2026/07/10 07:17:43 07:17:07 WARN Pod/llmisvc-controller-manager-7b46b9fbbb-bmtnh: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "llmisvc-webhook-server-cert" not found 2026/07/10 07:17:43 07:17:07 INFO Pod/kserve-controller-manager-dd9f6cf9-l5vmt: AddedInterface - Add eth0 [10.129.1.164/23] from ovn-kubernetes 2026/07/10 07:17:43 07:17:07 INFO Pod/kserve-controller-manager-dd9f6cf9-l5vmt: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" 2026/07/10 07:17:43 07:17:07 INFO Pod/kserve-controller-manager-dd9f6cf9-l5vmt: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" in 498ms (498ms including waiting). Image size: 238435613 bytes. 2026/07/10 07:17:43 07:17:07 INFO Pod/kserve-controller-manager-dd9f6cf9-l5vmt: Created - Created container: manager 2026/07/10 07:17:43 07:17:06 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-bmtnh: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-7b46b9fbbb-bmtnh to ip-10-0-1-76.us-east-2.compute.internal 2026/07/10 07:17:43 07:17:06 INFO Pod/kserve-controller-manager-dd9f6cf9-l5vmt: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-dd9f6cf9-l5vmt to ip-10-0-1-45.us-east-2.compute.internal 2026/07/10 07:17:43 07:17:06 INFO ReplicaSet/kserve-controller-manager-dd9f6cf9: SuccessfulCreate - Created pod: kserve-controller-manager-dd9f6cf9-l5vmt 2026/07/10 07:17:43 07:17:06 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-dd9f6cf9 from 0 to 1 2026/07/10 07:17:43 07:17:06 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-7b46b9fbbb from 0 to 1 2026/07/10 07:17:43 07:17:06 INFO ReplicaSet/llmisvc-controller-manager-7b46b9fbbb: SuccessfulCreate - Created pod: llmisvc-controller-manager-7b46b9fbbb-bmtnh 2026/07/10 07:17:43 07:16:55 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/10 07:17:43 07:16:55 INFO Pod/model-serving-api-5679d6b4f-lvpsv: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/10 07:17:43 07:16:55 INFO Pod/model-serving-api-5679d6b4f-lvpsv: AddedInterface - Add eth0 [10.128.0.204/23] from ovn-kubernetes 2026/07/10 07:17:43 07:16:55 INFO Pod/model-serving-api-5679d6b4f-lvpsv: Created - Created container: server 2026/07/10 07:17:43 07:16:55 INFO Pod/model-serving-api-5679d6b4f-lvpsv: Started - Started container server 2026/07/10 07:17:43 07:16:54 INFO Pod/model-serving-api-5679d6b4f-lvpsv: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-5679d6b4f-lvpsv to ip-10-0-1-76.us-east-2.compute.internal 2026/07/10 07:17:43 07:16:54 INFO ReplicaSet/model-serving-api-5679d6b4f: SuccessfulCreate - Created pod: model-serving-api-5679d6b4f-lvpsv 2026/07/10 07:17:43 07:16:54 INFO Pod/odh-model-controller-bd4c5cbcb-5798s: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 682ms (682ms including waiting). Image size: 238959091 bytes. 2026/07/10 07:17:43 07:16:54 INFO Pod/odh-model-controller-bd4c5cbcb-5798s: Created - Created container: manager 2026/07/10 07:17:43 07:16:54 WARN ReplicaSet/model-serving-api-5679d6b4f: FailedCreate - Error creating: pods "model-serving-api-5679d6b4f-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/10 07:17:43 07:16:54 INFO Pod/odh-model-controller-bd4c5cbcb-5798s: Started - Started container manager 2026/07/10 07:17:43 07:16:54 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-5679d6b4f from 0 to 1 2026/07/10 07:17:43 07:16:53 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/10 07:17:43 07:16:53 INFO Pod/odh-model-controller-bd4c5cbcb-5798s: AddedInterface - Add eth0 [10.128.0.203/23] from ovn-kubernetes 2026/07/10 07:17:43 07:16:53 INFO Pod/odh-model-controller-bd4c5cbcb-5798s: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/10 07:17:43 07:16:52 INFO Pod/odh-model-controller-bd4c5cbcb-5798s: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-bd4c5cbcb-5798s to ip-10-0-1-76.us-east-2.compute.internal 2026/07/10 07:17:43 07:16:52 INFO ReplicaSet/odh-model-controller-bd4c5cbcb: SuccessfulCreate - Created pod: odh-model-controller-bd4c5cbcb-5798s 2026/07/10 07:17:43 07:16:52 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-bd4c5cbcb from 0 to 1 2026/07/10 07:17:43 07:16:52 WARN Pod/odh-model-controller-bd4c5cbcb-5798s: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/10 07:17:43 07:16:45 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-dd9f6cf9 from 0 to 1 2026/07/10 07:17:43 07:16:45 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-7b46b9fbbb from 0 to 1 2026/07/10 07:17:43 07:16:45 WARN ReplicaSet/kserve-controller-manager-dd9f6cf9: FailedCreate - Error creating: pods "kserve-controller-manager-dd9f6cf9-" is forbidden: error looking up service account redhat-ods-applications/kserve-controller-manager: serviceaccount "kserve-controller-manager" not found 2026/07/10 07:17:43 07:16:45 WARN ReplicaSet/llmisvc-controller-manager-7b46b9fbbb: FailedCreate - Error creating: pods "llmisvc-controller-manager-7b46b9fbbb-" is forbidden: error looking up service account redhat-ods-applications/llmisvc-controller-manager: serviceaccount "llmisvc-controller-manager" not found 2026/07/10 07:17:43 07:16:43 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/10 07:17:43 07:16:43 INFO Pod/model-serving-api-5679d6b4f-m6jc4: Killing - Stopping container server 2026/07/10 07:17:43 07:16:43 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-vtl9n: Killing - Stopping container manager 2026/07/10 07:17:43 07:16:43 INFO Pod/kserve-controller-manager-dd9f6cf9-jzw78: Killing - Stopping container manager 2026/07/10 07:17:43 07:16:43 INFO Pod/odh-model-controller-bd4c5cbcb-9vpsv: Killing - Stopping container manager 2026/07/10 07:17:43 07:16:25 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-7b46b9fbbb-vtl9n_71540589-69dd-4c3f-9b9d-18387377e457 became leader 2026/07/10 07:17:43 07:16:22 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-dd9f6cf9-jzw78_25d34ac5-f8b1-4ff8-abfb-163ef242f9b7 became leader 2026/07/10 07:17:43 07:16:12 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-bd4c5cbcb-9vpsv_69613fa6-1e3d-4a84-bd14-42e4b364f600 became leader 2026/07/10 07:17:43 07:16:07 INFO Pod/kserve-controller-manager-dd9f6cf9-jzw78: Created - Created container: manager 2026/07/10 07:17:43 07:16:07 INFO Pod/kserve-controller-manager-dd9f6cf9-jzw78: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" in 603ms (603ms including waiting). Image size: 238435613 bytes. 2026/07/10 07:17:43 07:16:07 INFO Pod/kserve-controller-manager-dd9f6cf9-jzw78: Started - Started container manager 2026/07/10 07:17:43 07:16:06 INFO Pod/kserve-controller-manager-dd9f6cf9-jzw78: AddedInterface - Add eth0 [10.129.1.163/23] from ovn-kubernetes 2026/07/10 07:17:43 07:16:06 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-vtl9n: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" in 574ms (574ms including waiting). Image size: 236366599 bytes. 2026/07/10 07:17:43 07:16:06 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-vtl9n: Created - Created container: manager 2026/07/10 07:17:43 07:16:06 INFO Pod/kserve-controller-manager-dd9f6cf9-jzw78: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" 2026/07/10 07:17:43 07:16:06 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-vtl9n: Started - Started container manager 2026/07/10 07:17:43 07:16:05 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-vtl9n: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-7b46b9fbbb-vtl9n to ip-10-0-1-76.us-east-2.compute.internal 2026/07/10 07:17:43 07:16:05 INFO Pod/kserve-controller-manager-dd9f6cf9-jzw78: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-dd9f6cf9-jzw78 to ip-10-0-1-45.us-east-2.compute.internal 2026/07/10 07:17:43 07:16:05 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-7b46b9fbbb from 0 to 1 2026/07/10 07:17:43 07:16:05 INFO ReplicaSet/kserve-controller-manager-dd9f6cf9: SuccessfulCreate - Created pod: kserve-controller-manager-dd9f6cf9-jzw78 2026/07/10 07:17:43 07:16:05 INFO ReplicaSet/llmisvc-controller-manager-7b46b9fbbb: SuccessfulCreate - Created pod: llmisvc-controller-manager-7b46b9fbbb-vtl9n 2026/07/10 07:17:43 07:16:05 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-vtl9n: AddedInterface - Add eth0 [10.128.0.202/23] from ovn-kubernetes 2026/07/10 07:17:43 07:16:05 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-vtl9n: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" 2026/07/10 07:17:43 07:16:05 WARN Pod/kserve-controller-manager-dd9f6cf9-jzw78: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "kserve-webhook-server-cert" not found 2026/07/10 07:17:43 07:16:05 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-dd9f6cf9 from 0 to 1 2026/07/10 07:17:43 07:16:04 INFO Lease/trainer.kubeflow.org: LeaderElection - kubeflow-trainer-controller-manager-b5fd84b9c-txzcj_18af7def-ba20-44f3-956a-6411d0309882 became leader 2026/07/10 07:17:43 07:15:55 INFO Pod/model-serving-api-5679d6b4f-m6jc4: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/10 07:17:43 07:15:55 INFO Pod/model-serving-api-5679d6b4f-m6jc4: AddedInterface - Add eth0 [10.128.0.201/23] from ovn-kubernetes 2026/07/10 07:17:43 07:15:55 INFO Pod/model-serving-api-5679d6b4f-m6jc4: Started - Started container server 2026/07/10 07:17:43 07:15:55 INFO Pod/model-serving-api-5679d6b4f-m6jc4: Created - Created container: server 2026/07/10 07:17:43 07:15:54 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/10 07:17:43 07:15:54 INFO Pod/model-serving-api-5679d6b4f-m6jc4: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-5679d6b4f-m6jc4 to ip-10-0-1-76.us-east-2.compute.internal 2026/07/10 07:17:43 07:15:54 INFO ReplicaSet/model-serving-api-5679d6b4f: SuccessfulCreate - Created pod: model-serving-api-5679d6b4f-m6jc4 2026/07/10 07:17:43 07:15:54 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-5679d6b4f from 0 to 1 2026/07/10 07:17:43 07:15:54 WARN ReplicaSet/model-serving-api-5679d6b4f: FailedCreate - Error creating: pods "model-serving-api-5679d6b4f-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/10 07:17:43 07:15:54 INFO Pod/odh-model-controller-bd4c5cbcb-9vpsv: Started - Started container manager 2026/07/10 07:17:43 07:15:54 INFO Pod/odh-model-controller-bd4c5cbcb-9vpsv: Created - Created container: manager 2026/07/10 07:17:43 07:15:54 INFO Pod/odh-model-controller-bd4c5cbcb-9vpsv: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 894ms (894ms including waiting). Image size: 238959091 bytes. 2026/07/10 07:17:43 07:15:53 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/10 07:17:43 07:15:53 INFO Pod/odh-model-controller-bd4c5cbcb-9vpsv: AddedInterface - Add eth0 [10.128.0.200/23] from ovn-kubernetes 2026/07/10 07:17:43 07:15:53 INFO Pod/odh-model-controller-bd4c5cbcb-9vpsv: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/10 07:17:43 07:15:52 INFO Pod/odh-model-controller-bd4c5cbcb-9vpsv: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-bd4c5cbcb-9vpsv to ip-10-0-1-76.us-east-2.compute.internal 2026/07/10 07:17:43 07:15:52 WARN Pod/odh-model-controller-bd4c5cbcb-9vpsv: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/10 07:17:43 07:15:52 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-bd4c5cbcb from 0 to 1 2026/07/10 07:17:43 07:15:52 INFO ReplicaSet/odh-model-controller-bd4c5cbcb: SuccessfulCreate - Created pod: odh-model-controller-bd4c5cbcb-9vpsv 2026/07/10 07:17:43 07:15:49 WARN ReplicaSet/llmisvc-controller-manager-7b46b9fbbb: FailedCreate - Error creating: pods "llmisvc-controller-manager-7b46b9fbbb-" is forbidden: error looking up service account redhat-ods-applications/llmisvc-controller-manager: serviceaccount "llmisvc-controller-manager" not found 2026/07/10 07:17:43 07:15:49 WARN ReplicaSet/kserve-controller-manager-dd9f6cf9: FailedCreate - Error creating: pods "kserve-controller-manager-dd9f6cf9-" is forbidden: error looking up service account redhat-ods-applications/kserve-controller-manager: serviceaccount "kserve-controller-manager" not found 2026/07/10 07:17:43 07:15:49 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-dd9f6cf9 from 0 to 1 2026/07/10 07:17:43 07:15:49 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-7b46b9fbbb from 0 to 1 2026/07/10 07:17:43 07:15:48 INFO Pod/kubeflow-trainer-controller-manager-b5fd84b9c-txzcj: Scheduled - Successfully assigned redhat-ods-applications/kubeflow-trainer-controller-manager-b5fd84b9c-txzcj to ip-10-0-1-76.us-east-2.compute.internal 2026/07/10 07:17:43 07:15:48 INFO Deployment/kubeflow-trainer-controller-manager: ScalingReplicaSet - Scaled up replica set kubeflow-trainer-controller-manager-b5fd84b9c from 0 to 1 2026/07/10 07:17:43 07:15:48 INFO ReplicaSet/kubeflow-trainer-controller-manager-b5fd84b9c: SuccessfulCreate - Created pod: kubeflow-trainer-controller-manager-b5fd84b9c-txzcj 2026/07/10 07:17:43 07:15:48 INFO Pod/kubeflow-trainer-controller-manager-b5fd84b9c-txzcj: Started - Started container manager 2026/07/10 07:17:43 07:15:48 INFO Pod/kubeflow-trainer-controller-manager-b5fd84b9c-txzcj: Created - Created container: manager 2026/07/10 07:17:43 07:15:48 INFO Pod/kubeflow-trainer-controller-manager-b5fd84b9c-txzcj: Pulled - Container image "quay.io/rhoai/odh-trainer-rhel9@sha256:59ea523e2df9c1e705c755308ac11420682081cc63d54890e2b15fbd7d8ba570" already present on machine 2026/07/10 07:17:43 07:15:48 INFO Pod/kubeflow-trainer-controller-manager-b5fd84b9c-txzcj: AddedInterface - Add eth0 [10.128.0.199/23] from ovn-kubernetes 2026/07/10 07:17:43 07:15:42 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-jhq6r: Killing - Stopping container manager 2026/07/10 07:17:43 07:15:42 INFO Pod/odh-model-controller-bd4c5cbcb-lkclj: Killing - Stopping container manager 2026/07/10 07:17:43 07:15:42 INFO Pod/model-serving-api-5679d6b4f-k52s5: Killing - Stopping container server 2026/07/10 07:17:43 07:15:42 INFO Pod/kserve-controller-manager-dd9f6cf9-t5ntz: Killing - Stopping container manager 2026/07/10 07:17:43 07:15:33 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/10 07:17:43 07:15:33 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/10 07:17:43 07:15:33 INFO Pod/maas-api-5b7784b768-cc8nq: Killing - Stopping container maas-api 2026/07/10 07:17:43 07:15:32 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/10 07:17:43 07:15:32 INFO Pod/maas-controller-75bbfd978-s2kkl: Killing - Stopping container manager 2026/07/10 07:17:43 07:15:31 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/10 07:17:43 07:15:02 INFO Pod/maas-api-5b7784b768-cc8nq: Started - Started container maas-api 2026/07/10 07:17:43 07:15:02 INFO Pod/maas-api-5b7784b768-cc8nq: Pulled - Successfully pulled image "quay.io/rhoai/odh-maas-api-rhel9@sha256:9c87c8243770b9ebbff6ccab0dd663ccb402488cf5ee11f491ec90cf3b4686f6" in 706ms (706ms including waiting). Image size: 175018396 bytes. 2026/07/10 07:17:43 07:15:02 INFO Pod/maas-api-5b7784b768-cc8nq: Created - Created container: maas-api 2026/07/10 07:17:43 07:15:01 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/10 07:17:43 07:15:01 INFO Pod/maas-api-5b7784b768-cc8nq: Scheduled - Successfully assigned redhat-ods-applications/maas-api-5b7784b768-cc8nq to ip-10-0-1-45.us-east-2.compute.internal 2026/07/10 07:17:43 07:15:01 INFO ReplicaSet/maas-api-5b7784b768: SuccessfulCreate - Created pod: maas-api-5b7784b768-cc8nq 2026/07/10 07:17:43 07:15:01 INFO Pod/maas-api-5b7784b768-cc8nq: AddedInterface - Add eth0 [10.129.1.160/23] from ovn-kubernetes 2026/07/10 07:17:43 07:15:01 INFO Pod/maas-api-5b7784b768-cc8nq: Pulling - Pulling image "quay.io/rhoai/odh-maas-api-rhel9@sha256:9c87c8243770b9ebbff6ccab0dd663ccb402488cf5ee11f491ec90cf3b4686f6" 2026/07/10 07:17:43 07:15:00 WARN ReplicaSet/maas-api-5b7784b768: FailedCreate - Error creating: pods "maas-api-5b7784b768-" is forbidden: error looking up service account redhat-ods-applications/maas-api: serviceaccount "maas-api" not found 2026/07/10 07:17:43 07:15:00 INFO Deployment/maas-api: ScalingReplicaSet - Scaled up replica set maas-api-5b7784b768 from 0 to 1 2026/07/10 07:17:43 07:14:45 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-7b46b9fbbb-jhq6r_565fdfbb-9205-4cc7-a058-b33932d75fa8 became leader 2026/07/10 07:17:43 07:14:42 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-dd9f6cf9-t5ntz_12ed1cf9-99a4-4f1b-99d5-45af7a2d2515 became leader 2026/07/10 07:17:43 07:14:32 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/10 07:17:43 07:14:29 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-bd4c5cbcb-lkclj_63f25b97-8e7a-48a7-aba7-3695e711f9a8 became leader 2026/07/10 07:17:43 07:14:29 INFO Lease/maas-controller.models-as-a-service.opendatahub.io: LeaderElection - maas-controller-75bbfd978-s2kkl_8cf188f7-e5a3-40ea-98ca-82b5449bdb19 became leader 2026/07/10 07:17:43 07:14:26 INFO Pod/kserve-controller-manager-dd9f6cf9-t5ntz: Started - Started container manager 2026/07/10 07:17:43 07:14:26 INFO Pod/kserve-controller-manager-dd9f6cf9-t5ntz: Created - Created container: manager 2026/07/10 07:17:43 07:14:26 INFO Pod/kserve-controller-manager-dd9f6cf9-t5ntz: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" in 551ms (551ms including waiting). Image size: 238435613 bytes. 2026/07/10 07:17:43 07:14:25 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-jhq6r: Created - Created container: manager 2026/07/10 07:17:43 07:14:25 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-jhq6r: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" 2026/07/10 07:17:43 07:14:25 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-jhq6r: AddedInterface - Add eth0 [10.128.0.198/23] from ovn-kubernetes 2026/07/10 07:17:43 07:14:25 INFO Pod/kserve-controller-manager-dd9f6cf9-t5ntz: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" 2026/07/10 07:17:43 07:14:25 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-jhq6r: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" in 553ms (553ms including waiting). Image size: 236366599 bytes. 2026/07/10 07:17:43 07:14:25 INFO Pod/kserve-controller-manager-dd9f6cf9-t5ntz: AddedInterface - Add eth0 [10.128.0.197/23] from ovn-kubernetes 2026/07/10 07:17:43 07:14:25 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-jhq6r: Started - Started container manager 2026/07/10 07:17:43 07:14:24 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-jhq6r: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-7b46b9fbbb-jhq6r to ip-10-0-1-76.us-east-2.compute.internal 2026/07/10 07:17:43 07:14:24 INFO Pod/kserve-controller-manager-dd9f6cf9-t5ntz: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-dd9f6cf9-t5ntz to ip-10-0-1-76.us-east-2.compute.internal 2026/07/10 07:17:43 07:14:24 INFO ReplicaSet/kserve-controller-manager-dd9f6cf9: SuccessfulCreate - Created pod: kserve-controller-manager-dd9f6cf9-t5ntz 2026/07/10 07:17:43 07:14:24 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-7b46b9fbbb from 0 to 1 2026/07/10 07:17:43 07:14:24 WARN Pod/kserve-controller-manager-dd9f6cf9-t5ntz: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "kserve-webhook-server-cert" not found 2026/07/10 07:17:43 07:14:24 INFO ReplicaSet/llmisvc-controller-manager-7b46b9fbbb: SuccessfulCreate - Created pod: llmisvc-controller-manager-7b46b9fbbb-jhq6r 2026/07/10 07:17:43 07:14:24 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-dd9f6cf9 from 0 to 1 2026/07/10 07:17:43 07:14:13 INFO Pod/model-serving-api-5679d6b4f-k52s5: AddedInterface - Add eth0 [10.129.1.159/23] from ovn-kubernetes 2026/07/10 07:17:43 07:14:13 INFO Pod/model-serving-api-5679d6b4f-k52s5: Created - Created container: server 2026/07/10 07:17:43 07:14:13 INFO Pod/model-serving-api-5679d6b4f-k52s5: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/10 07:17:43 07:14:13 INFO Pod/model-serving-api-5679d6b4f-k52s5: Started - Started container server 2026/07/10 07:17:43 07:14:12 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/10 07:17:43 07:14:12 INFO Pod/model-serving-api-5679d6b4f-k52s5: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-5679d6b4f-k52s5 to ip-10-0-1-45.us-east-2.compute.internal 2026/07/10 07:17:43 07:14:12 INFO Pod/odh-model-controller-bd4c5cbcb-lkclj: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 558ms (558ms including waiting). Image size: 238959091 bytes. 2026/07/10 07:17:43 07:14:12 INFO Pod/odh-model-controller-bd4c5cbcb-lkclj: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/10 07:17:43 07:14:12 WARN ReplicaSet/model-serving-api-5679d6b4f: FailedCreate - Error creating: pods "model-serving-api-5679d6b4f-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/10 07:17:43 07:14:12 INFO ReplicaSet/model-serving-api-5679d6b4f: SuccessfulCreate - Created pod: model-serving-api-5679d6b4f-k52s5 2026/07/10 07:17:43 07:14:12 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-5679d6b4f from 0 to 1 2026/07/10 07:17:43 07:14:12 INFO Pod/odh-model-controller-bd4c5cbcb-lkclj: Started - Started container manager 2026/07/10 07:17:43 07:14:12 INFO Pod/odh-model-controller-bd4c5cbcb-lkclj: Created - Created container: manager 2026/07/10 07:17:43 07:14:11 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/10 07:17:43 07:14:11 INFO Pod/maas-controller-75bbfd978-s2kkl: Pulled - Successfully pulled image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" in 516ms (516ms including waiting). Image size: 212866077 bytes. 2026/07/10 07:17:43 07:14:11 INFO Pod/maas-controller-75bbfd978-s2kkl: Started - Started container manager 2026/07/10 07:17:43 07:14:11 INFO Pod/odh-model-controller-bd4c5cbcb-lkclj: AddedInterface - Add eth0 [10.128.0.196/23] from ovn-kubernetes 2026/07/10 07:17:43 07:14:11 WARN Pod/odh-model-controller-bd4c5cbcb-lkclj: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/10 07:17:43 07:14:11 INFO Pod/maas-controller-75bbfd978-s2kkl: Created - Created container: manager 2026/07/10 07:17:43 07:14:10 INFO Pod/odh-model-controller-bd4c5cbcb-lkclj: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-bd4c5cbcb-lkclj to ip-10-0-1-76.us-east-2.compute.internal 2026/07/10 07:17:43 07:14:10 INFO Pod/maas-controller-75bbfd978-s2kkl: Scheduled - Successfully assigned redhat-ods-applications/maas-controller-75bbfd978-s2kkl to ip-10-0-1-76.us-east-2.compute.internal 2026/07/10 07:17:43 07:14:10 INFO Deployment/maas-controller: ScalingReplicaSet - Scaled up replica set maas-controller-75bbfd978 from 0 to 1 2026/07/10 07:17:43 07:14:10 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-bd4c5cbcb from 0 to 1 2026/07/10 07:17:43 07:14:10 INFO ReplicaSet/maas-controller-75bbfd978: SuccessfulCreate - Created pod: maas-controller-75bbfd978-s2kkl 2026/07/10 07:17:43 07:14:10 INFO ReplicaSet/odh-model-controller-bd4c5cbcb: SuccessfulCreate - Created pod: odh-model-controller-bd4c5cbcb-lkclj 2026/07/10 07:17:43 07:14:10 INFO Pod/maas-controller-75bbfd978-s2kkl: AddedInterface - Add eth0 [10.128.0.195/23] from ovn-kubernetes 2026/07/10 07:17:43 07:14:10 INFO Pod/maas-controller-75bbfd978-s2kkl: Pulling - Pulling image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" 2026/07/10 07:17:43 07:14:02 INFO Pod/maas-postgres-bb9d55498-hd27p: Started - Started container postgres 2026/07/10 07:17:43 07:14:02 INFO Pod/maas-postgres-bb9d55498-hd27p: Created - Created container: postgres 2026/07/10 07:17:43 07:14:02 INFO Pod/maas-postgres-bb9d55498-hd27p: Pulled - Successfully pulled image "registry.redhat.io/rhel9/postgresql-15:latest" in 2.785s (2.785s including waiting). Image size: 400818801 bytes. 2026/07/10 07:17:43 07:13:59 INFO Pod/maas-postgres-bb9d55498-hd27p: Scheduled - Successfully assigned redhat-ods-applications/maas-postgres-bb9d55498-hd27p to ip-10-0-1-76.us-east-2.compute.internal 2026/07/10 07:17:43 07:13:59 INFO ReplicaSet/maas-postgres-bb9d55498: SuccessfulCreate - Created pod: maas-postgres-bb9d55498-hd27p 2026/07/10 07:17:43 07:13:59 INFO Pod/maas-postgres-bb9d55498-hd27p: AddedInterface - Add eth0 [10.128.0.194/23] from ovn-kubernetes 2026/07/10 07:17:43 07:13:59 INFO Pod/maas-postgres-bb9d55498-hd27p: Pulling - Pulling image "registry.redhat.io/rhel9/postgresql-15:latest" 2026/07/10 07:17:43 07:13:59 INFO Deployment/maas-postgres: ScalingReplicaSet - Scaled up replica set maas-postgres-bb9d55498 from 0 to 1 2026/07/10 07:17:43 07:13:49 INFO Pod/trustyai-service-operator-controller-manager-5dccbd95db-wzvd7: Killing - Stopping container manager 2026/07/10 07:17:43 07:13:40 INFO Pod/odh-model-controller-bd4c5cbcb-wctmz: Killing - Stopping container manager 2026/07/10 07:17:43 07:13:40 INFO Pod/model-serving-api-5679d6b4f-hqpmr: Killing - Stopping container server 2026/07/10 07:17:43 07:13:40 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-jd9wr: Killing - Stopping container manager 2026/07/10 07:17:43 07:13:40 WARN Endpoints/maas-controller-webhook-service: FailedToUpdateEndpoint - Failed to update endpoint redhat-ods-applications/maas-controller-webhook-service: Operation cannot be fulfilled on endpoints "maas-controller-webhook-service": StorageError: invalid object, Code: 4, Key: /kubernetes.io/services/endpoints/redhat-ods-applications/maas-controller-webhook-service, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: a15d08e4-d7c8-4b1c-b003-a5e7517fea61, UID in object meta: 2026/07/10 07:17:43 07:13:40 INFO Pod/kserve-controller-manager-dd9f6cf9-28v7t: Killing - Stopping container manager 2026/07/10 07:17:43 07:13:40 INFO Pod/maas-api-5b7784b768-lhj94: Killing - Stopping container maas-api 2026/07/10 07:17:43 07:13:40 INFO Lease/b7e9931f.trustyai.opendatahub.io: LeaderElection - trustyai-service-operator-controller-manager-5dccbd95db-wzvd7_51dbf755-5400-41a0-a3fe-39a1afa8e7e4 became leader 2026/07/10 07:17:43 07:13:23 INFO Pod/trustyai-service-operator-controller-manager-5dccbd95db-wzvd7: Scheduled - Successfully assigned redhat-ods-applications/trustyai-service-operator-controller-manager-5dccbd95db-wzvd7 to ip-10-0-1-76.us-east-2.compute.internal 2026/07/10 07:17:43 07:13:23 INFO Pod/trustyai-service-operator-controller-manager-5dccbd95db-wzvd7: Pulled - Container image "quay.io/rhoai/odh-trustyai-service-operator-rhel9@sha256:17b36dfec55d1d9aeb3e778d1cf468b918e2b762efb53af1aa6af095387e4da2" already present on machine 2026/07/10 07:17:43 07:13:23 INFO Pod/trustyai-service-operator-controller-manager-5dccbd95db-8mtp9: Killing - Stopping container manager 2026/07/10 07:17:43 07:13:23 INFO Pod/trustyai-service-operator-controller-manager-5dccbd95db-wzvd7: AddedInterface - Add eth0 [10.128.0.193/23] from ovn-kubernetes 2026/07/10 07:17:43 07:13:23 INFO Pod/trustyai-service-operator-controller-manager-5dccbd95db-wzvd7: Created - Created container: manager 2026/07/10 07:17:43 07:13:23 INFO Pod/trustyai-service-operator-controller-manager-5dccbd95db-wzvd7: Started - Started container manager 2026/07/10 07:17:43 07:13:23 INFO Pod/trustyai-service-operator-controller-manager-5dccbd95db-z6d9j: Killing - Stopping container manager 2026/07/10 07:17:43 07:13:23 INFO ReplicaSet/trustyai-service-operator-controller-manager-5dccbd95db: SuccessfulCreate - Created pod: trustyai-service-operator-controller-manager-5dccbd95db-wzvd7 2026/07/10 07:17:43 07:13:23 INFO Deployment/trustyai-service-operator-controller-manager: ScalingReplicaSet - Scaled up replica set trustyai-service-operator-controller-manager-5dccbd95db from 0 to 1 2026/07/10 07:17:43 07:12:55 INFO Pod/maas-controller-75bbfd978-skltg: Pulled - Successfully pulled image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" in 599ms (599ms including waiting). Image size: 212866077 bytes. 2026/07/10 07:17:43 07:12:55 INFO Pod/maas-controller-75bbfd978-skltg: Started - Started container manager 2026/07/10 07:17:43 07:12:55 INFO Pod/maas-controller-75bbfd978-skltg: Created - Created container: manager 2026/07/10 07:17:43 07:12:54 INFO Pod/maas-controller-75bbfd978-skltg: Pulling - Pulling image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" 2026/07/10 07:17:43 === RESOURCE QUOTAS === 2026/07/10 07:17:43 No resource quotas found in monitored namespaces 2026/07/10 07:17:43 Classification for components: infrastructure/storage (code=1006, confidence=medium) 2026/07/10 07:17:43 Evidence: event Pod/llmisvc-controller-manager-7b46b9fbbb-bmtnh: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "llmisvc-webhook-server-cert" not found FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"storage","error_code":1006,"evidence":["event Pod/llmisvc-controller-manager-7b46b9fbbb-bmtnh: FailedMount - MountVolume.SetUp failed for volume \"cert\" : secret \"llmisvc-webhook-server-cert\" not found"],"confidence":"medium"} 2026/07/10 07:17:43 Diagnostics complete for failed test: components controller_test.go:484: Stopping: components test failed. 2026/07/10 07:17:43 Circuit breaker: failure classified as infrastructure/storage (code=1006) 2026/07/10 07:17:43 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) Deletion Policy: Never. Skipping deletion tests. 2026/07/10 07:17:44 Circuit breaker: no trips during this run (threshold=3) --- FAIL: TestOdhOperator (1029.09s)