=== RUN TestOdhOperator/components/group_2/kueue/Validate_component_removed_to_unmanaged_transition 2026-07-19T13:38:16Z INFO Cluster config {"Operator Namespace": "redhat-ods-operator", "Application Namespace": "redhat-ods-applications", "Release": {"name":"OpenShift AI Self-Managed","version":"3.5.0-ea.2"}, "Cluster": {"type":"OpenShift","version":"4.21.19"}} helper_test.go:150: Setting Kueue component (default-kueue) to Removed mode to start with a clean state. helper_test.go:150: Cleaning up any existing Kueue test resources. helper_test.go:150: Cleaning up Kueue cluster-scoped resources helper_test.go:150: Will attempt to delete 2 Kueue resources helper_test.go:150: Attempting to delete ClusterQueue /default helper_test.go:150: Successfully processed deletion of ClusterQueue /default helper_test.go:150: Attempting to delete Kueue /cluster helper_test.go:150: Successfully processed deletion of Kueue /cluster helper_test.go:150: Creating test namespace (test-kueue-managed-d9ed6i15uabs74og2ts0) with Kueue management annotation. helper_test.go:150: Creating Kueue ConfigMap (kueue-manager-config) to verify default Kueue resource is created correctly. helper_test.go:150: Creating Kueue ConfigMap (kueue-manager-config) in namespace (redhat-ods-applications). helper_test.go:150: Updating the management state of the component in the DataScienceCluster to Unmanaged (expecting NotReady initially due to missing operator). helper_test.go:150: Ensuring LWS Operator is installed (namespace=openshift-lws-operator, name=leader-worker-set). helper_test.go:150: Ensuring LeaderWorkerSetOperator CR exists (namespace=openshift-lws-operator, name=cluster). helper_test.go:150: Ensuring OCP Kueue Operator is installed (namespace=openshift-kueue-operator, name=kueue-operator). helper_test.go:150: Verifying the component in the DataScienceCluster transitions to Ready state. helper_test.go:150: Verifying Kueue configuration (cluster) is created with all expected frameworks. testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_2/kueue/Validate_component_removed_to_unmanaged_transition (30.66s) === RUN TestOdhOperator/components/group_2/kueue === PAUSE TestOdhOperator/components/group_2/kueue === CONT TestOdhOperator/components/group_2/kueue test_context_test.go:223: Timed out after 10.001s. The function passed to Eventually failed at /workspace/tests/e2e/resource_applier_test.go:114 with: Expected resource 'cluster' of kind 'Kueue' to match condition {(["BatchJob", "Deployment", "JobSet", "LeaderWorkerSet", "MPIJob", "PaddleJob", "Pod", "PyTorchJob", "RayCluster", "RayJob", "StatefulSet", "TFJob", "TrainJob", "XGBoostJob"] - .spec.config.integrations.frameworks) | length == 0 []}. Expected <string>: map[apiVersion:kueue.openshift.io/v1 kind:Kueue metadata:map[annotations:map[platform.opendatahub.io/instance.generation:1 platform.opendatahub.io/instance.name:default-kueue platform.opendatahub.io/instance.uid:6eed7308-de6d-49d5-a676-9882fc6a032d platform.opendatahub.io/type:OpenShift AI Self-Managed platform.opendatahub.io/version:3.5.0-ea.2] creationTimestamp:2026-07-19T13:01:28Z deletionGracePeriodSeconds:0 deletionTimestamp:2026-07-19T13:38:26Z finalizers:[kueue.openshift.io/finalizer] generation:2 labels:map[platform.opendatahub.io/part-of:kueue] managedFields:[map[apiVersion:kueue.openshift.io/v1 fieldsType:FieldsV1 fieldsV1:map[f:status:map[f:conditions:map[k:{"type":"Available"}:map[.:map[] f:lastTransitionTime:map[] f:message:map[] f:reason:map[] f:status:map[] f:type:map[]] k:{"type":"CertManagerAvailable"}:map[.:map[] f:lastTransitionTime:map[] f:message:map[] f:reason:map[] f:status:map[] f:type:map[]] k:{"type":"Degraded"}:map[.:map[] f:lastTransitionTime:map[] f:message:map[] f:reason:map[] f:status:map[] f:type:map[]] k:{"type":"Progressing"}:map[.:map[] f:lastTransitionTime:map[] f:message:map[] f:reason:map[] f:status:map[] f:type:map[]]] f:readyReplicas:map[]]] manager:kueue-operator operation:Apply subresource:status time:2026-07-19T13:02:16Z] map[apiVersion:kueue.openshift.io/v1 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:annotations:map[.:map[] f:platform.opendatahub.io/instance.generation:map[] f:platform.opendatahub.io/instance.name:map[] f:platform.opendatahub.io/instance.uid:map[] f:platform.opendatahub.io/type:map[] f:platform.opendatahub.io/version:map[]] f:labels:map[.:map[] f:platform.opendatahub.io/part-of:map[]]] f:spec:map[.:map[] f:config:map[.:map[] f:integrations:map[.:map[] f:frameworks:map[.:map[] v:"Deployment":map[] v:"Pod":map[] v:"PyTorchJob":map[] v:"RayCluster":map[] v:"RayJob":map[] v:"StatefulSet":map[] v:"TrainJob":map[]]]] f:logLevel:map[] f:managementState:map[] f:operatorLogLevel:map[]]] manager:manager operation:Update time:2026-07-19T13:01:28Z] map[apiVersion:kueue.openshift.io/v1 fieldsType:FieldsV1 fieldsV1:map[f:metadata:map[f:finalizers:map[.:map[] v:"kueue.openshift.io/finalizer":map[]]]] manager:kueue-operator operation:Update time:2026-07-19T13:38:26Z]] name:cluster resourceVersion:39681209 uid:91cec92f-8d05-488d-a6e5-41546833d029] spec:map[config:map[integrations:map[frameworks:[Deployment Pod PyTorchJob RayCluster RayJob StatefulSet TrainJob]]] logLevel:Normal managementState:Managed operatorLogLevel:Normal] status:map[conditions:[map[lastTransitionTime:2026-07-19T13:02:11Z message:All 2 replicas are ready reason:AllReplicasReady status:True type:Available] map[lastTransitionTime:2026-07-19T13:02:11Z message:Deployment is up to date reason:AsExpected status:False type:Progressing] map[lastTransitionTime:2026-07-19T13:02:11Z message: reason:AsExpected status:False type:Degraded] map[lastTransitionTime:2026-07-19T13:01:54Z message:cert-manager is installed reason:CertManagerInstalled status:True type:CertManagerAvailable]] readyReplicas:2]] to match expression <string>: (["BatchJob", "Deployment", "JobSet", "LeaderWorkerSet", "MPIJob", "PaddleJob", "Pod", "PyTorchJob", "RayCluster", "RayJob", "StatefulSet", "TFJob", "TrainJob", "XGBoostJob"] - .spec.config.integrations.frameworks) | length == 0 test_context_test.go:1106: cleanup: best-effort delete failed for /, Kind= redhat-ods-applications/kueue-manager-config: Object 'Kind' is missing in 'unstructured object has no kind' --- FAIL: TestOdhOperator/components/group_2/kueue (30.82s) === RUN TestOdhOperator/components/group_2 --- FAIL: TestOdhOperator/components/group_2 (0.00s) === RUN TestOdhOperator/components/group_4/modelsasservice/Validate_Tenant_deleted_on_disable helper_test.go:150: Verifying Tenant models-as-a-service/default-tenant is present before disable helper_test.go:150: Disabling MaaS subcomponent (setting to Removed) helper_test.go:150: Waiting for Tenant models-as-a-service/default-tenant to be deleted testing.go:1913: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test --- FAIL: TestOdhOperator/components/group_4/modelsasservice/Validate_Tenant_deleted_on_disable (430.16s) === RUN TestOdhOperator/components/group_4/modelsasservice === PAUSE TestOdhOperator/components/group_4/modelsasservice === CONT TestOdhOperator/components/group_4/modelsasservice 2026-07-19T13:46:07Z INFO Cluster config {"Operator Namespace": "redhat-ods-operator", "Application Namespace": "redhat-ods-applications", "Release": {"name":"OpenShift AI Self-Managed","version":"3.5.0-ea.2"}, "Cluster": {"type":"OpenShift","version":"4.21.19"}} controller_test.go:753: Creating MaaS PostgreSQL instance in namespace redhat-ods-applications controller_test.go:753: MaaS PostgreSQL instance and maas-db-config secret created in namespace redhat-ods-applications controller_test.go:753: Creating MaaS GatewayClass and Gateway: openshift-ingress/maas-default-gateway controller_test.go:753: Using hostname=maas.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com, certSecret=2r2qdb90ltvhjc43lp6iqc6aqb7bgjda-primary-cert-bundle-secret controller_test.go:753: MaaS Gateway openshift-ingress/maas-default-gateway created with HTTP+HTTPS listeners test_context_test.go:615: Timed out after 420.009s. The function passed to Eventually failed at /workspace/tests/e2e/test_context_test.go:1824 with: Expected resource list 'models-as-a-service/default-tenant' of kind 'Tenant' to be empty, but it contains resources. Expected <[]unstructured.Unstructured | len:1, cap:1>: [ { Object: { "kind": <string>"Tenant", "metadata": <map[string]interface {} | len:8>{ "uid": <string>"da1c2e5e-d87b-4e62-8295-04527596c357", "creationTimestamp": <string>"2026-07-19T13:47:07Z", "generation": <int64>1, "managedFields": <[]interface {} | len:2, cap:2>[ <map[string]interface {} | len:6>{ "fieldsType": <string>"FieldsV1", "fieldsV1": <map[string]interface {} | len:2>{ "f:metadata": <map[string]interface {} | len:1>{ "f:ownerReferences": <map[string]interface {} | len:2>{ ".": <map[string]interface {} | len:0>{}, "k:{\"uid\":\"09fd5b38-44f4-4a90-b95a-a18ca41c8263\"}": <map[string]interface {} | len:0>{}, }, }, "f:spec": <map[string]interface {} | len:2>{ ".": <map[string]interface {} | len:0>{}, "f:gatewayRef": <map[string]interface {} | len:3>{ ".": <map[string]interface {} | len:0>{}, "f:name": <map[string]interface {} | len:0>{}, "f:namespace": <map[string]interface {} | len:0>{}, }, }, }, "manager": <string>"manager", "operation": <string>"Update", "time": <string>"2026-07-19T13:47:07Z", "apiVersion": <string>"maas.opendatahub.io/v1alpha1", }, <map[string]interface {} | len:7>{ "fieldsType": <string>"FieldsV1", "fieldsV1": <map[string]interface {} | len:1>{ "f:status": <map[string]interface {} | len:3>{ ".": <map[string]interface {} | len:0>{}, "f:conditions": <map[string]interface {} | len:0>{}, "f:phase": <map[string]interface {} | len:0>{}, }, }, "manager": <string>"manager", "operation": <string>"Update", "subresource": <string>"status", "time": <string>"2026-07-19T13:47:31Z", "apiVersion": <string>"maas.opendatahub.io/v1alpha1", }, ], "name": <string>"default-tenant", "namespace": <string>"models-as-a-service", "ownerReferences": <[]interface {} | len:1, cap:1>[ <map[string]interface {} | len:4>{ "apiVersion": <string>"maas.opendatahub.io/v1alpha1", "kind": <string>"Config", "name": <string>"default", "uid": <string>"09fd5b38-44f4-4a90-b95a-a18ca41c8263", }, ], "resourceVersion": <string>"39692792", }, "spec": <map[string]interface {} | len:1>{ "gatewayRef": <map[string]interface {} | len:2>{ "name": <string>"maas-default-gateway", "namespace": <string>"openshift-ingress", }, }, "status": <map[string]interface {} | len:2>{ "conditions": <[]interface {} | len:5, cap:8>[ <map[string]interface {} | len:6>{ "type": <string>"DependenciesAvailable", "lastTransitionTime": <string>"2026-07-19T13:47:10Z", "message": <string>"AuthConfig CRD (Kuadrant) is available on the cluster", "observedGeneration": <int64>1, "reason": <string>"DependenciesMet", "status": <string>"True", }, <map[string]interface {} | len:6>{ "reason": <string>"PrerequisitesMet", "status": <string>"True", "type": <string>"MaaSPrerequisitesAvailable", "lastTransitionTime": <string>"2026-07-19T13:47:10Z", "message": <string>"All prerequisites are satisfied", "observedGeneration": <int64>1, }, <map[string]interface {} | len:6>{ "lastTransitionTime": <string>"2026-07-19T13:47:10Z", "message": <string>"All prerequisites are satisfied", "observedGeneration": <int64>1, "reason": <string>"PrerequisitesMet", "status": <string>"False", "type": <string>"Degraded", }, <map[string]interface {} | len:6>{ "type": <string>"DeploymentsAvailable", "lastTransitionTime": <string>"2026-07-19T13:47:31Z", "message": <string>"maas-api deployment is available", "observedGeneration": <int64>1, "reason": <string>"DeploymentsReady", "status": <string>"True", }, <map[string]interface {} | len:6>{ "type": <string>"Ready", "lastTransitionTime": <string>"2026-07-19T13:47:31Z", "message": <string>"MaaS platform manifests applied and maas-api deployment is available", "observedGeneration": <int64>1, "reason": <string>"Reconciled", "status": <string>"True", }, ], "phase": <string>"Active", }, "apiVersion": <string>"maas.opendatahub.io/v1alpha1", }, }, ] to be empty --- FAIL: TestOdhOperator/components/group_4/modelsasservice (521.43s) === RUN TestOdhOperator/components/group_4 --- FAIL: TestOdhOperator/components/group_4 (0.00s) === RUN TestOdhOperator/components 2026/07/19 13:41:58 TEST FAILURE DETECTED: group 2 2026/07/19 13:41:58 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:group 2) === 2026/07/19 13:41:58 === CLUSTER STATE === 2026/07/19 13:41:58 Node: ip-10-0-1-227.us-east-2.compute.internal 2026/07/19 13:41:58 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/19 13:41:58 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/19 13:41:58 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/19 13:41:58 Condition Ready: True - kubelet is posting ready status 2026/07/19 13:41:58 Resources: 15500m CPU, 63871016Ki memory allocatable (of 16000m CPU, 65021992Ki memory capacity) 2026/07/19 13:41:58 Node: ip-10-0-1-23.us-east-2.compute.internal 2026/07/19 13:41:58 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/19 13:41:58 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/19 13:41:58 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/19 13:41:58 Condition Ready: True - kubelet is posting ready status 2026/07/19 13:41:58 Resources: 15500m CPU, 63182900Ki memory allocatable (of 16000m CPU, 64333876Ki memory capacity) 2026/07/19 13:41:58 Node: ip-10-0-1-45.us-east-2.compute.internal 2026/07/19 13:41:58 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/19 13:41:58 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/19 13:41:58 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/19 13:41:58 Condition Ready: True - kubelet is posting ready status 2026/07/19 13:41:58 Resources: 15500m CPU, 63871012Ki memory allocatable (of 16000m CPU, 65021988Ki memory capacity) 2026/07/19 13:41:58 === DEPLOYMENTS === 2026/07/19 13:41:58 Namespace: redhat-ods-applications 2026/07/19 13:41:58 Deployment maas-api: 1/1 (READY) 2026/07/19 13:41:58 Deployment maas-controller: 1/1 (READY) 2026/07/19 13:41:58 Deployment mlflow-operator-controller-manager: 1/1 (READY) 2026/07/19 13:41:58 Deployment trustyai-service-operator-controller-manager: 1/1 (READY) 2026/07/19 13:41:58 Namespace: kube-system 2026/07/19 13:41:58 === PODS === 2026/07/19 13:41:58 Namespace: redhat-ods-applications 2026/07/19 13:41:58 Pod maas-api-key-cleanup-29736435-89msd: Succeeded 2026/07/19 13:41:58 Container cleanup: ready=false, terminated: Completed (exit 0) 2026/07/19 13:41:58 Namespace kube-system: no problematic pods found 2026/07/19 13:41:58 === LOGS (CURRENT) for container cleanup === 2026/07/19 13:41:58 {"deletedCount":0,"message":"Successfully deleted 0 expired ephemeral key(s)"} 2026/07/19 13:41:58 === END LOGS === 2026/07/19 13:41:58 === OPERATOR STATUS === 2026/07/19 13:41:58 Operator deployment: 3/3 ready 2026/07/19 13:41:58 === DSCI STATUS === 2026/07/19 13:41:58 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/19 13:41:58 === DSC STATUS === 2026/07/19 13:41:58 Failed to collect DSC data: default-dsc unhealthy conditions: ModelsAsServiceReady=False 2026/07/19 13:41:58 === RECENT EVENTS === 2026/07/19 13:41:58 13:41:52 WARN Pod/maas-controller-75bbfd978-cxzxp: Unhealthy - Readiness probe failed: HTTP probe failed with statuscode: 500 2026/07/19 13:41:58 13:41:49 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/19 13:41:58 13:41:49 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/19 13:41:58 13:41:49 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-drfx2: Killing - Stopping container manager 2026/07/19 13:41:58 13:41:49 INFO Pod/odh-model-controller-65fdb9c5c-ss28z: Killing - Stopping container manager 2026/07/19 13:41:58 13:41:49 INFO Pod/model-serving-api-5679d6b4f-fcpz8: Killing - Stopping container server 2026/07/19 13:41:58 13:41:49 INFO Pod/model-registry-operator-controller-manager-74db7cdccd-j74np: Killing - Stopping container manager 2026/07/19 13:41:58 13:41:49 INFO Pod/workload-variant-autoscaler-controller-manager-7f5b9cd77b-zjn8m: Killing - Stopping container manager 2026/07/19 13:41:58 13:41:49 INFO Pod/kserve-controller-manager-dd9f6cf9-prgcw: Killing - Stopping container manager 2026/07/19 13:41:58 13:41:48 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/19 13:41:58 13:41:48 WARN ServiceMonitor/workload-variant-autoscaler-controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/workload-variant-autoscaler-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: bearerTokenSecret: [REDACTED] to get secret "workload-variant-autoscaler-controller-manager-token": secrets "workload-variant-autoscaler-controller-manager-token" not found 2026/07/19 13:41:58 13:41:34 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-65fdb9c5c-ss28z_59c6e2c8-3c66-43d1-a61f-f1be31a3507f became leader 2026/07/19 13:41:58 13:41:33 INFO Pod/workload-variant-autoscaler-controller-manager-7f5b9cd77b-zjn8m: AddedInterface - Add eth0 [10.131.1.155/23] from ovn-kubernetes 2026/07/19 13:41:58 13:41:33 INFO Pod/workload-variant-autoscaler-controller-manager-7f5b9cd77b-zjn8m: Pulled - Container image "quay.io/rhoai/odh-workload-variant-autoscaler-controller-rhel9@sha256:042a83d343bb9733c322eb22eeb445d8991365a2184ace45556acb7b110b6a83" already present on machine 2026/07/19 13:41:58 13:41:33 INFO Pod/workload-variant-autoscaler-controller-manager-7f5b9cd77b-zjn8m: Created - Created container: manager 2026/07/19 13:41:58 13:41:33 INFO Pod/workload-variant-autoscaler-controller-manager-7f5b9cd77b-zjn8m: Started - Started container manager 2026/07/19 13:41:58 13:41:33 INFO Lease/72dd1cf1.llm-d.ai: LeaderElection - workload-variant-autoscaler-controller-manager-7f5b9cd77b-zjn8m_a2d2b039-c326-4c55-a867-3fbadfd4fdec became leader 2026/07/19 13:41:58 13:41:32 INFO Pod/workload-variant-autoscaler-controller-manager-7f5b9cd77b-zjn8m: Scheduled - Successfully assigned redhat-ods-applications/workload-variant-autoscaler-controller-manager-7f5b9cd77b-zjn8m to ip-10-0-1-227.us-east-2.compute.internal 2026/07/19 13:41:58 13:41:32 INFO Pod/workload-variant-autoscaler-controller-manager-7f5b9cd77b-kn2zj: Killing - Stopping container manager 2026/07/19 13:41:58 13:41:32 INFO Deployment/workload-variant-autoscaler-controller-manager: ScalingReplicaSet - Scaled up replica set workload-variant-autoscaler-controller-manager-7f5b9cd77b from 0 to 1 2026/07/19 13:41:58 13:41:32 INFO ReplicaSet/workload-variant-autoscaler-controller-manager-7f5b9cd77b: SuccessfulCreate - Created pod: workload-variant-autoscaler-controller-manager-7f5b9cd77b-zjn8m 2026/07/19 13:41:58 13:41:32 INFO Pod/workload-variant-autoscaler-controller-manager-7f5b9cd77b-6d2cm: Killing - Stopping container manager 2026/07/19 13:41:58 13:41:18 INFO Pod/odh-model-controller-65fdb9c5c-ss28z: Created - Created container: manager 2026/07/19 13:41:58 13:41:18 INFO Pod/odh-model-controller-65fdb9c5c-ss28z: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/19 13:41:58 13:41:18 INFO Pod/odh-model-controller-65fdb9c5c-ss28z: AddedInterface - Add eth0 [10.131.1.154/23] from ovn-kubernetes 2026/07/19 13:41:58 13:41:18 INFO Pod/odh-model-controller-65fdb9c5c-ss28z: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 517ms (517ms including waiting). Image size: 238959091 bytes. 2026/07/19 13:41:58 13:41:18 INFO Pod/odh-model-controller-65fdb9c5c-ss28z: Started - Started container manager 2026/07/19 13:41:58 13:41:17 INFO Pod/odh-model-controller-65fdb9c5c-ss28z: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-65fdb9c5c-ss28z to ip-10-0-1-227.us-east-2.compute.internal 2026/07/19 13:41:58 13:41:17 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-65fdb9c5c from 0 to 1 2026/07/19 13:41:58 13:41:17 INFO ReplicaSet/odh-model-controller-65fdb9c5c: SuccessfulCreate - Created pod: odh-model-controller-65fdb9c5c-ss28z 2026/07/19 13:41:58 13:41:17 INFO Pod/odh-model-controller-65fdb9c5c-66bfc: Killing - Stopping container manager 2026/07/19 13:41:58 13:41:17 INFO Pod/odh-model-controller-65fdb9c5c-w2xwf: Killing - Stopping container manager 2026/07/19 13:41:58 13:41:03 INFO Pod/model-serving-api-5679d6b4f-fcpz8: Created - Created container: server 2026/07/19 13:41:58 13:41:03 INFO Pod/model-serving-api-5679d6b4f-fcpz8: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/19 13:41:58 13:41:03 INFO Pod/model-serving-api-5679d6b4f-fcpz8: AddedInterface - Add eth0 [10.131.1.153/23] from ovn-kubernetes 2026/07/19 13:41:58 13:41:03 INFO Pod/model-serving-api-5679d6b4f-fcpz8: Started - Started container server 2026/07/19 13:41:58 13:41:02 INFO Pod/model-serving-api-5679d6b4f-fcpz8: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-5679d6b4f-fcpz8 to ip-10-0-1-227.us-east-2.compute.internal 2026/07/19 13:41:58 13:41:02 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-5679d6b4f from 0 to 1 2026/07/19 13:41:58 13:41:02 INFO ReplicaSet/model-serving-api-5679d6b4f: SuccessfulCreate - Created pod: model-serving-api-5679d6b4f-fcpz8 2026/07/19 13:41:58 13:41:02 INFO Pod/model-serving-api-5679d6b4f-zkzgh: Killing - Stopping container server 2026/07/19 13:41:58 13:41:02 INFO Pod/model-serving-api-5679d6b4f-sw6zb: Killing - Stopping container server 2026/07/19 13:41:58 13:40:02 INFO Pod/workload-variant-autoscaler-controller-manager-7f5b9cd77b-6d2cm: AddedInterface - Add eth0 [10.131.1.152/23] from ovn-kubernetes 2026/07/19 13:41:58 13:40:02 INFO Pod/workload-variant-autoscaler-controller-manager-7f5b9cd77b-6d2cm: Pulled - Container image "quay.io/rhoai/odh-workload-variant-autoscaler-controller-rhel9@sha256:042a83d343bb9733c322eb22eeb445d8991365a2184ace45556acb7b110b6a83" already present on machine 2026/07/19 13:41:58 13:40:02 INFO Pod/workload-variant-autoscaler-controller-manager-7f5b9cd77b-6d2cm: Created - Created container: manager 2026/07/19 13:41:58 13:40:02 INFO Pod/workload-variant-autoscaler-controller-manager-7f5b9cd77b-6d2cm: Started - Started container manager 2026/07/19 13:41:58 13:40:01 INFO Pod/workload-variant-autoscaler-controller-manager-7f5b9cd77b-6d2cm: Scheduled - Successfully assigned redhat-ods-applications/workload-variant-autoscaler-controller-manager-7f5b9cd77b-6d2cm to ip-10-0-1-227.us-east-2.compute.internal 2026/07/19 13:41:58 13:40:01 INFO ReplicaSet/workload-variant-autoscaler-controller-manager-7f5b9cd77b: SuccessfulCreate - Created pod: workload-variant-autoscaler-controller-manager-7f5b9cd77b-6d2cm 2026/07/19 13:41:58 13:40:01 INFO Deployment/workload-variant-autoscaler-controller-manager: ScalingReplicaSet - Scaled up replica set workload-variant-autoscaler-controller-manager-7f5b9cd77b from 1 to 2 2026/07/19 13:41:58 13:40:01 INFO Lease/72dd1cf1.llm-d.ai: LeaderElection - workload-variant-autoscaler-controller-manager-7f5b9cd77b-kn2zj_0ab0137a-ee23-43f6-9e5a-8f78dea7b318 became leader 2026/07/19 13:41:58 13:39:43 INFO Pod/odh-model-controller-65fdb9c5c-w2xwf: Created - Created container: manager 2026/07/19 13:41:58 13:39:43 INFO Pod/odh-model-controller-65fdb9c5c-w2xwf: Started - Started container manager 2026/07/19 13:41:58 13:39:42 INFO Pod/odh-model-controller-65fdb9c5c-w2xwf: AddedInterface - Add eth0 [10.131.1.150/23] from ovn-kubernetes 2026/07/19 13:41:58 13:39:42 INFO Pod/odh-model-controller-65fdb9c5c-w2xwf: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 567ms (567ms including waiting). Image size: 238959091 bytes. 2026/07/19 13:41:58 13:39:42 INFO Pod/odh-model-controller-65fdb9c5c-w2xwf: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/19 13:41:58 13:39:41 INFO Pod/odh-model-controller-65fdb9c5c-w2xwf: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-65fdb9c5c-w2xwf to ip-10-0-1-227.us-east-2.compute.internal 2026/07/19 13:41:58 13:39:41 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-65fdb9c5c from 1 to 2 2026/07/19 13:41:58 13:39:41 INFO ReplicaSet/odh-model-controller-65fdb9c5c: SuccessfulCreate - Created pod: odh-model-controller-65fdb9c5c-w2xwf 2026/07/19 13:41:58 13:39:22 INFO Pod/model-serving-api-5679d6b4f-zkzgh: AddedInterface - Add eth0 [10.131.1.149/23] from ovn-kubernetes 2026/07/19 13:41:58 13:39:22 INFO Pod/model-serving-api-5679d6b4f-zkzgh: Started - Started container server 2026/07/19 13:41:58 13:39:22 INFO Pod/model-serving-api-5679d6b4f-zkzgh: Created - Created container: server 2026/07/19 13:41:58 13:39:22 INFO Pod/model-serving-api-5679d6b4f-zkzgh: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/19 13:41:58 13:39:21 INFO Pod/model-serving-api-5679d6b4f-zkzgh: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-5679d6b4f-zkzgh to ip-10-0-1-227.us-east-2.compute.internal 2026/07/19 13:41:58 13:39:21 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-5679d6b4f from 1 to 2 2026/07/19 13:41:58 13:39:21 INFO ReplicaSet/model-serving-api-5679d6b4f: SuccessfulCreate - Created pod: model-serving-api-5679d6b4f-zkzgh 2026/07/19 13:41:58 13:39:08 INFO Pod/maas-api-5b7784b768-fm6gl: Started - Started container maas-api 2026/07/19 13:41:58 13:39:08 INFO Pod/maas-api-5b7784b768-fm6gl: Created - Created container: maas-api 2026/07/19 13:41:58 13:39:08 INFO Pod/maas-api-5b7784b768-fm6gl: Pulled - Successfully pulled image "quay.io/rhoai/odh-maas-api-rhel9@sha256:9c87c8243770b9ebbff6ccab0dd663ccb402488cf5ee11f491ec90cf3b4686f6" in 538ms (538ms including waiting). Image size: 175018396 bytes. 2026/07/19 13:41:58 13:39:07 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/19 13:41:58 13:39:07 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/19 13:41:58 13:39:07 INFO Pod/maas-api-5b7784b768-fm6gl: Scheduled - Successfully assigned redhat-ods-applications/maas-api-5b7784b768-fm6gl to ip-10-0-1-23.us-east-2.compute.internal 2026/07/19 13:41:58 13:39:07 INFO Pod/maas-api-5b7784b768-fm6gl: Pulling - Pulling image "quay.io/rhoai/odh-maas-api-rhel9@sha256:9c87c8243770b9ebbff6ccab0dd663ccb402488cf5ee11f491ec90cf3b4686f6" 2026/07/19 13:41:58 13:39:07 INFO Pod/maas-api-5b7784b768-fm6gl: AddedInterface - Add eth0 [10.130.1.135/23] from ovn-kubernetes 2026/07/19 13:41:58 13:39:06 WARN ReplicaSet/maas-api-5b7784b768: FailedCreate - Error creating: pods "maas-api-5b7784b768-" is forbidden: error looking up service account redhat-ods-applications/maas-api: serviceaccount "maas-api" not found 2026/07/19 13:41:58 13:39:06 INFO ReplicaSet/maas-api-5b7784b768: SuccessfulCreate - Created pod: maas-api-5b7784b768-fm6gl 2026/07/19 13:41:58 13:39:06 INFO Deployment/maas-api: ScalingReplicaSet - Scaled up replica set maas-api-5b7784b768 from 0 to 1 2026/07/19 13:41:58 13:39:01 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/19 13:41:58 13:39:01 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/19 13:41:58 13:38:52 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-7b46b9fbbb-drfx2_f91e3e9a-f66c-42c6-81d1-2cf84c7a398a became leader 2026/07/19 13:41:58 13:38:51 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-dd9f6cf9-prgcw_5422d535-d981-40b9-9afc-74ba84e5af17 became leader 2026/07/19 13:41:58 13:38:42 INFO Pod/workload-variant-autoscaler-controller-manager-7f5b9cd77b-kn2zj: Started - Started container manager 2026/07/19 13:41:58 13:38:42 INFO Pod/workload-variant-autoscaler-controller-manager-7f5b9cd77b-kn2zj: Pulled - Container image "quay.io/rhoai/odh-workload-variant-autoscaler-controller-rhel9@sha256:042a83d343bb9733c322eb22eeb445d8991365a2184ace45556acb7b110b6a83" already present on machine 2026/07/19 13:41:58 13:38:42 INFO Pod/workload-variant-autoscaler-controller-manager-7f5b9cd77b-kn2zj: Created - Created container: manager 2026/07/19 13:41:58 13:38:42 INFO Pod/workload-variant-autoscaler-controller-manager-7f5b9cd77b-kn2zj: AddedInterface - Add eth0 [10.130.1.132/23] from ovn-kubernetes 2026/07/19 13:41:58 13:38:41 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/19 13:41:58 13:38:41 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/19 13:41:58 13:38:41 WARN ServiceMonitor/workload-variant-autoscaler-controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/workload-variant-autoscaler-controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: bearerTokenSecret: [REDACTED] to get secret "workload-variant-autoscaler-controller-manager-token": secrets "workload-variant-autoscaler-controller-manager-token" not found 2026/07/19 13:41:58 13:38:41 INFO Pod/workload-variant-autoscaler-controller-manager-7f5b9cd77b-kn2zj: Scheduled - Successfully assigned redhat-ods-applications/workload-variant-autoscaler-controller-manager-7f5b9cd77b-kn2zj to ip-10-0-1-23.us-east-2.compute.internal 2026/07/19 13:41:58 13:38:41 WARN Pod/workload-variant-autoscaler-controller-manager-7f5b9cd77b-kn2zj: FailedMount - MountVolume.SetUp failed for volume "epp-metrics-token" : secret "workload-variant-autoscaler-epp-metrics-token" not found 2026/07/19 13:41:58 13:38:41 INFO ReplicaSet/workload-variant-autoscaler-controller-manager-7f5b9cd77b: SuccessfulCreate - Created pod: workload-variant-autoscaler-controller-manager-7f5b9cd77b-kn2zj 2026/07/19 13:41:58 13:38:40 INFO Deployment/workload-variant-autoscaler-controller-manager: ScalingReplicaSet - Scaled up replica set workload-variant-autoscaler-controller-manager-7f5b9cd77b from 0 to 1 2026/07/19 13:41:58 13:38:40 INFO Lease/85f368d1.opendatahub.io: LeaderElection - model-registry-operator-controller-manager-74db7cdccd-j74np_6bcea5f9-a0d2-43cb-b358-1a3a384c6289 became leader 2026/07/19 13:41:58 13:38:40 WARN ReplicaSet/workload-variant-autoscaler-controller-manager-7f5b9cd77b: FailedCreate - Error creating: pods "workload-variant-autoscaler-controller-manager-7f5b9cd77b-" is forbidden: error looking up service account redhat-ods-applications/workload-variant-autoscaler-controller-manager: serviceaccount "workload-variant-autoscaler-controller-manager" not found 2026/07/19 13:41:58 13:38:38 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/19 13:41:58 13:38:38 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/19 13:41:58 13:38:38 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-65fdb9c5c-66bfc_e9c733f7-ec9f-44c8-be89-dde9b5051827 became leader 2026/07/19 13:41:58 13:38:37 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/19 13:41:58 13:38:37 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/19 13:41:58 13:38:36 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-drfx2: Started - Started container manager 2026/07/19 13:41:58 13:38:36 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-drfx2: Created - Created container: manager 2026/07/19 13:41:58 13:38:36 INFO Pod/kserve-controller-manager-dd9f6cf9-prgcw: Started - Started container manager 2026/07/19 13:41:58 13:38:35 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-drfx2: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" 2026/07/19 13:41:58 13:38:35 INFO Pod/kserve-controller-manager-dd9f6cf9-prgcw: AddedInterface - Add eth0 [10.130.1.130/23] from ovn-kubernetes 2026/07/19 13:41:58 13:38:35 INFO Pod/kserve-controller-manager-dd9f6cf9-prgcw: Created - Created container: manager 2026/07/19 13:41:58 13:38:35 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-drfx2: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" in 483ms (483ms including waiting). Image size: 236366599 bytes. 2026/07/19 13:41:58 13:38:35 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-drfx2: AddedInterface - Add eth0 [10.130.1.131/23] from ovn-kubernetes 2026/07/19 13:41:58 13:38:35 INFO Lease/maas-controller.models-as-a-service.opendatahub.io: LeaderElection - maas-controller-75bbfd978-cxzxp_4015f45f-88ba-43ed-b3a7-8923b7da746b became leader 2026/07/19 13:41:58 13:38:35 INFO Pod/kserve-controller-manager-dd9f6cf9-prgcw: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" in 511ms (511ms including waiting). Image size: 238435613 bytes. 2026/07/19 13:41:58 13:38:35 INFO Pod/kserve-controller-manager-dd9f6cf9-prgcw: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" 2026/07/19 13:41:58 13:38:34 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-drfx2: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-7b46b9fbbb-drfx2 to ip-10-0-1-23.us-east-2.compute.internal 2026/07/19 13:41:58 13:38:34 INFO Pod/kserve-controller-manager-dd9f6cf9-prgcw: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-dd9f6cf9-prgcw to ip-10-0-1-23.us-east-2.compute.internal 2026/07/19 13:41:58 13:38:34 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-dd9f6cf9 from 0 to 1 2026/07/19 13:41:58 13:38:34 INFO ReplicaSet/llmisvc-controller-manager-7b46b9fbbb: SuccessfulCreate - Created pod: llmisvc-controller-manager-7b46b9fbbb-drfx2 2026/07/19 13:41:58 13:38:34 INFO ReplicaSet/kserve-controller-manager-dd9f6cf9: SuccessfulCreate - Created pod: kserve-controller-manager-dd9f6cf9-prgcw 2026/07/19 13:41:58 13:38:34 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-7b46b9fbbb from 0 to 1 2026/07/19 13:41:58 13:38:21 INFO Pod/model-serving-api-5679d6b4f-sw6zb: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/19 13:41:58 13:38:21 INFO Pod/model-serving-api-5679d6b4f-sw6zb: Started - Started container server 2026/07/19 13:41:58 13:38:21 INFO Pod/model-serving-api-5679d6b4f-sw6zb: Created - Created container: server 2026/07/19 13:41:58 13:38:21 INFO Pod/model-serving-api-5679d6b4f-sw6zb: AddedInterface - Add eth0 [10.130.1.129/23] from ovn-kubernetes 2026/07/19 13:41:58 13:38:20 INFO Pod/model-serving-api-5679d6b4f-sw6zb: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-5679d6b4f-sw6zb to ip-10-0-1-23.us-east-2.compute.internal 2026/07/19 13:41:58 13:38:20 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/19 13:41:58 13:38:20 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/19 13:41:58 13:38:20 INFO Pod/model-registry-operator-controller-manager-74db7cdccd-j74np: Pulled - Container image "quay.io/rhoai/odh-model-registry-operator-rhel9@sha256:ebcec5c43468283ed3cbab50530f939166f8f3e8c9d306e91a8107d05d7fa9d4" already present on machine 2026/07/19 13:41:58 13:38:20 WARN ReplicaSet/model-serving-api-5679d6b4f: FailedCreate - Error creating: pods "model-serving-api-5679d6b4f-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/19 13:41:58 13:38:20 INFO Pod/model-registry-operator-controller-manager-74db7cdccd-j74np: Started - Started container manager 2026/07/19 13:41:58 13:38:20 INFO Pod/model-registry-operator-controller-manager-74db7cdccd-j74np: Created - Created container: manager 2026/07/19 13:41:58 13:38:20 INFO Pod/odh-model-controller-65fdb9c5c-66bfc: Started - Started container manager 2026/07/19 13:41:58 13:38:20 INFO Pod/odh-model-controller-65fdb9c5c-66bfc: Created - Created container: manager 2026/07/19 13:41:58 13:38:20 INFO Pod/odh-model-controller-65fdb9c5c-66bfc: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 669ms (669ms including waiting). Image size: 238959091 bytes. 2026/07/19 13:41:58 13:38:20 INFO Pod/model-registry-operator-controller-manager-74db7cdccd-j74np: AddedInterface - Add eth0 [10.130.1.128/23] from ovn-kubernetes 2026/07/19 13:41:58 13:38:20 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-5679d6b4f from 0 to 1 2026/07/19 13:41:58 13:38:20 INFO ReplicaSet/model-serving-api-5679d6b4f: SuccessfulCreate - Created pod: model-serving-api-5679d6b4f-sw6zb 2026/07/19 13:41:58 13:38:19 INFO Pod/model-registry-operator-controller-manager-74db7cdccd-j74np: Scheduled - Successfully assigned redhat-ods-applications/model-registry-operator-controller-manager-74db7cdccd-j74np to ip-10-0-1-23.us-east-2.compute.internal 2026/07/19 13:41:58 13:38:19 INFO Deployment/model-registry-operator-controller-manager: ScalingReplicaSet - Scaled up replica set model-registry-operator-controller-manager-74db7cdccd from 0 to 1 2026/07/19 13:41:58 13:38:19 INFO Pod/odh-model-controller-65fdb9c5c-66bfc: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/19 13:41:58 13:38:19 INFO Pod/odh-model-controller-65fdb9c5c-66bfc: AddedInterface - Add eth0 [10.130.1.127/23] from ovn-kubernetes 2026/07/19 13:41:58 13:38:19 WARN Pod/model-registry-operator-controller-manager-74db7cdccd-j74np: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "model-registry-operator-webhook-service" not found 2026/07/19 13:41:58 13:38:19 INFO ReplicaSet/model-registry-operator-controller-manager-74db7cdccd: SuccessfulCreate - Created pod: model-registry-operator-controller-manager-74db7cdccd-j74np 2026/07/19 13:41:58 13:38:19 INFO Pod/maas-controller-75bbfd978-cxzxp: Started - Started container manager 2026/07/19 13:41:58 13:38:19 INFO Pod/maas-controller-75bbfd978-cxzxp: Created - Created container: manager 2026/07/19 13:41:58 13:38:19 INFO Pod/maas-controller-75bbfd978-cxzxp: Pulled - Successfully pulled image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" in 537ms (537ms including waiting). Image size: 212866077 bytes. 2026/07/19 13:41:58 13:38:18 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/19 13:41:58 13:38:18 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/19 13:41:58 13:38:18 INFO Pod/odh-model-controller-65fdb9c5c-66bfc: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-65fdb9c5c-66bfc to ip-10-0-1-23.us-east-2.compute.internal 2026/07/19 13:41:58 13:38:18 INFO Pod/maas-controller-75bbfd978-cxzxp: Scheduled - Successfully assigned redhat-ods-applications/maas-controller-75bbfd978-cxzxp to ip-10-0-1-23.us-east-2.compute.internal 2026/07/19 13:41:58 13:38:18 INFO ReplicaSet/maas-controller-75bbfd978: SuccessfulCreate - Created pod: maas-controller-75bbfd978-cxzxp 2026/07/19 13:41:58 13:38:18 INFO Deployment/maas-controller: ScalingReplicaSet - Scaled up replica set maas-controller-75bbfd978 from 0 to 1 2026/07/19 13:41:58 13:38:18 INFO Pod/maas-controller-75bbfd978-cxzxp: Pulling - Pulling image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" 2026/07/19 13:41:58 13:38:18 INFO ReplicaSet/odh-model-controller-65fdb9c5c: SuccessfulCreate - Created pod: odh-model-controller-65fdb9c5c-66bfc 2026/07/19 13:41:58 13:38:18 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-65fdb9c5c from 0 to 1 2026/07/19 13:41:58 13:38:18 INFO Pod/maas-controller-75bbfd978-cxzxp: AddedInterface - Add eth0 [10.130.1.126/23] from ovn-kubernetes 2026/07/19 13:41:58 13:38:18 WARN Pod/odh-model-controller-65fdb9c5c-66bfc: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/19 13:41:58 13:38:08 INFO Pod/dashboard-redirect-84fcf4d86-sc9d5: Killing - Stopping container nginx 2026/07/19 13:41:58 13:38:08 INFO Pod/dashboard-redirect-84fcf4d86-5gjcb: Killing - Stopping container nginx 2026/07/19 13:41:58 13:37:36 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/19 13:41:58 13:37:36 INFO Pod/maas-controller-75bbfd978-cp5zp: Killing - Stopping container manager 2026/07/19 13:41:58 13:37:36 INFO Pod/model-serving-api-5679d6b4f-nl2mz: Killing - Stopping container server 2026/07/19 13:41:58 13:37:36 INFO Pod/odh-model-controller-bd4c5cbcb-s67pv: Killing - Stopping container manager 2026/07/19 13:41:58 13:37:36 INFO Pod/kserve-controller-manager-dd9f6cf9-5dn5s: Killing - Stopping container manager 2026/07/19 13:41:58 13:37:36 INFO Pod/maas-api-5b7784b768-vfd2v: Killing - Stopping container maas-api 2026/07/19 13:41:58 13:37:35 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/19 13:41:58 13:37:35 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/19 13:41:58 13:37:35 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-2gl2l: Killing - Stopping container manager 2026/07/19 13:41:58 13:37:26 INFO Pod/rhods-dashboard-84cdfcd5d8-4hw2g: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:137ad2806fdab58f8611063a64b6cbaa3f1cf160b6d40adea8e58eb1c5d1e548" in 487ms (487ms including waiting). Image size: 185092464 bytes. 2026/07/19 13:41:58 13:37:26 INFO Pod/rhods-dashboard-84cdfcd5d8-87jtj: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:137ad2806fdab58f8611063a64b6cbaa3f1cf160b6d40adea8e58eb1c5d1e548" in 483ms (483ms including waiting). Image size: 185092464 bytes. 2026/07/19 13:41:58 13:37:25 INFO Pod/rhods-dashboard-84cdfcd5d8-87jtj: Created - Created container: maas-ui 2026/07/19 13:41:58 13:37:25 INFO Pod/rhods-dashboard-84cdfcd5d8-87jtj: Started - Started container mlflow-ui 2026/07/19 13:41:58 13:37:25 INFO Pod/rhods-dashboard-84cdfcd5d8-4hw2g: Created - Created container: maas-ui 2026/07/19 13:41:58 13:37:25 INFO Pod/rhods-dashboard-84cdfcd5d8-4hw2g: Started - Started container maas-ui 2026/07/19 13:41:58 13:37:25 INFO Pod/rhods-dashboard-84cdfcd5d8-4hw2g: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:83f85b7a7d0b4bb9d2c1c2bd6c546ab3f9237cfc8c6820395d7d0b7f24f95410" 2026/07/19 13:41:58 13:37:25 INFO Pod/rhods-dashboard-84cdfcd5d8-4hw2g: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:83f85b7a7d0b4bb9d2c1c2bd6c546ab3f9237cfc8c6820395d7d0b7f24f95410" in 595ms (595ms including waiting). Image size: 177808591 bytes. 2026/07/19 13:41:58 13:37:25 INFO Pod/rhods-dashboard-84cdfcd5d8-4hw2g: Created - Created container: mlflow-ui 2026/07/19 13:41:58 13:37:25 INFO Pod/rhods-dashboard-84cdfcd5d8-4hw2g: Started - Started container mlflow-ui 2026/07/19 13:41:58 13:37:25 INFO Pod/rhods-dashboard-84cdfcd5d8-4hw2g: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:137ad2806fdab58f8611063a64b6cbaa3f1cf160b6d40adea8e58eb1c5d1e548" 2026/07/19 13:41:58 13:37:25 INFO Pod/rhods-dashboard-84cdfcd5d8-87jtj: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-maas-rhel9@sha256:c66d4116ea3689b73ea85fa75e451b643d3e18f5c52124c18e0d56ab368bf8ef" in 416ms (416ms including waiting). Image size: 187802205 bytes. 2026/07/19 13:41:58 13:37:25 INFO Pod/rhods-dashboard-84cdfcd5d8-4hw2g: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-maas-rhel9@sha256:c66d4116ea3689b73ea85fa75e451b643d3e18f5c52124c18e0d56ab368bf8ef" in 549ms (549ms including waiting). Image size: 187802205 bytes. 2026/07/19 13:41:58 13:37:25 INFO Pod/rhods-dashboard-84cdfcd5d8-87jtj: Started - Started container maas-ui 2026/07/19 13:41:58 13:37:25 INFO Pod/rhods-dashboard-84cdfcd5d8-87jtj: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-eval-hub-rhel9@sha256:137ad2806fdab58f8611063a64b6cbaa3f1cf160b6d40adea8e58eb1c5d1e548" 2026/07/19 13:41:58 13:37:25 INFO Pod/rhods-dashboard-84cdfcd5d8-87jtj: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:83f85b7a7d0b4bb9d2c1c2bd6c546ab3f9237cfc8c6820395d7d0b7f24f95410" 2026/07/19 13:41:58 13:37:25 INFO Pod/rhods-dashboard-84cdfcd5d8-87jtj: Created - Created container: mlflow-ui 2026/07/19 13:41:58 13:37:25 INFO Pod/rhods-dashboard-84cdfcd5d8-87jtj: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-mlflow-rhel9@sha256:83f85b7a7d0b4bb9d2c1c2bd6c546ab3f9237cfc8c6820395d7d0b7f24f95410" in 479ms (479ms including waiting). Image size: 177808591 bytes. 2026/07/19 13:41:58 13:37:24 INFO Pod/rhods-dashboard-84cdfcd5d8-87jtj: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-maas-rhel9@sha256:c66d4116ea3689b73ea85fa75e451b643d3e18f5c52124c18e0d56ab368bf8ef" 2026/07/19 13:41:58 13:37:24 INFO Pod/rhods-dashboard-84cdfcd5d8-4hw2g: Created - Created container: gen-ai-ui 2026/07/19 13:41:58 13:37:24 INFO Pod/rhods-dashboard-84cdfcd5d8-4hw2g: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:a32d767a12da1752d119f9901982622f76cf1e681fdbcda9e00d62b91af8538d" in 564ms (564ms including waiting). Image size: 241242949 bytes. 2026/07/19 13:41:58 13:37:24 INFO Pod/rhods-dashboard-84cdfcd5d8-4hw2g: Started - Started container gen-ai-ui 2026/07/19 13:41:58 13:37:24 INFO Pod/rhods-dashboard-84cdfcd5d8-4hw2g: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-maas-rhel9@sha256:c66d4116ea3689b73ea85fa75e451b643d3e18f5c52124c18e0d56ab368bf8ef" 2026/07/19 13:41:58 13:37:24 INFO Pod/rhods-dashboard-84cdfcd5d8-87jtj: Created - Created container: model-registry-ui 2026/07/19 13:41:58 13:37:24 INFO Pod/rhods-dashboard-84cdfcd5d8-87jtj: Started - Started container model-registry-ui 2026/07/19 13:41:58 13:37:24 INFO Pod/rhods-dashboard-84cdfcd5d8-87jtj: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:a32d767a12da1752d119f9901982622f76cf1e681fdbcda9e00d62b91af8538d" 2026/07/19 13:41:58 13:37:24 INFO Pod/rhods-dashboard-84cdfcd5d8-87jtj: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:a32d767a12da1752d119f9901982622f76cf1e681fdbcda9e00d62b91af8538d" in 407ms (407ms including waiting). Image size: 241242949 bytes. 2026/07/19 13:41:58 13:37:24 INFO Pod/rhods-dashboard-84cdfcd5d8-87jtj: Created - Created container: gen-ai-ui 2026/07/19 13:41:58 13:37:24 INFO Pod/rhods-dashboard-84cdfcd5d8-87jtj: Started - Started container gen-ai-ui 2026/07/19 13:41:58 13:37:23 INFO Pod/rhods-dashboard-84cdfcd5d8-87jtj: Started - Started container kube-rbac-proxy 2026/07/19 13:41:58 13:37:23 INFO Pod/rhods-dashboard-84cdfcd5d8-4hw2g: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:7114e8975a11bc2146cd6b3c0760d9d5b506c94d83d5e1f611a920fd7b15a97c" in 529ms (529ms including waiting). Image size: 182438697 bytes. 2026/07/19 13:41:58 13:37:23 INFO Pod/rhods-dashboard-84cdfcd5d8-87jtj: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:7114e8975a11bc2146cd6b3c0760d9d5b506c94d83d5e1f611a920fd7b15a97c" 2026/07/19 13:41:58 13:37:23 INFO Pod/rhods-dashboard-84cdfcd5d8-87jtj: Created - Created container: kube-rbac-proxy 2026/07/19 13:41:58 13:37:23 INFO Pod/rhods-dashboard-84cdfcd5d8-87jtj: Pulled - Container image "quay.io/rhoai/odh-kube-rbac-proxy-rhel9@sha256:1e614e46799fd020ac18d95c06a463421154ce86e59ccb3fa6007e66fed9910e" already present on machine 2026/07/19 13:41:58 13:37:23 INFO Pod/rhods-dashboard-84cdfcd5d8-87jtj: Started - Started container rhods-dashboard 2026/07/19 13:41:58 13:37:23 INFO Pod/rhods-dashboard-84cdfcd5d8-87jtj: Created - Created container: rhods-dashboard 2026/07/19 13:41:58 13:37:23 INFO Pod/rhods-dashboard-84cdfcd5d8-87jtj: Pulled - Successfully pulled image "quay.io/rhoai/odh-dashboard-rhel9@sha256:91277c86a8c3cac87e10664137e4d827f09e904d3cdbed0fed837ebe2a9a0022" in 602ms (602ms including waiting). Image size: 799822033 bytes. 2026/07/19 13:41:58 13:37:23 INFO Pod/rhods-dashboard-84cdfcd5d8-4hw2g: Started - Started container rhods-dashboard 2026/07/19 13:41:58 13:37:23 INFO Pod/rhods-dashboard-84cdfcd5d8-4hw2g: Pulled - Container image "quay.io/rhoai/odh-kube-rbac-proxy-rhel9@sha256:1e614e46799fd020ac18d95c06a463421154ce86e59ccb3fa6007e66fed9910e" already present on machine 2026/07/19 13:41:58 13:37:23 INFO Pod/rhods-dashboard-84cdfcd5d8-4hw2g: Created - Created container: kube-rbac-proxy 2026/07/19 13:41:58 13:37:23 INFO Pod/rhods-dashboard-84cdfcd5d8-87jtj: Pulled - Successfully pulled image "quay.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:7114e8975a11bc2146cd6b3c0760d9d5b506c94d83d5e1f611a920fd7b15a97c" in 487ms (487ms including waiting). Image size: 182438697 bytes. 2026/07/19 13:41:58 13:37:23 INFO Pod/rhods-dashboard-84cdfcd5d8-4hw2g: Started - Started container kube-rbac-proxy 2026/07/19 13:41:58 13:37:23 INFO Pod/rhods-dashboard-84cdfcd5d8-4hw2g: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-model-registry-rhel9@sha256:7114e8975a11bc2146cd6b3c0760d9d5b506c94d83d5e1f611a920fd7b15a97c" 2026/07/19 13:41:58 13:37:23 INFO Pod/rhods-dashboard-84cdfcd5d8-4hw2g: Pulled - Successfully pulled image "quay.io/rhoai/odh-dashboard-rhel9@sha256:91277c86a8c3cac87e10664137e4d827f09e904d3cdbed0fed837ebe2a9a0022" in 451ms (451ms including waiting). Image size: 799822033 bytes. 2026/07/19 13:41:58 13:37:23 INFO Pod/rhods-dashboard-84cdfcd5d8-4hw2g: Created - Created container: rhods-dashboard 2026/07/19 13:41:58 13:37:23 INFO Pod/rhods-dashboard-84cdfcd5d8-4hw2g: Pulling - Pulling image "quay.io/rhoai/odh-mod-arch-gen-ai-rhel9@sha256:a32d767a12da1752d119f9901982622f76cf1e681fdbcda9e00d62b91af8538d" 2026/07/19 13:41:58 13:37:23 INFO Pod/rhods-dashboard-84cdfcd5d8-4hw2g: Started - Started container model-registry-ui 2026/07/19 13:41:58 13:37:23 INFO Pod/rhods-dashboard-84cdfcd5d8-4hw2g: Created - Created container: model-registry-ui 2026/07/19 13:41:58 13:37:22 INFO Pod/rhods-dashboard-84cdfcd5d8-4hw2g: Pulling - Pulling image "quay.io/rhoai/odh-dashboard-rhel9@sha256:91277c86a8c3cac87e10664137e4d827f09e904d3cdbed0fed837ebe2a9a0022" 2026/07/19 13:41:58 13:37:22 INFO Pod/rhods-dashboard-84cdfcd5d8-4hw2g: AddedInterface - Add eth0 [10.131.1.146/23] from ovn-kubernetes 2026/07/19 13:41:58 13:37:22 INFO Pod/rhods-dashboard-84cdfcd5d8-87jtj: AddedInterface - Add eth0 [10.130.1.125/23] from ovn-kubernetes 2026/07/19 13:41:58 13:37:22 INFO Pod/rhods-dashboard-84cdfcd5d8-87jtj: Pulling - Pulling image "quay.io/rhoai/odh-dashboard-rhel9@sha256:91277c86a8c3cac87e10664137e4d827f09e904d3cdbed0fed837ebe2a9a0022" 2026/07/19 13:41:58 13:37:21 INFO Pod/rhods-dashboard-84cdfcd5d8-4hw2g: Scheduled - Successfully assigned redhat-ods-applications/rhods-dashboard-84cdfcd5d8-4hw2g to ip-10-0-1-227.us-east-2.compute.internal 2026/07/19 13:41:58 13:37:21 INFO Pod/rhods-dashboard-84cdfcd5d8-87jtj: Scheduled - Successfully assigned redhat-ods-applications/rhods-dashboard-84cdfcd5d8-87jtj to ip-10-0-1-23.us-east-2.compute.internal 2026/07/19 13:41:58 13:37:21 INFO ReplicaSet/rhods-dashboard-84cdfcd5d8: SuccessfulCreate - Created pod: rhods-dashboard-84cdfcd5d8-4hw2g 2026/07/19 13:41:58 13:37:21 INFO Deployment/rhods-dashboard: ScalingReplicaSet - Scaled up replica set rhods-dashboard-84cdfcd5d8 from 0 to 2 2026/07/19 13:41:58 13:37:21 INFO ReplicaSet/rhods-dashboard-84cdfcd5d8: SuccessfulCreate - Created pod: rhods-dashboard-84cdfcd5d8-87jtj 2026/07/19 13:41:58 13:37:18 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-7b46b9fbbb-2gl2l_ed485908-a303-4e13-9898-326903b9dcbc became leader 2026/07/19 13:41:58 13:37:10 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/19 13:41:58 13:37:10 WARN ServiceMonitor/controller-manager-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/19 13:41:58 13:37:10 INFO Pod/spark-operator-controller-5dd7d48f87-z5dpb: Killing - Stopping container controller 2026/07/19 13:41:58 13:37:10 INFO Pod/spark-operator-webhook-5f8768dbfd-7vzbq: Killing - Stopping container webhook 2026/07/19 13:41:58 13:37:01 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-2gl2l: Created - Created container: manager 2026/07/19 13:41:58 13:37:01 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-2gl2l: Started - Started container manager 2026/07/19 13:41:58 13:37:00 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-2gl2l: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" 2026/07/19 13:41:58 13:37:00 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-2gl2l: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" in 600ms (600ms including waiting). Image size: 236366599 bytes. 2026/07/19 13:41:58 13:37:00 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-2gl2l: AddedInterface - Add eth0 [10.131.1.145/23] from ovn-kubernetes 2026/07/19 13:41:58 13:36:59 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-2gl2l: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-7b46b9fbbb-2gl2l to ip-10-0-1-227.us-east-2.compute.internal 2026/07/19 13:41:58 13:36:59 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-5g7nr: Killing - Stopping container manager 2026/07/19 13:41:58 13:36:59 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-c8txb: Killing - Stopping container manager 2026/07/19 13:41:58 13:36:59 INFO ReplicaSet/llmisvc-controller-manager-7b46b9fbbb: SuccessfulCreate - Created pod: llmisvc-controller-manager-7b46b9fbbb-2gl2l 2026/07/19 13:41:58 13:36:59 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-7b46b9fbbb from 0 to 1 2026/07/19 13:41:58 === RESOURCE QUOTAS === 2026/07/19 13:41:58 No resource quotas found in monitored namespaces 2026/07/19 13:41:58 Classification for group 2: infrastructure/storage (code=1006, confidence=medium) 2026/07/19 13:41:58 Evidence: event Pod/workload-variant-autoscaler-controller-manager-7f5b9cd77b-kn2zj: FailedMount - MountVolume.SetUp failed for volume "epp-metrics-token" : secret "workload-variant-autoscaler-epp-metrics-token" not found FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"storage","error_code":1006,"evidence":["event Pod/workload-variant-autoscaler-controller-manager-7f5b9cd77b-kn2zj: FailedMount - MountVolume.SetUp failed for volume \"epp-metrics-token\" : secret \"workload-variant-autoscaler-epp-metrics-token\" not found"],"confidence":"medium"} 2026/07/19 13:41:58 Diagnostics complete for failed test: group 2 controller_test.go:753: Stopping: group 2 test failed. 2026/07/19 13:41:58 Circuit breaker: failure classified as infrastructure/storage (code=1006) 2026/07/19 13:41:58 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) 2026/07/19 13:46:06 Circuit breaker: success recorded, resetting failure counter (was 1) 2026/07/19 13:54:48 TEST FAILURE DETECTED: group 4 2026/07/19 13:54:48 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:group 4) === 2026/07/19 13:54:48 === CLUSTER STATE === 2026/07/19 13:54:48 Node: ip-10-0-1-227.us-east-2.compute.internal 2026/07/19 13:54:48 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/19 13:54:48 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/19 13:54:48 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/19 13:54:48 Condition Ready: True - kubelet is posting ready status 2026/07/19 13:54:48 Resources: 15500m CPU, 63871016Ki memory allocatable (of 16000m CPU, 65021992Ki memory capacity) 2026/07/19 13:54:48 Node: ip-10-0-1-23.us-east-2.compute.internal 2026/07/19 13:54:48 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/19 13:54:48 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/19 13:54:48 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/19 13:54:48 Condition Ready: True - kubelet is posting ready status 2026/07/19 13:54:48 Resources: 15500m CPU, 63182900Ki memory allocatable (of 16000m CPU, 64333876Ki memory capacity) 2026/07/19 13:54:48 Node: ip-10-0-1-45.us-east-2.compute.internal 2026/07/19 13:54:48 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/19 13:54:48 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/19 13:54:48 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/19 13:54:48 Condition Ready: True - kubelet is posting ready status 2026/07/19 13:54:48 Resources: 15500m CPU, 63871012Ki memory allocatable (of 16000m CPU, 65021988Ki memory capacity) 2026/07/19 13:54:48 === DEPLOYMENTS === 2026/07/19 13:54:48 Failed to collect deployment data: deployments not ready: redhat-ods-applications/maas-controller (0/1) 2026/07/19 13:54:48 === PODS === 2026/07/19 13:54:48 Failed to collect pod data: redhat-ods-applications/maas-controller-75bbfd978-nhkw7: container manager not ready 2026/07/19 13:54:48 === OPERATOR STATUS === 2026/07/19 13:54:48 Operator deployment: 3/3 ready 2026/07/19 13:54:48 === DSCI STATUS === 2026/07/19 13:54:48 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/19 13:54:48 === DSC STATUS === 2026/07/19 13:54:48 DSC default-dsc: 2026/07/19 13:54:48 Ready: True - 2026/07/19 13:54:48 ProvisioningSucceeded: True - 2026/07/19 13:54:48 ComponentsReady: True - 2026/07/19 13:54:48 AIPipelinesReady: False - Component ManagementState is set to Removed 2026/07/19 13:54:48 DashboardReady: False - Component ManagementState is set to Removed 2026/07/19 13:54:48 FeastOperatorReady: False - Component ManagementState is set to Removed 2026/07/19 13:54:48 KserveLLMInferenceServiceDependencies: True - 2026/07/19 13:54:48 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/19 13:54:48 KserveReady: True - 2026/07/19 13:54:48 KueueReady: True - 2026/07/19 13:54:48 LLM-D-WVADependencies: True - 2026/07/19 13:54:48 MLflowOperatorReady: False - Component ManagementState is set to Removed 2026/07/19 13:54:48 ModelControllerReady: True - 2026/07/19 13:54:48 ModelRegistryReady: False - Component ManagementState is set to Removed 2026/07/19 13:54:48 ModelsAsServiceReady: False - Component ManagementState is set to Removed 2026/07/19 13:54:48 OGXReady: False - Component ManagementState is set to Removed 2026/07/19 13:54:48 RayReady: False - Component ManagementState is set to Removed 2026/07/19 13:54:48 SparkOperatorReady: False - Component ManagementState is set to Removed 2026/07/19 13:54:48 TrainerReady: False - Component ManagementState is set to Removed 2026/07/19 13:54:48 TrainingOperatorReady: False - Component ManagementState is set to Removed 2026/07/19 13:54:48 TrustyAIReady: False - Component ManagementState is set to Removed 2026/07/19 13:54:48 WorkbenchesReady: False - Component ManagementState is set to Removed 2026/07/19 13:54:48 === RECENT EVENTS === 2026/07/19 13:54:48 13:53:26 INFO Pod/maas-controller-75bbfd978-nhkw7: Pulled - Successfully pulled image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" in 455ms (455ms including waiting). Image size: 212866077 bytes. 2026/07/19 13:54:48 13:53:25 INFO Pod/maas-controller-75bbfd978-nhkw7: Pulling - Pulling image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" 2026/07/19 13:54:48 13:52:58 WARN Pod/maas-controller-75bbfd978-nhkw7: FailedToRetrieveImagePullSecret - Unable to retrieve some image pull secrets (maas-controller-dockercfg-knq5p); attempting to pull the image may not succeed. 2026/07/19 13:54:48 13:50:42 INFO Pod/maas-controller-75bbfd978-nhkw7: Created - Created container: manager 2026/07/19 13:54:48 13:50:42 INFO Pod/maas-controller-75bbfd978-nhkw7: Started - Started container manager 2026/07/19 13:54:48 13:50:42 INFO Pod/maas-controller-75bbfd978-nhkw7: Pulled - Successfully pulled image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" in 540ms (540ms including waiting). Image size: 212866077 bytes. 2026/07/19 13:54:48 === RESOURCE QUOTAS === 2026/07/19 13:54:48 No resource quotas found in monitored namespaces 2026/07/19 13:54:48 Classification for group 4: infrastructure/pod-startup (code=1002, confidence=medium) 2026/07/19 13:54:48 Evidence: container maas-controller-75bbfd978-nhkw7/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-75bbfd978-nhkw7_redhat-ods-applications(12ea2cfb-9eba-4d2e-9b67-7c70aa2546b4) FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"pod-startup","error_code":1002,"evidence":["container maas-controller-75bbfd978-nhkw7/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-75bbfd978-nhkw7_redhat-ods-applications(12ea2cfb-9eba-4d2e-9b67-7c70aa2546b4)"],"confidence":"medium"} 2026/07/19 13:54:48 Diagnostics complete for failed test: group 4 controller_test.go:753: Stopping: group 4 test failed. 2026/07/19 13:54:48 Circuit breaker: failure classified as infrastructure/pod-startup (code=1002) 2026/07/19 13:54:48 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) 2026/07/19 13:55:59 Circuit breaker: success recorded, resetting failure counter (was 1) --- FAIL: TestOdhOperator/components (1433.99s) === RUN TestOdhOperator 2026/07/19 13:32:55 ARTIFACT_DIR not set, skipping periodic metrics collection 2026/07/19 13:57:00 TEST FAILURE DETECTED: components 2026/07/19 13:57:00 === RUNNING CLUSTER DIAGNOSTICS (triggered by test:components) === 2026/07/19 13:57:01 === CLUSTER STATE === 2026/07/19 13:57:01 Node: ip-10-0-1-227.us-east-2.compute.internal 2026/07/19 13:57:01 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/19 13:57:01 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/19 13:57:01 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/19 13:57:01 Condition Ready: True - kubelet is posting ready status 2026/07/19 13:57:01 Resources: 15500m CPU, 63871016Ki memory allocatable (of 16000m CPU, 65021992Ki memory capacity) 2026/07/19 13:57:01 Node: ip-10-0-1-23.us-east-2.compute.internal 2026/07/19 13:57:01 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/19 13:57:01 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/19 13:57:01 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/19 13:57:01 Condition Ready: True - kubelet is posting ready status 2026/07/19 13:57:01 Resources: 15500m CPU, 63182900Ki memory allocatable (of 16000m CPU, 64333876Ki memory capacity) 2026/07/19 13:57:01 Node: ip-10-0-1-45.us-east-2.compute.internal 2026/07/19 13:57:01 Condition MemoryPressure: False - kubelet has sufficient memory available 2026/07/19 13:57:01 Condition DiskPressure: False - kubelet has no disk pressure 2026/07/19 13:57:01 Condition PIDPressure: False - kubelet has sufficient PID available 2026/07/19 13:57:01 Condition Ready: True - kubelet is posting ready status 2026/07/19 13:57:01 Resources: 15500m CPU, 63871012Ki memory allocatable (of 16000m CPU, 65021988Ki memory capacity) 2026/07/19 13:57:01 === DEPLOYMENTS === 2026/07/19 13:57:01 Failed to collect deployment data: deployments not ready: redhat-ods-applications/maas-controller (0/1) 2026/07/19 13:57:01 === PODS === 2026/07/19 13:57:01 Failed to collect pod data: redhat-ods-applications/maas-controller-75bbfd978-nhkw7: container manager not ready 2026/07/19 13:57:01 === OPERATOR STATUS === 2026/07/19 13:57:01 Operator deployment: 3/3 ready 2026/07/19 13:57:01 === DSCI STATUS === 2026/07/19 13:57:01 Failed to collect DSCI data: default-dsci unhealthy conditions: MonitoringStackAvailable=False; TempoAvailable=False; OpenTelemetryCollectorAvailable=False; AlertingAvailable=False; ThanosQuerierAvailable=False; PersesAvailable=False; NodeMetricsEndpointAvailable=False 2026/07/19 13:57:01 === DSC STATUS === 2026/07/19 13:57:01 DSC default-dsc: 2026/07/19 13:57:01 Ready: True - 2026/07/19 13:57:01 ProvisioningSucceeded: True - 2026/07/19 13:57:01 ComponentsReady: True - 2026/07/19 13:57:01 AIPipelinesReady: False - Component ManagementState is set to Removed 2026/07/19 13:57:01 DashboardReady: False - Component ManagementState is set to Removed 2026/07/19 13:57:01 FeastOperatorReady: False - Component ManagementState is set to Removed 2026/07/19 13:57:01 KserveLLMInferenceServiceDependencies: True - 2026/07/19 13:57:01 KserveLLMInferenceServiceWideEPDependencies: True - 2026/07/19 13:57:01 KserveReady: True - 2026/07/19 13:57:01 KueueReady: True - 2026/07/19 13:57:01 LLM-D-WVADependencies: True - 2026/07/19 13:57:01 MLflowOperatorReady: False - Component ManagementState is set to Removed 2026/07/19 13:57:01 ModelControllerReady: True - 2026/07/19 13:57:01 ModelRegistryReady: False - Component ManagementState is set to Removed 2026/07/19 13:57:01 ModelsAsServiceReady: False - Component ManagementState is set to Removed 2026/07/19 13:57:01 OGXReady: False - Component ManagementState is set to Removed 2026/07/19 13:57:01 RayReady: False - Component ManagementState is set to Removed 2026/07/19 13:57:01 SparkOperatorReady: False - Component ManagementState is set to Removed 2026/07/19 13:57:01 TrainerReady: True - 2026/07/19 13:57:01 TrainingOperatorReady: False - Component ManagementState is set to Removed 2026/07/19 13:57:01 TrustyAIReady: False - Component ManagementState is set to Removed 2026/07/19 13:57:01 WorkbenchesReady: False - Component ManagementState is set to Removed 2026/07/19 13:57:01 === RECENT EVENTS === 2026/07/19 13:57:01 13:56:44 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-dd9f6cf9-sqkcl_9a46aca9-501a-4bad-8547-2b557e323d4c became leader 2026/07/19 13:57:01 13:56:44 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-7b46b9fbbb-zb4jd_7b27c0d2-6000-4d1c-b04e-769309cad437 became leader 2026/07/19 13:57:01 13:56:27 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-bd4c5cbcb-xtd52_a24765c3-3c28-4cb1-9cfa-e9863b5f6de3 became leader 2026/07/19 13:57:01 13:56:26 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-zb4jd: Created - Created container: manager 2026/07/19 13:57:01 13:56:26 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-zb4jd: Started - Started container manager 2026/07/19 13:57:01 13:56:26 INFO Pod/kserve-controller-manager-dd9f6cf9-sqkcl: Created - Created container: manager 2026/07/19 13:57:01 13:56:26 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-zb4jd: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" in 535ms (535ms including waiting). Image size: 236366599 bytes. 2026/07/19 13:57:01 13:56:26 INFO Pod/kserve-controller-manager-dd9f6cf9-sqkcl: Started - Started container manager 2026/07/19 13:57:01 13:56:26 INFO Pod/kserve-controller-manager-dd9f6cf9-sqkcl: AddedInterface - Add eth0 [10.131.1.180/23] from ovn-kubernetes 2026/07/19 13:57:01 13:56:26 INFO Pod/kserve-controller-manager-dd9f6cf9-sqkcl: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" 2026/07/19 13:57:01 13:56:26 INFO Pod/kserve-controller-manager-dd9f6cf9-sqkcl: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" in 632ms (632ms including waiting). Image size: 238435613 bytes. 2026/07/19 13:57:01 13:56:25 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-zb4jd: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-7b46b9fbbb-zb4jd to ip-10-0-1-23.us-east-2.compute.internal 2026/07/19 13:57:01 13:56:25 INFO Pod/kserve-controller-manager-dd9f6cf9-sqkcl: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-dd9f6cf9-sqkcl to ip-10-0-1-227.us-east-2.compute.internal 2026/07/19 13:57:01 13:56:25 INFO ReplicaSet/kserve-controller-manager-dd9f6cf9: SuccessfulCreate - Created pod: kserve-controller-manager-dd9f6cf9-sqkcl 2026/07/19 13:57:01 13:56:25 INFO ReplicaSet/llmisvc-controller-manager-7b46b9fbbb: SuccessfulCreate - Created pod: llmisvc-controller-manager-7b46b9fbbb-zb4jd 2026/07/19 13:57:01 13:56:25 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-zb4jd: AddedInterface - Add eth0 [10.130.1.143/23] from ovn-kubernetes 2026/07/19 13:57:01 13:56:25 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-zb4jd: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" 2026/07/19 13:57:01 13:56:25 WARN Pod/kserve-controller-manager-dd9f6cf9-sqkcl: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "kserve-webhook-server-cert" not found 2026/07/19 13:57:01 13:56:25 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-dd9f6cf9 from 0 to 1 2026/07/19 13:57:01 13:56:25 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-7b46b9fbbb from 0 to 1 2026/07/19 13:57:01 13:56:12 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/19 13:57:01 13:56:12 INFO Pod/model-serving-api-5679d6b4f-mtvnb: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-5679d6b4f-mtvnb to ip-10-0-1-227.us-east-2.compute.internal 2026/07/19 13:57:01 13:56:12 INFO Pod/odh-model-controller-bd4c5cbcb-xtd52: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 600ms (600ms including waiting). Image size: 238959091 bytes. 2026/07/19 13:57:01 13:56:12 INFO Pod/model-serving-api-5679d6b4f-mtvnb: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/19 13:57:01 13:56:12 INFO Pod/model-serving-api-5679d6b4f-mtvnb: Created - Created container: server 2026/07/19 13:57:01 13:56:12 INFO Pod/model-serving-api-5679d6b4f-mtvnb: AddedInterface - Add eth0 [10.131.1.179/23] from ovn-kubernetes 2026/07/19 13:57:01 13:56:12 INFO Pod/odh-model-controller-bd4c5cbcb-xtd52: Started - Started container manager 2026/07/19 13:57:01 13:56:12 INFO Pod/model-serving-api-5679d6b4f-mtvnb: Started - Started container server 2026/07/19 13:57:01 13:56:12 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-5679d6b4f from 0 to 1 2026/07/19 13:57:01 13:56:12 WARN ReplicaSet/model-serving-api-5679d6b4f: FailedCreate - Error creating: pods "model-serving-api-5679d6b4f-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/19 13:57:01 13:56:12 INFO ReplicaSet/model-serving-api-5679d6b4f: SuccessfulCreate - Created pod: model-serving-api-5679d6b4f-mtvnb 2026/07/19 13:57:01 13:56:12 INFO Pod/odh-model-controller-bd4c5cbcb-xtd52: Created - Created container: manager 2026/07/19 13:57:01 13:56:11 INFO Pod/odh-model-controller-bd4c5cbcb-xtd52: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/19 13:57:01 13:56:11 INFO Pod/odh-model-controller-bd4c5cbcb-xtd52: AddedInterface - Add eth0 [10.131.1.178/23] from ovn-kubernetes 2026/07/19 13:57:01 13:56:10 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/19 13:57:01 13:56:10 INFO Pod/odh-model-controller-bd4c5cbcb-xtd52: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-bd4c5cbcb-xtd52 to ip-10-0-1-227.us-east-2.compute.internal 2026/07/19 13:57:01 13:56:10 INFO ReplicaSet/odh-model-controller-bd4c5cbcb: SuccessfulCreate - Created pod: odh-model-controller-bd4c5cbcb-xtd52 2026/07/19 13:57:01 13:56:10 WARN Pod/odh-model-controller-bd4c5cbcb-xtd52: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/19 13:57:01 13:56:10 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-bd4c5cbcb from 0 to 1 2026/07/19 13:57:01 13:56:01 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-gt9lm: Killing - Stopping container manager 2026/07/19 13:57:01 13:56:01 INFO Pod/model-serving-api-5679d6b4f-qzg7k: Killing - Stopping container server 2026/07/19 13:57:01 13:56:01 INFO Pod/odh-model-controller-bd4c5cbcb-crdnn: Killing - Stopping container manager 2026/07/19 13:57:01 13:56:01 INFO Pod/kserve-controller-manager-dd9f6cf9-skkcc: Killing - Stopping container manager 2026/07/19 13:57:01 13:56:00 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/19 13:57:01 13:55:36 INFO Lease/llminferenceservice-kserve-controller-manager: LeaderElection - llmisvc-controller-manager-7b46b9fbbb-gt9lm_9251e49e-a9e3-4a5c-af06-a9c53b039040 became leader 2026/07/19 13:57:01 13:55:36 INFO Lease/kserve-controller-manager-leader-lock: LeaderElection - kserve-controller-manager-dd9f6cf9-skkcc_ccac0306-7c06-410a-a84e-54308b455eb6 became leader 2026/07/19 13:57:01 13:55:19 INFO Pod/kserve-controller-manager-dd9f6cf9-skkcc: Created - Created container: manager 2026/07/19 13:57:01 13:55:19 INFO Pod/kserve-controller-manager-dd9f6cf9-skkcc: Pulling - Pulling image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" 2026/07/19 13:57:01 13:55:19 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-gt9lm: Started - Started container manager 2026/07/19 13:57:01 13:55:19 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-gt9lm: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" in 439ms (439ms including waiting). Image size: 236366599 bytes. 2026/07/19 13:57:01 13:55:19 INFO Pod/kserve-controller-manager-dd9f6cf9-skkcc: AddedInterface - Add eth0 [10.131.1.176/23] from ovn-kubernetes 2026/07/19 13:57:01 13:55:19 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-gt9lm: Created - Created container: manager 2026/07/19 13:57:01 13:55:19 INFO Pod/kserve-controller-manager-dd9f6cf9-skkcc: Started - Started container manager 2026/07/19 13:57:01 13:55:19 INFO Lease/odh-model-controller.opendatahub.io: LeaderElection - odh-model-controller-bd4c5cbcb-crdnn_64f5bb0f-501f-49b0-a1b8-961fa1a0c7a5 became leader 2026/07/19 13:57:01 13:55:19 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-gt9lm: Pulling - Pulling image "quay.io/rhoai/odh-kserve-llmisvc-controller-rhel9@sha256:3963f882e12e82b3725c7b940109b995b1a690929039b601b1eb8b24318bc960" 2026/07/19 13:57:01 13:55:19 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-gt9lm: AddedInterface - Add eth0 [10.131.1.177/23] from ovn-kubernetes 2026/07/19 13:57:01 13:55:19 INFO Pod/kserve-controller-manager-dd9f6cf9-skkcc: Pulled - Successfully pulled image "quay.io/rhoai/odh-kserve-controller-rhel9@sha256:272e4a1ff2ad34d0c660ea1e6d0e48691106380580d5067ac06f2a132781606c" in 480ms (480ms including waiting). Image size: 238435613 bytes. 2026/07/19 13:57:01 13:55:18 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-gt9lm: Scheduled - Successfully assigned redhat-ods-applications/llmisvc-controller-manager-7b46b9fbbb-gt9lm to ip-10-0-1-227.us-east-2.compute.internal 2026/07/19 13:57:01 13:55:18 INFO Pod/kserve-controller-manager-dd9f6cf9-skkcc: Scheduled - Successfully assigned redhat-ods-applications/kserve-controller-manager-dd9f6cf9-skkcc to ip-10-0-1-227.us-east-2.compute.internal 2026/07/19 13:57:01 13:55:18 INFO Deployment/kserve-controller-manager: ScalingReplicaSet - Scaled up replica set kserve-controller-manager-dd9f6cf9 from 0 to 1 2026/07/19 13:57:01 13:55:18 INFO ReplicaSet/kserve-controller-manager-dd9f6cf9: SuccessfulCreate - Created pod: kserve-controller-manager-dd9f6cf9-skkcc 2026/07/19 13:57:01 13:55:18 INFO Deployment/llmisvc-controller-manager: ScalingReplicaSet - Scaled up replica set llmisvc-controller-manager-7b46b9fbbb from 0 to 1 2026/07/19 13:57:01 13:55:18 INFO ReplicaSet/llmisvc-controller-manager-7b46b9fbbb: SuccessfulCreate - Created pod: llmisvc-controller-manager-7b46b9fbbb-gt9lm 2026/07/19 13:57:01 13:55:11 INFO Lease/trainer.kubeflow.org: LeaderElection - kubeflow-trainer-controller-manager-b5fd84b9c-slt4f_cf6a441e-e9ab-4cc4-b8a1-d7377efa5676 became leader 2026/07/19 13:57:01 13:55:02 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/19 13:57:01 13:55:02 INFO Pod/model-serving-api-5679d6b4f-qzg7k: Scheduled - Successfully assigned redhat-ods-applications/model-serving-api-5679d6b4f-qzg7k to ip-10-0-1-227.us-east-2.compute.internal 2026/07/19 13:57:01 13:55:02 INFO Pod/odh-model-controller-bd4c5cbcb-crdnn: Created - Created container: manager 2026/07/19 13:57:01 13:55:02 INFO Pod/model-serving-api-5679d6b4f-qzg7k: Started - Started container server 2026/07/19 13:57:01 13:55:02 INFO Pod/model-serving-api-5679d6b4f-qzg7k: Created - Created container: server 2026/07/19 13:57:01 13:55:02 WARN ReplicaSet/model-serving-api-5679d6b4f: FailedCreate - Error creating: pods "model-serving-api-5679d6b4f-" is forbidden: error looking up service account redhat-ods-applications/model-serving-api: serviceaccount "model-serving-api" not found 2026/07/19 13:57:01 13:55:02 INFO Deployment/model-serving-api: ScalingReplicaSet - Scaled up replica set model-serving-api-5679d6b4f from 0 to 1 2026/07/19 13:57:01 13:55:02 INFO Pod/model-serving-api-5679d6b4f-qzg7k: AddedInterface - Add eth0 [10.131.1.175/23] from ovn-kubernetes 2026/07/19 13:57:01 13:55:02 INFO Pod/model-serving-api-5679d6b4f-qzg7k: Pulled - Container image "quay.io/rhoai/odh-model-serving-api-rhel9@sha256:2ed53d2b291a04804cf3c09e359c92b04ec0af9f3a842beba232efd42f7e47a0" already present on machine 2026/07/19 13:57:01 13:55:02 INFO Pod/odh-model-controller-bd4c5cbcb-crdnn: Pulled - Successfully pulled image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" in 446ms (446ms including waiting). Image size: 238959091 bytes. 2026/07/19 13:57:01 13:55:02 INFO ReplicaSet/model-serving-api-5679d6b4f: SuccessfulCreate - Created pod: model-serving-api-5679d6b4f-qzg7k 2026/07/19 13:57:01 13:55:02 INFO Pod/odh-model-controller-bd4c5cbcb-crdnn: Started - Started container manager 2026/07/19 13:57:01 13:55:01 INFO Pod/odh-model-controller-bd4c5cbcb-crdnn: Pulling - Pulling image "quay.io/rhoai/odh-model-controller-rhel9@sha256:b645d95a1d0ff3105935c0b8f980544d1b6003206262e4ceb77bac50fee78528" 2026/07/19 13:57:01 13:55:01 INFO Pod/odh-model-controller-bd4c5cbcb-crdnn: AddedInterface - Add eth0 [10.131.1.174/23] from ovn-kubernetes 2026/07/19 13:57:01 13:55:00 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/19 13:57:01 13:55:00 INFO Pod/odh-model-controller-bd4c5cbcb-crdnn: Scheduled - Successfully assigned redhat-ods-applications/odh-model-controller-bd4c5cbcb-crdnn to ip-10-0-1-227.us-east-2.compute.internal 2026/07/19 13:57:01 13:55:00 INFO ReplicaSet/odh-model-controller-bd4c5cbcb: SuccessfulCreate - Created pod: odh-model-controller-bd4c5cbcb-crdnn 2026/07/19 13:57:01 13:55:00 WARN Pod/odh-model-controller-bd4c5cbcb-crdnn: FailedMount - MountVolume.SetUp failed for volume "cert" : secret "odh-model-controller-webhook-cert" not found 2026/07/19 13:57:01 13:55:00 INFO Deployment/odh-model-controller: ScalingReplicaSet - Scaled up replica set odh-model-controller-bd4c5cbcb from 0 to 1 2026/07/19 13:57:01 13:54:53 INFO Pod/kubeflow-trainer-controller-manager-b5fd84b9c-slt4f: Scheduled - Successfully assigned redhat-ods-applications/kubeflow-trainer-controller-manager-b5fd84b9c-slt4f to ip-10-0-1-227.us-east-2.compute.internal 2026/07/19 13:57:01 13:54:53 INFO Pod/kubeflow-trainer-controller-manager-b5fd84b9c-slt4f: Created - Created container: manager 2026/07/19 13:57:01 13:54:53 INFO Deployment/kubeflow-trainer-controller-manager: ScalingReplicaSet - Scaled up replica set kubeflow-trainer-controller-manager-b5fd84b9c from 0 to 1 2026/07/19 13:57:01 13:54:53 INFO Pod/kubeflow-trainer-controller-manager-b5fd84b9c-slt4f: AddedInterface - Add eth0 [10.131.1.173/23] from ovn-kubernetes 2026/07/19 13:57:01 13:54:53 INFO Pod/kubeflow-trainer-controller-manager-b5fd84b9c-slt4f: Pulled - Container image "quay.io/rhoai/odh-trainer-rhel9@sha256:59ea523e2df9c1e705c755308ac11420682081cc63d54890e2b15fbd7d8ba570" already present on machine 2026/07/19 13:57:01 13:54:53 INFO Pod/kubeflow-trainer-controller-manager-b5fd84b9c-slt4f: Started - Started container manager 2026/07/19 13:57:01 13:54:53 INFO ReplicaSet/kubeflow-trainer-controller-manager-b5fd84b9c: SuccessfulCreate - Created pod: kubeflow-trainer-controller-manager-b5fd84b9c-slt4f 2026/07/19 13:57:01 13:54:50 INFO Pod/odh-model-controller-bd4c5cbcb-g6fzh: Killing - Stopping container manager 2026/07/19 13:57:01 13:54:49 WARN ServiceMonitor/odh-model-controller-metrics-monitor: InvalidConfiguration - "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token [REDACTED] which Prometheus specification prohibits 2026/07/19 13:57:01 13:54:49 INFO Pod/llmisvc-controller-manager-7b46b9fbbb-84lvr: Killing - Stopping container manager 2026/07/19 13:57:01 13:54:49 INFO Pod/kserve-controller-manager-dd9f6cf9-fqbll: Killing - Stopping container manager 2026/07/19 13:57:01 13:54:49 INFO Pod/model-serving-api-5679d6b4f-xhcs9: Killing - Stopping container server 2026/07/19 13:57:01 13:53:26 INFO Pod/maas-controller-75bbfd978-nhkw7: Pulled - Successfully pulled image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" in 455ms (455ms including waiting). Image size: 212866077 bytes. 2026/07/19 13:57:01 13:53:25 INFO Pod/maas-controller-75bbfd978-nhkw7: Pulling - Pulling image "quay.io/rhoai/odh-maas-controller-rhel9@sha256:e291a29f5393e1114702511221154cf754b853bf1c227292acafe7777f583b07" 2026/07/19 13:57:01 13:52:58 WARN Pod/maas-controller-75bbfd978-nhkw7: FailedToRetrieveImagePullSecret - Unable to retrieve some image pull secrets (maas-controller-dockercfg-knq5p); attempting to pull the image may not succeed. 2026/07/19 13:57:01 === RESOURCE QUOTAS === 2026/07/19 13:57:01 No resource quotas found in monitored namespaces 2026/07/19 13:57:01 Classification for components: infrastructure/pod-startup (code=1002, confidence=medium) 2026/07/19 13:57:01 Evidence: container maas-controller-75bbfd978-nhkw7/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-75bbfd978-nhkw7_redhat-ods-applications(12ea2cfb-9eba-4d2e-9b67-7c70aa2546b4) FAILURE_CLASSIFICATION: {"category":"infrastructure","subcategory":"pod-startup","error_code":1002,"evidence":["container maas-controller-75bbfd978-nhkw7/manager waiting: CrashLoopBackOff back-off 5m0s restarting failed container=manager pod=maas-controller-75bbfd978-nhkw7_redhat-ods-applications(12ea2cfb-9eba-4d2e-9b67-7c70aa2546b4)"],"confidence":"medium"} 2026/07/19 13:57:01 Diagnostics complete for failed test: components controller_test.go:484: Stopping: components test failed. 2026/07/19 13:57:01 Circuit breaker: failure classified as infrastructure/pod-startup (code=1002) 2026/07/19 13:57:01 Circuit breaker: infrastructure/unknown failure recorded (1/3 consecutive) Deletion Policy: Never. Skipping deletion tests. 2026/07/19 13:57:02 Circuit breaker: no trips during this run (threshold=3) --- FAIL: TestOdhOperator (1446.43s)