Sep 11 14:13:09.279504 ip-10-0-135-150 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 11 14:13:09.279517 ip-10-0-135-150 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 11 14:13:09.279526 ip-10-0-135-150 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 11 14:13:09.279732 ip-10-0-135-150 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 11 14:13:19.388448 ip-10-0-135-150 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 11 14:13:19.388467 ip-10-0-135-150 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 682bcfac60704f4b98d068b48a2c12b5 --
Sep 11 14:14:55.154220 ip-10-0-135-150 systemd[1]: Starting Kubernetes Kubelet...
Sep 11 14:14:55.565037 ip-10-0-135-150 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 14:14:55.565037 ip-10-0-135-150 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 11 14:14:55.565037 ip-10-0-135-150 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 14:14:55.565037 ip-10-0-135-150 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 11 14:14:55.565037 ip-10-0-135-150 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 14:14:55.565719 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.565140    2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 11 14:14:55.570172 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570154    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 14:14:55.570172 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570171    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 14:14:55.570248 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570177    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 14:14:55.570248 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570180    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 14:14:55.570248 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570189    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 14:14:55.570248 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570192    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 14:14:55.570248 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570195    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 14:14:55.570248 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570197    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 14:14:55.570248 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570209    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 14:14:55.570248 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570214    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 14:14:55.570248 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570217    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 14:14:55.570248 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570220    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 14:14:55.570248 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570223    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 14:14:55.570248 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570226    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 14:14:55.570248 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570229    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 14:14:55.570248 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570232    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 14:14:55.570248 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570235    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 14:14:55.570248 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570238    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 14:14:55.570248 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570240    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 14:14:55.570248 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570243    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 14:14:55.570248 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570246    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 14:14:55.570248 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570249    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 14:14:55.570730 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570252    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 14:14:55.570730 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570255    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 14:14:55.570730 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570258    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 14:14:55.570730 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570260    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 14:14:55.570730 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570263    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 14:14:55.570730 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570266    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 14:14:55.570730 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570270    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 14:14:55.570730 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570273    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 14:14:55.570730 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570276    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 14:14:55.570730 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570279    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 14:14:55.570730 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570282    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 14:14:55.570730 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570285    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 14:14:55.570730 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570287    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 14:14:55.570730 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570290    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 14:14:55.570730 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570296    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 14:14:55.570730 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570299    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 14:14:55.570730 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570301    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 14:14:55.570730 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570304    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 14:14:55.570730 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570306    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 14:14:55.571409 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570309    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 14:14:55.571409 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570311    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 14:14:55.571409 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570314    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 14:14:55.571409 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570316    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 14:14:55.571409 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570319    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 14:14:55.571409 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570321    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 14:14:55.571409 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570324    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 14:14:55.571409 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570326    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 14:14:55.571409 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570329    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 14:14:55.571409 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570331    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 14:14:55.571409 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570334    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 14:14:55.571409 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570336    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 14:14:55.571409 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570339    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 14:14:55.571409 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570341    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 14:14:55.571409 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570345    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 14:14:55.571409 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570347    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 14:14:55.571409 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570350    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 14:14:55.571409 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570353    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 14:14:55.571409 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570356    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 14:14:55.571409 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570358    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 14:14:55.572025 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570361    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 14:14:55.572025 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570363    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 14:14:55.572025 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570366    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 14:14:55.572025 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570369    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 14:14:55.572025 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570372    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 14:14:55.572025 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570374    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 14:14:55.572025 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570376    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 14:14:55.572025 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570379    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 14:14:55.572025 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570382    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 14:14:55.572025 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570385    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 14:14:55.572025 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570392    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 14:14:55.572025 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570394    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 14:14:55.572025 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570397    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 14:14:55.572025 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570399    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 14:14:55.572025 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570402    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 14:14:55.572025 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570404    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 14:14:55.572025 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570408    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 14:14:55.572025 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570412    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 14:14:55.572025 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570415    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 14:14:55.572025 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570418    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 14:14:55.572516 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570420    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 14:14:55.572516 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570423    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 14:14:55.572516 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570425    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 14:14:55.572516 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570428    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 14:14:55.572516 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570430    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 14:14:55.572516 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570433    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 14:14:55.572516 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570437    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 14:14:55.572516 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570440    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 14:14:55.572516 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570443    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 14:14:55.572516 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570446    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 14:14:55.572516 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570449    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 14:14:55.572516 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570452    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 14:14:55.572516 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570454    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 14:14:55.572516 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570457    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 14:14:55.572516 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570460    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 14:14:55.572516 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570462    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 14:14:55.572516 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570465    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 14:14:55.572516 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.570467    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 14:14:55.572516 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572508    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 14:14:55.572516 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572523    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 14:14:55.573056 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572526    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 14:14:55.573056 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572529    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 14:14:55.573056 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572534    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 14:14:55.573056 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572537    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 14:14:55.573056 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572540    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 14:14:55.573056 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572543    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 14:14:55.573056 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572546    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 14:14:55.573056 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572550    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 14:14:55.573056 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572555    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 14:14:55.573056 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572558    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 14:14:55.573056 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572561    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 14:14:55.573056 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572564    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 14:14:55.573056 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572567    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 14:14:55.573056 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572570    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 14:14:55.573056 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572574    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 14:14:55.573056 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572576    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 14:14:55.573056 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572579    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 14:14:55.573056 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572587    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 14:14:55.573056 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572590    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 14:14:55.573536 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572593    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 14:14:55.573536 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572595    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 14:14:55.573536 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572598    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 14:14:55.573536 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572600    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 14:14:55.573536 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572603    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 14:14:55.573536 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572606    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 14:14:55.573536 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572608    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 14:14:55.573536 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572611    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 14:14:55.573536 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572614    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 14:14:55.573536 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572618    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 14:14:55.573536 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572623    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 14:14:55.573536 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572626    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 14:14:55.573536 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572629    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 14:14:55.573536 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572631    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 14:14:55.573536 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572635    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 14:14:55.573536 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572637    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 14:14:55.573536 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572640    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 14:14:55.573536 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572642    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 14:14:55.573536 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572645    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 14:14:55.573536 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572648    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 14:14:55.574058 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572650    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 14:14:55.574058 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572653    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 14:14:55.574058 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572655    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 14:14:55.574058 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572658    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 14:14:55.574058 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572660    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 14:14:55.574058 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572663    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 14:14:55.574058 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572666    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 14:14:55.574058 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572668    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 14:14:55.574058 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572670    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 14:14:55.574058 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572673    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 14:14:55.574058 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572676    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 14:14:55.574058 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572678    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 14:14:55.574058 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572681    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 14:14:55.574058 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572684    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 14:14:55.574058 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572686    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 14:14:55.574058 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572689    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 14:14:55.574058 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572691    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 14:14:55.574058 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572694    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 14:14:55.574058 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572696    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 14:14:55.574058 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572698    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 14:14:55.574566 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572701    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 14:14:55.574566 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572704    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 14:14:55.574566 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572706    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 14:14:55.574566 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572709    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 14:14:55.574566 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572712    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 14:14:55.574566 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572714    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 14:14:55.574566 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572717    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 14:14:55.574566 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572723    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 14:14:55.574566 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572727    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 14:14:55.574566 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572730    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 14:14:55.574566 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572732    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 14:14:55.574566 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572735    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 14:14:55.574566 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572738    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 14:14:55.574566 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572740    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 14:14:55.574566 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572743    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 14:14:55.574566 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572745    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 14:14:55.574566 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572748    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 14:14:55.574566 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572750    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 14:14:55.574566 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572753    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 14:14:55.575058 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572755    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 14:14:55.575058 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572757    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 14:14:55.575058 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572760    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 14:14:55.575058 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572763    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 14:14:55.575058 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572766    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 14:14:55.575058 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572769    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 14:14:55.575058 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572771    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 14:14:55.575058 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572774    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 14:14:55.575058 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572776    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 14:14:55.575058 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572779    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 14:14:55.575058 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572781    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 14:14:55.575058 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572784    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 14:14:55.575058 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572786    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 14:14:55.575058 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572789    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 14:14:55.575058 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572791    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 14:14:55.575058 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572794    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 14:14:55.575058 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572797    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 14:14:55.575058 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572801    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 14:14:55.575058 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.572805    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 14:14:55.575526 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573220    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 14:14:55.575526 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573227    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 14:14:55.575526 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573231    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 14:14:55.575526 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573234    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 14:14:55.575526 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573237    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 14:14:55.575526 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573240    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 14:14:55.575526 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573243    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 14:14:55.575526 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573246    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 14:14:55.575526 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573249    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 14:14:55.575526 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573251    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 14:14:55.575526 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573254    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 14:14:55.575526 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573257    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 14:14:55.575526 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573259    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 14:14:55.575526 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573262    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 14:14:55.575526 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573265    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 14:14:55.575526 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573267    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 14:14:55.575526 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573271    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 14:14:55.575526 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573274    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 14:14:55.575526 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573276    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 14:14:55.575526 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573279    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 14:14:55.576034 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573282    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 14:14:55.576034 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573284    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 14:14:55.576034 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573287    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 14:14:55.576034 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573290    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 14:14:55.576034 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573292    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 14:14:55.576034 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573295    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 14:14:55.576034 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573297    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 14:14:55.576034 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573300    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 14:14:55.576034 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573302    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 14:14:55.576034 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573305    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 14:14:55.576034 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573308    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 14:14:55.576034 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573310    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 14:14:55.576034 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573313    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 14:14:55.576034 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573316    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 14:14:55.576034 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573318    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 14:14:55.576034 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573321    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 14:14:55.576034 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573324    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 14:14:55.576034 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573326    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 14:14:55.576034 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573329    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 14:14:55.576034 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573331    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 14:14:55.576531 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573334    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 14:14:55.576531 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573336    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 14:14:55.576531 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573339    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 14:14:55.576531 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573341    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 14:14:55.576531 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573344    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 14:14:55.576531 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573346    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 14:14:55.576531 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573349    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 14:14:55.576531 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573352    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 14:14:55.576531 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573354    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 14:14:55.576531 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573357    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 14:14:55.576531 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573359    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 14:14:55.576531 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573362    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 14:14:55.576531 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573365    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 14:14:55.576531 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573367    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 14:14:55.576531 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573369    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 14:14:55.576531 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573372    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 14:14:55.576531 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573374    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 14:14:55.576531 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573377    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 14:14:55.576531 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573379    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 14:14:55.576531 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573382    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 14:14:55.577033 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573392    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 14:14:55.577033 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573396    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 14:14:55.577033 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573400    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 14:14:55.577033 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573403    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 14:14:55.577033 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573408    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 14:14:55.577033 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573411    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 14:14:55.577033 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573414    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 14:14:55.577033 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573417    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 14:14:55.577033 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573420    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 14:14:55.577033 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573422    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 14:14:55.577033 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573425    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 14:14:55.577033 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573428    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 14:14:55.577033 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573430    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 14:14:55.577033 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573433    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 14:14:55.577033 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573436    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 14:14:55.577033 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573438    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 14:14:55.577033 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573441    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 14:14:55.577033 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573444    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 14:14:55.577033 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573447    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 14:14:55.577585 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573449    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 14:14:55.577585 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573452    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 14:14:55.577585 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573455    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 14:14:55.577585 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573458    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 14:14:55.577585 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573461    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 14:14:55.577585 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573464    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 14:14:55.577585 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573467    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 14:14:55.577585 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573470    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 14:14:55.577585 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573472    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 14:14:55.577585 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573474    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 14:14:55.577585 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573477    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 14:14:55.577585 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573480    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 14:14:55.577585 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573484    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 14:14:55.577585 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573486    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 14:14:55.577585 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573489    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 14:14:55.577585 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573492    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 14:14:55.577585 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573496    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 14:14:55.577585 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573501    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 14:14:55.577585 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573504    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 14:14:55.578102 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.573507    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 14:14:55.578102 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573584    2581 flags.go:64] FLAG: --address="0.0.0.0"
Sep 11 14:14:55.578102 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573619    2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 11 14:14:55.578102 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573626    2581 flags.go:64] FLAG: --anonymous-auth="true"
Sep 11 14:14:55.578102 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573631    2581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 11 14:14:55.578102 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573641    2581 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 11 14:14:55.578102 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573644    2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 11 14:14:55.578102 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573648    2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 11 14:14:55.578102 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573653    2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 11 14:14:55.578102 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573656    2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 11 14:14:55.578102 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573659    2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 11 14:14:55.578102 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573662    2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 11 14:14:55.578102 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573665    2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 11 14:14:55.578102 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573668    2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 11 14:14:55.578102 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573671    2581 flags.go:64] FLAG: --cgroup-root=""
Sep 11 14:14:55.578102 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573674    2581 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 11 14:14:55.578102 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573677    2581 flags.go:64] FLAG: --client-ca-file=""
Sep 11 14:14:55.578102 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573681    2581 flags.go:64] FLAG: --cloud-provider="external"
Sep 11 14:14:55.578102 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573683    2581 flags.go:64] FLAG: --cluster-dns="[]"
Sep 11 14:14:55.578102 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573688    2581 flags.go:64] FLAG: --cluster-domain=""
Sep 11 14:14:55.578102 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573690    2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 11 14:14:55.578102 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573694    2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 11 14:14:55.578102 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573697    2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 11 14:14:55.578102 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573700    2581 flags.go:64] FLAG: --container-log-max-files="5"
Sep 11 14:14:55.578683 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573704    2581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 11 14:14:55.578683 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573706    2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 11 14:14:55.578683 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573709    2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 11 14:14:55.578683 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573713    2581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 11 14:14:55.578683 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573716    2581 flags.go:64] FLAG: --contention-profiling="false"
Sep 11 14:14:55.578683 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573719    2581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 11 14:14:55.578683 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573723    2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 11 14:14:55.578683 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573726    2581 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 11 14:14:55.578683 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573729    2581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 11 14:14:55.578683 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573734    2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 11 14:14:55.578683 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573737    2581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 11 14:14:55.578683 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573741    2581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 11 14:14:55.578683 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573743    2581 flags.go:64] FLAG: --enable-load-reader="false"
Sep 11 14:14:55.578683 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573746    2581 flags.go:64] FLAG: --enable-server="true"
Sep 11 14:14:55.578683 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573749    2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 11 14:14:55.578683 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573754    2581 flags.go:64] FLAG: --event-burst="100"
Sep 11 14:14:55.578683 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573757    2581 flags.go:64] FLAG: --event-qps="50"
Sep 11 14:14:55.578683 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573760    2581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 11 14:14:55.578683 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573763    2581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 11 14:14:55.578683 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573766    2581 flags.go:64] FLAG: --eviction-hard=""
Sep 11 14:14:55.578683 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573769    2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 11 14:14:55.578683 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573773    2581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 11 14:14:55.578683 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573776    2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 11 14:14:55.578683 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573779    2581 flags.go:64] FLAG: --eviction-soft=""
Sep 11 14:14:55.578683 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573781    2581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 11 14:14:55.579296 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573784    2581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 11 14:14:55.579296 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573788    2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 11 14:14:55.579296 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573791    2581 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 11 14:14:55.579296 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573794    2581 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 11 14:14:55.579296 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573797    2581 flags.go:64] FLAG: --fail-swap-on="true"
Sep 11 14:14:55.579296 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573800    2581 flags.go:64] FLAG: --feature-gates=""
Sep 11 14:14:55.579296 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573804    2581 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 11 14:14:55.579296 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573806    2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 11 14:14:55.579296 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573810    2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 11 14:14:55.579296 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573814    2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 11 14:14:55.579296 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573817    2581 flags.go:64] FLAG: --healthz-port="10248"
Sep 11 14:14:55.579296 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573820    2581 flags.go:64] FLAG: --help="false"
Sep 11 14:14:55.579296 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573822    2581 flags.go:64] FLAG: --hostname-override="ip-10-0-135-150.ec2.internal"
Sep 11 14:14:55.579296 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573827    2581 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 11 14:14:55.579296 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573830    2581 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 11 14:14:55.579296 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573833    2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 11 14:14:55.579296 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573838    2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 11 14:14:55.579296 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573842    2581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 11 14:14:55.579296 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573845    2581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 11 14:14:55.579296 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573848    2581 flags.go:64] FLAG: --image-service-endpoint=""
Sep 11 14:14:55.579296 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573851    2581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 11 14:14:55.579296 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573854    2581 flags.go:64] FLAG: --kube-api-burst="100"
Sep 11 14:14:55.579296 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573856    2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 11 14:14:55.579899 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573859    2581 flags.go:64] FLAG: --kube-api-qps="50"
Sep 11 14:14:55.579899 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573862    2581 flags.go:64] FLAG: --kube-reserved=""
Sep 11 14:14:55.579899 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573865    2581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 11 14:14:55.579899 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573868    2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 11 14:14:55.579899 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573871    2581 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 11 14:14:55.579899 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573874    2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 11 14:14:55.579899 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573877    2581 flags.go:64] FLAG: --lock-file=""
Sep 11 14:14:55.579899 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573879    2581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 11 14:14:55.579899 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573882    2581 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 11 14:14:55.579899 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573885    2581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 11 14:14:55.579899 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573891    2581 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 11 14:14:55.579899 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573894    2581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 11 14:14:55.579899 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573897    2581 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 11 14:14:55.579899 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573900    2581 flags.go:64] FLAG: --logging-format="text"
Sep 11 14:14:55.579899 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573915    2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 11 14:14:55.579899 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573918    2581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 11 14:14:55.579899 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573921    2581 flags.go:64] FLAG: --manifest-url=""
Sep 11 14:14:55.579899 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573924    2581 flags.go:64] FLAG: --manifest-url-header=""
Sep 11 14:14:55.579899 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573928    2581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 11 14:14:55.579899 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573931    2581 flags.go:64] FLAG: --max-open-files="1000000"
Sep 11 14:14:55.579899 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573936    2581 flags.go:64] FLAG: --max-pods="110"
Sep 11 14:14:55.579899 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573939    2581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 11 14:14:55.579899 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573943    2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 11 14:14:55.579899 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573946    2581 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 11 14:14:55.579899 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573949    2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 11 14:14:55.580524 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573954    2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 11 14:14:55.580524 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573956    2581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 11 14:14:55.580524 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573959    2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 11 14:14:55.580524 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573967    2581 flags.go:64] FLAG: --node-status-max-images="50"
Sep 11 14:14:55.580524 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573970    2581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 11 14:14:55.580524 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573973    2581 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 11 14:14:55.580524 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573976    2581 flags.go:64] FLAG: --pod-cidr=""
Sep 11 14:14:55.580524 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573979    2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 11 14:14:55.580524 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573985    2581 flags.go:64] FLAG: --pod-manifest-path=""
Sep 11 14:14:55.580524 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573988    2581 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 11 14:14:55.580524 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573991    2581 flags.go:64] FLAG: --pods-per-core="0"
Sep 11 14:14:55.580524 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573994    2581 flags.go:64] FLAG: --port="10250"
Sep 11 14:14:55.580524 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.573997    2581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 11 14:14:55.580524 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.574001    2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f1423d7bd5c6d1e7"
Sep 11 14:14:55.580524 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.574004    2581 flags.go:64] FLAG: --qos-reserved=""
Sep 11 14:14:55.580524 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.574007    2581 flags.go:64] FLAG: --read-only-port="10255"
Sep 11 14:14:55.580524 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.574010    2581 flags.go:64] FLAG: --register-node="true"
Sep 11 14:14:55.580524 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.574013    2581 flags.go:64] FLAG: --register-with-taints=""
Sep 11 14:14:55.580524 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.574017    2581 flags.go:64] FLAG: --registry-burst="10"
Sep 11 14:14:55.580524 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.574020    2581 flags.go:64] FLAG: --registry-qps="5"
Sep 11 14:14:55.580524 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.574023    2581 flags.go:64] FLAG: --reserved-cpus=""
Sep 11 14:14:55.580524 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.574026    2581 flags.go:64] FLAG: --reserved-memory=""
Sep 11 14:14:55.580524 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.574029    2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 11 14:14:55.580524 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.574032    2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 11 14:14:55.580524 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.574035    2581 flags.go:64] FLAG: --rotate-certificates="false"
Sep 11 14:14:55.581188 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.574038    2581 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 11 14:14:55.581188 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.574041    2581 flags.go:64] FLAG: --runonce="false"
Sep 11 14:14:55.581188 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.574044    2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 11 14:14:55.581188 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.574047    2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 11 14:14:55.581188 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.574050    2581 flags.go:64] FLAG: --seccomp-default="false"
Sep 11 14:14:55.581188 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.574054    2581 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 11 14:14:55.581188 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.574057    2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 11 14:14:55.581188 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.574062    2581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 11 14:14:55.581188 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.574065    2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 11 14:14:55.581188 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.574069    2581 flags.go:64] FLAG: --storage-driver-password="root"
Sep 11 14:14:55.581188 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.574072    2581 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 11 14:14:55.581188 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.574074    2581 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 11 14:14:55.581188 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.574077    2581 flags.go:64] FLAG: --storage-driver-user="root"
Sep 11 14:14:55.581188 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.574080    2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 11 14:14:55.581188 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.574083    2581 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 11 14:14:55.581188 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.574086    2581 flags.go:64] FLAG: --system-cgroups=""
Sep 11 14:14:55.581188 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.574089    2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 11 14:14:55.581188 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.574094    2581 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 11 14:14:55.581188 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.574097    2581 flags.go:64] FLAG: --tls-cert-file=""
Sep 11 14:14:55.581188 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.574100    2581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 11 14:14:55.581188 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.574104    2581 flags.go:64] FLAG: --tls-min-version=""
Sep 11 14:14:55.581188 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.574107    2581 flags.go:64] FLAG: --tls-private-key-file=""
Sep 11 14:14:55.581188 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.574110    2581 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 11 14:14:55.581188 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.574113    2581 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 11 14:14:55.581188 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.574116    2581 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 11 14:14:55.581827 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.574119    2581 flags.go:64] FLAG: --v="2"
Sep 11 14:14:55.581827 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.574123    2581 flags.go:64] FLAG: --version="false"
Sep 11 14:14:55.581827 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.574127    2581 flags.go:64] FLAG: --vmodule=""
Sep 11 14:14:55.581827 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.574131    2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 11 14:14:55.581827 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.574134    2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 11 14:14:55.581827 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574256    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 14:14:55.581827 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574260    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 14:14:55.581827 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574263    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 14:14:55.581827 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574266    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 14:14:55.581827 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574269    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 14:14:55.581827 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574272    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 14:14:55.581827 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574274    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 14:14:55.581827 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574280    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 14:14:55.581827 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574283    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 14:14:55.581827 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574288    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 14:14:55.581827 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574291    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 14:14:55.581827 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574294    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 14:14:55.581827 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574297    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 14:14:55.581827 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574300    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 14:14:55.581827 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574303    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 14:14:55.582344 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574306    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 14:14:55.582344 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574309    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 14:14:55.582344 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574311    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 14:14:55.582344 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574314    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 14:14:55.582344 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574317    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 14:14:55.582344 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574320    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 14:14:55.582344 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574323    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 14:14:55.582344 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574325    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 14:14:55.582344 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574328    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 14:14:55.582344 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574330    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 14:14:55.582344 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574333    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 14:14:55.582344 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574336    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 14:14:55.582344 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574338    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 14:14:55.582344 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574340    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 14:14:55.582344 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574343    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 14:14:55.582344 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574347    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 14:14:55.582344 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574349    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 14:14:55.582344 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574352    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 14:14:55.582344 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574355    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 14:14:55.582344 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574357    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 14:14:55.582849 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574360    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 14:14:55.582849 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574363    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 14:14:55.582849 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574367    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 14:14:55.582849 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574370    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 14:14:55.582849 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574375    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 14:14:55.582849 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574377    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 14:14:55.582849 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574381    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 14:14:55.582849 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574384    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 14:14:55.582849 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574387    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 14:14:55.582849 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574389    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 14:14:55.582849 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574391    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 14:14:55.582849 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574394    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 14:14:55.582849 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574397    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 14:14:55.582849 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574399    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 14:14:55.582849 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574402    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 14:14:55.582849 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574404    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 14:14:55.582849 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574407    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 14:14:55.582849 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574410    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 14:14:55.582849 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574414    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 14:14:55.583330 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574417    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 14:14:55.583330 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574420    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 14:14:55.583330 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574423    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 14:14:55.583330 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574426    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 14:14:55.583330 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574429    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 14:14:55.583330 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574431    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 14:14:55.583330 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574434    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 14:14:55.583330 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574437    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 14:14:55.583330 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574439    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 14:14:55.583330 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574442    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 14:14:55.583330 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574445    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 14:14:55.583330 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574447    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 14:14:55.583330 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574450    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 14:14:55.583330 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574452    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 14:14:55.583330 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574455    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 14:14:55.583330 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574458    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 14:14:55.583330 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574460    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 14:14:55.583330 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574465    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 14:14:55.583330 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574468    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 14:14:55.583330 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574472    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 14:14:55.583823 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574474    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 14:14:55.583823 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574477    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 14:14:55.583823 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574479    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 14:14:55.583823 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574482    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 14:14:55.583823 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574485    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 14:14:55.583823 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574487    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 14:14:55.583823 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574490    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 14:14:55.583823 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574493    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 14:14:55.583823 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574495    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 14:14:55.583823 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574498    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 14:14:55.583823 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574501    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 14:14:55.583823 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574503    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 14:14:55.583823 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574506    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 14:14:55.583823 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574508    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 14:14:55.583823 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574510    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 14:14:55.583823 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574513    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 14:14:55.583823 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574515    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 14:14:55.583823 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574518    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 14:14:55.583823 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574520    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 14:14:55.583823 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574523    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 14:14:55.584329 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574525    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 14:14:55.584329 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574528    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 14:14:55.584329 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574531    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 14:14:55.584329 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574533    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 14:14:55.584329 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.574536    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 14:14:55.584329 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.575359    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 14:14:55.584329 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.583101    2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 11 14:14:55.584329 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.583117    2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 11 14:14:55.584329 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.583996    2581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 11 14:14:55.584329 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.584003    2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 11 14:14:55.584329 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584065    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 14:14:55.584329 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584070    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 14:14:55.584329 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584073    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 14:14:55.584329 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584077    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 14:14:55.584329 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584080    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 14:14:55.584712 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584082    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 14:14:55.584712 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584085    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 14:14:55.584712 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584088    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 14:14:55.584712 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584091    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 14:14:55.584712 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584094    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 14:14:55.584712 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584096    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 14:14:55.584712 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584099    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 14:14:55.584712 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584103    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 14:14:55.584712 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584107    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 14:14:55.584712 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584110    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 14:14:55.584712 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584112    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 14:14:55.584712 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584115    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 14:14:55.584712 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584118    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 14:14:55.584712 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584120    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 14:14:55.584712 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584123    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 14:14:55.584712 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584125    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 14:14:55.584712 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584128    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 14:14:55.584712 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584130    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 14:14:55.584712 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584134    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 14:14:55.584712 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584138    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 14:14:55.585259 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584142    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 14:14:55.585259 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584145    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 14:14:55.585259 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584148    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 14:14:55.585259 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584151    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 14:14:55.585259 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584155    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 14:14:55.585259 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584157    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 14:14:55.585259 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584160    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 14:14:55.585259 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584163    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 14:14:55.585259 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584166    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 14:14:55.585259 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584168    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 14:14:55.585259 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584171    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 14:14:55.585259 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584173    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 14:14:55.585259 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584176    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 14:14:55.585259 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584179    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 14:14:55.585259 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584182    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 14:14:55.585259 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584184    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 14:14:55.585259 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584187    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 14:14:55.585259 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584190    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 14:14:55.585259 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584192    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 14:14:55.585259 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584195    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 14:14:55.585748 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584198    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 14:14:55.585748 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584200    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 14:14:55.585748 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584203    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 14:14:55.585748 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584205    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 14:14:55.585748 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584208    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 14:14:55.585748 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584210    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 14:14:55.585748 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584213    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 14:14:55.585748 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584215    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 14:14:55.585748 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584218    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 14:14:55.585748 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584220    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 14:14:55.585748 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584223    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 14:14:55.585748 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584226    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 14:14:55.585748 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584229    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 14:14:55.585748 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584232    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 14:14:55.585748 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584235    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 14:14:55.585748 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584237    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 14:14:55.585748 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584240    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 14:14:55.585748 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584243    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 14:14:55.585748 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584246    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 14:14:55.585748 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584248    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 14:14:55.586259 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584251    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 14:14:55.586259 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584253    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 14:14:55.586259 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584256    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 14:14:55.586259 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584258    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 14:14:55.586259 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584261    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 14:14:55.586259 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584264    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 14:14:55.586259 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584266    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 14:14:55.586259 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584269    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 14:14:55.586259 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584271    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 14:14:55.586259 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584274    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 14:14:55.586259 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584276    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 14:14:55.586259 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584279    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 14:14:55.586259 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584281    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 14:14:55.586259 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584284    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 14:14:55.586259 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584286    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 14:14:55.586259 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584289    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 14:14:55.586259 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584291    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 14:14:55.586259 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584294    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 14:14:55.586259 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584296    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 14:14:55.586259 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584299    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 14:14:55.586747 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584303    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 14:14:55.586747 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584306    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 14:14:55.586747 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584309    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 14:14:55.586747 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584312    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 14:14:55.586747 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584315    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 14:14:55.586747 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584318    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 14:14:55.586747 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584321    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 14:14:55.586747 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584323    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 14:14:55.586747 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584326    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 14:14:55.586747 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584329    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 14:14:55.586747 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584339    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 14:14:55.586747 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584342    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 14:14:55.586747 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584345    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 14:14:55.586747 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584348    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 14:14:55.586747 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.584353    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 14:14:55.587132 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584466    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 14:14:55.587132 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584471    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 14:14:55.587132 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584474    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 14:14:55.587132 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584477    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 14:14:55.587132 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584480    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 14:14:55.587132 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584483    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 14:14:55.587132 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584485    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 14:14:55.587132 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584488    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 14:14:55.587132 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584490    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 14:14:55.587132 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584493    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 14:14:55.587132 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584496    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 14:14:55.587132 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584498    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 14:14:55.587132 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584501    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 14:14:55.587132 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584503    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 14:14:55.587132 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584506    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 14:14:55.587132 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584509    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 14:14:55.587132 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584511    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 14:14:55.587132 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584514    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 14:14:55.587132 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584516    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 14:14:55.587132 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584519    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 14:14:55.587623 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584521    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 14:14:55.587623 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584525    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 14:14:55.587623 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584529    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 14:14:55.587623 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584533    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 14:14:55.587623 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584536    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 14:14:55.587623 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584539    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 14:14:55.587623 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584542    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 14:14:55.587623 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584545    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 14:14:55.587623 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584548    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 14:14:55.587623 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584551    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 14:14:55.587623 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584554    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 14:14:55.587623 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584557    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 14:14:55.587623 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584560    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 14:14:55.587623 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584562    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 14:14:55.587623 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584565    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 14:14:55.587623 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584567    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 14:14:55.587623 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584570    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 14:14:55.587623 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584572    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 14:14:55.587623 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584575    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 14:14:55.588096 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584578    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 14:14:55.588096 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584580    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 14:14:55.588096 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584583    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 14:14:55.588096 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584585    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 14:14:55.588096 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584588    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 14:14:55.588096 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584590    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 14:14:55.588096 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584593    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 14:14:55.588096 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584595    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 14:14:55.588096 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584598    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 14:14:55.588096 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584601    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 14:14:55.588096 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584605    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 14:14:55.588096 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584608    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 14:14:55.588096 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584611    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 14:14:55.588096 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584614    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 14:14:55.588096 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584617    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 14:14:55.588096 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584619    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 14:14:55.588096 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584622    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 14:14:55.588096 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584624    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 14:14:55.588096 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584627    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 14:14:55.588553 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584629    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 14:14:55.588553 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584632    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 14:14:55.588553 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584635    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 14:14:55.588553 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584641    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 14:14:55.588553 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584643    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 14:14:55.588553 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584646    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 14:14:55.588553 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584648    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 14:14:55.588553 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584651    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 14:14:55.588553 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584654    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 14:14:55.588553 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584656    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 14:14:55.588553 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584659    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 14:14:55.588553 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584661    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 14:14:55.588553 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584664    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 14:14:55.588553 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584666    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 14:14:55.588553 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584669    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 14:14:55.588553 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584672    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 14:14:55.588553 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584674    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 14:14:55.588553 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584677    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 14:14:55.588553 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584679    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 14:14:55.588553 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584682    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 14:14:55.589124 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584684    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 14:14:55.589124 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584686    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 14:14:55.589124 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584689    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 14:14:55.589124 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584691    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 14:14:55.589124 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584694    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 14:14:55.589124 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584696    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 14:14:55.589124 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584699    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 14:14:55.589124 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584703    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 14:14:55.589124 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584706    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 14:14:55.589124 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584709    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 14:14:55.589124 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584712    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 14:14:55.589124 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584714    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 14:14:55.589124 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584717    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 14:14:55.589124 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584720    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 14:14:55.589124 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584722    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 14:14:55.589124 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584726    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 14:14:55.589124 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584729    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 14:14:55.589124 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584732    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 14:14:55.589124 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584734    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 14:14:55.589124 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584737    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 14:14:55.589603 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:55.584740    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 14:14:55.589603 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.584745    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 14:14:55.589603 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.584859    2581 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 11 14:14:55.589603 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.587254    2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 11 14:14:55.589603 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.588266    2581 server.go:1019] "Starting client certificate rotation"
Sep 11 14:14:55.589603 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.588364    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 14:14:55.589603 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.589196    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 14:14:55.591840 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.591827    2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 14:14:55.592958 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.592939    2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 14:14:55.604108 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.604086    2581 log.go:25] "Validated CRI v1 runtime API"
Sep 11 14:14:55.607455 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.607438    2581 log.go:25] "Validated CRI v1 image API"
Sep 11 14:14:55.609073 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.609056    2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 11 14:14:55.612578 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.612428    2581 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 c1a1e50a-f967-42ba-a354-142d43599a70:/dev/nvme0n1p4 f58cd860-085a-4f2e-9133-aff36370ffd7:/dev/nvme0n1p3]
Sep 11 14:14:55.612643 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.612577    2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 11 14:14:55.616754 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.616642    2581 manager.go:216] Machine: {Timestamp:2026-09-11 14:14:55.616170994 +0000 UTC m=+0.360785921 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3084173 MemoryCapacity:33164468224 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c4a2a5fd476f255535420e70cc932 SystemUUID:ec2c4a2a-5fd4-76f2-5553-5420e70cc932 BootID:682bcfac-6070-4f4b-98d0-68b48a2c12b5 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582234112 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:67:5b:ef:4a:69 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:67:5b:ef:4a:69 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:42:9c:8b:d9:4c:61 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164468224 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 11 14:14:55.616754 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.616748    2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 11 14:14:55.616866 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.616828    2581 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 11 14:14:55.620309 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.620277    2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 11 14:14:55.620450 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.620309    2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-150.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 11 14:14:55.621175 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.621164    2581 topology_manager.go:138] "Creating topology manager with none policy"
Sep 11 14:14:55.621214 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.621177    2581 container_manager_linux.go:309] "Creating device plugin manager"
Sep 11 14:14:55.621214 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.621185    2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 14:14:55.622127 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.622116    2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 14:14:55.623335 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.623324    2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 11 14:14:55.625534 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.625514    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 14:14:55.628274 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.628258    2581 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 14:14:55.628397 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.628388    2581 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 11 14:14:55.628429 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.628420    2581 kubelet.go:486] "Attempting to sync node with API server"
Sep 11 14:14:55.628470 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.628430    2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 11 14:14:55.628470 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.628444    2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 11 14:14:55.628470 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.628454    2581 kubelet.go:398] "Adding apiserver pod source"
Sep 11 14:14:55.628470 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.628463    2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 11 14:14:55.628847 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.628835    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 14:14:55.628887 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.628859    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 14:14:55.631265 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.631243    2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 11 14:14:55.634354 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.634340    2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 11 14:14:55.634397 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.634371    2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 11 14:14:55.637064 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.637049    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 11 14:14:55.637108 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.637072    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 11 14:14:55.637108 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.637078    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 11 14:14:55.637108 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.637084    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 11 14:14:55.637108 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.637093    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 11 14:14:55.637108 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.637099    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 11 14:14:55.637250 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.637114    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 11 14:14:55.637250 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.637120    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 11 14:14:55.637250 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.637127    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 11 14:14:55.637250 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.637132    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 11 14:14:55.637250 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.637146    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 11 14:14:55.637250 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.637152    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 11 14:14:55.638076 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.638066    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 11 14:14:55.638109 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.638078    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 11 14:14:55.641876 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.641855    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-150.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 14:14:55.641959 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:14:55.641882    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-150.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 11 14:14:55.641959 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:14:55.641883    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 11 14:14:55.642253 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.642241    2581 server.go:1268] "Started kubelet"
Sep 11 14:14:55.642331 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.642307    2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 11 14:14:55.642380 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.642319    2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 11 14:14:55.642425 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.642385    2581 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 11 14:14:55.645979 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.645874    2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 11 14:14:55.646269 ip-10-0-135-150 systemd[1]: Started Kubernetes Kubelet.
Sep 11 14:14:55.647397 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:14:55.647372    2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 11 14:14:55.648478 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.648089    2581 server.go:310] "Adding debug handlers to kubelet server"
Sep 11 14:14:55.653064 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:14:55.651973    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-150.ec2.internal.18d449ccbb956297  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-150.ec2.internal,UID:ip-10-0-135-150.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-150.ec2.internal,},FirstTimestamp:2026-09-11 14:14:55.642215063 +0000 UTC m=+0.386829990,LastTimestamp:2026-09-11 14:14:55.642215063 +0000 UTC m=+0.386829990,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-150.ec2.internal,}"
Sep 11 14:14:55.654871 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.654846    2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 11 14:14:55.654871 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.654862    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 11 14:14:55.655025 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:14:55.654991    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-150.ec2.internal\" not found"
Sep 11 14:14:55.655571 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.655554    2581 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 11 14:14:55.655571 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.655573    2581 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 11 14:14:55.655701 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.655586    2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 11 14:14:55.655701 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.655660    2581 reconstruct.go:97] "Volume reconstruction finished"
Sep 11 14:14:55.655701 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.655666    2581 reconciler.go:29] "Reconciler: start to sync state"
Sep 11 14:14:55.655890 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.655872    2581 factory.go:55] Registering systemd factory
Sep 11 14:14:55.655890 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.655893    2581 factory.go:223] Registration of the systemd container factory successfully
Sep 11 14:14:55.656146 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.656130    2581 factory.go:153] Registering CRI-O factory
Sep 11 14:14:55.656146 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.656149    2581 factory.go:223] Registration of the crio container factory successfully
Sep 11 14:14:55.656226 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.656192    2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 11 14:14:55.656226 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.656209    2581 factory.go:103] Registering Raw factory
Sep 11 14:14:55.656291 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.656228    2581 manager.go:1196] Started watching for new ooms in manager
Sep 11 14:14:55.656608 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.656597    2581 manager.go:349] Starting recovery of all containers
Sep 11 14:14:55.660353 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:14:55.660323    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 11 14:14:55.660495 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:14:55.660471    2581 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-150.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 11 14:14:55.666270 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.666243    2581 manager.go:354] Recovery completed
Sep 11 14:14:55.672058 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.672046    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 14:14:55.673633 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.673616    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-150.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 14:14:55.673713 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.673651    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-150.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 14:14:55.673713 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.673666    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-150.ec2.internal" event="NodeHasSufficientPID"
Sep 11 14:14:55.674241 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.674226    2581 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 11 14:14:55.674289 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.674242    2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 11 14:14:55.674289 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.674260    2581 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 14:14:55.674289 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.674252    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 14:14:55.675154 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.675139    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-150.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 14:14:55.675227 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.675170    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-150.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 14:14:55.675227 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.675185    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-150.ec2.internal" event="NodeHasSufficientPID"
Sep 11 14:14:55.676011 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:14:55.675925    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-150.ec2.internal.18d449ccbd74cac2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-150.ec2.internal,UID:ip-10-0-135-150.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-150.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-150.ec2.internal,},FirstTimestamp:2026-09-11 14:14:55.673633474 +0000 UTC m=+0.418248405,LastTimestamp:2026-09-11 14:14:55.673633474 +0000 UTC m=+0.418248405,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-150.ec2.internal,}"
Sep 11 14:14:55.677641 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.677159    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vstzz"
Sep 11 14:14:55.685814 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.685796    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vstzz"
Sep 11 14:14:55.685915 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:14:55.685793    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-150.ec2.internal.18d449ccbd752bfa  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-150.ec2.internal,UID:ip-10-0-135-150.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-150.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-150.ec2.internal,},FirstTimestamp:2026-09-11 14:14:55.673658362 +0000 UTC m=+0.418273292,LastTimestamp:2026-09-11 14:14:55.673658362 +0000 UTC m=+0.418273292,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-150.ec2.internal,}"
Sep 11 14:14:55.755849 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:14:55.755806    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-150.ec2.internal\" not found"
Sep 11 14:14:55.822964 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.821994    2581 policy_none.go:49] "None policy: Start"
Sep 11 14:14:55.822964 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.822023    2581 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 11 14:14:55.822964 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.822035    2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 11 14:14:55.824003 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.823989    2581 policy_none.go:47] "Start"
Sep 11 14:14:55.855452 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.855438    2581 manager.go:337] "Starting Device Plugin manager"
Sep 11 14:14:55.855557 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:14:55.855476    2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 11 14:14:55.855557 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.855486    2581 server.go:87] "Starting device plugin registration server"
Sep 11 14:14:55.855836 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.855825    2581 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 11 14:14:55.855875 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.855838    2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 11 14:14:55.855978 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.855951    2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 11 14:14:55.856050 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.856038    2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 11 14:14:55.856050 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.856047    2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 11 14:14:55.856565 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:14:55.856541    2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 11 14:14:55.856659 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:14:55.856599    2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-150.ec2.internal\" not found"
Sep 11 14:14:55.867701 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:14:55.867684    2581 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-135-150.ec2.internal\" not found" node="ip-10-0-135-150.ec2.internal"
Sep 11 14:14:55.897419 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.897385    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 11 14:14:55.898674 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.898654    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 11 14:14:55.898764 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.898706    2581 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 11 14:14:55.898764 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.898729    2581 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 11 14:14:55.898861 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:14:55.898774    2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 11 14:14:55.901709 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.901691    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 14:14:55.956189 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.956154    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 14:14:55.957149 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.957126    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-150.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 14:14:55.957217 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.957156    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-150.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 14:14:55.957217 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.957167    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-150.ec2.internal" event="NodeHasSufficientPID"
Sep 11 14:14:55.957217 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.957191    2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-150.ec2.internal"
Sep 11 14:14:55.967405 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.967390    2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-150.ec2.internal"
Sep 11 14:14:55.967405 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:14:55.967411    2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-150.ec2.internal\": node \"ip-10-0-135-150.ec2.internal\" not found"
Sep 11 14:14:55.999634 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.999605    2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-150.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-150.ec2.internal"]
Sep 11 14:14:55.999733 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:55.999677    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 14:14:56.000992 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:56.000979    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-150.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 14:14:56.001068 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:56.001005    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-150.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 14:14:56.001068 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:56.001014    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-150.ec2.internal" event="NodeHasSufficientPID"
Sep 11 14:14:56.003215 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:56.003202    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 14:14:56.003333 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:56.003315    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-150.ec2.internal"
Sep 11 14:14:56.003381 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:56.003353    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 14:14:56.003886 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:56.003872    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-150.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 14:14:56.004005 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:56.003894    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-150.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 14:14:56.004005 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:56.003875    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-150.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 14:14:56.004005 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:56.003935    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-150.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 14:14:56.004005 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:56.003948    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-150.ec2.internal" event="NodeHasSufficientPID"
Sep 11 14:14:56.004005 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:56.003915    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-150.ec2.internal" event="NodeHasSufficientPID"
Sep 11 14:14:56.005725 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:14:56.005708    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-150.ec2.internal\" not found"
Sep 11 14:14:56.006062 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:56.006048    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-150.ec2.internal"
Sep 11 14:14:56.006125 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:56.006072    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 14:14:56.006700 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:56.006684    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-150.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 14:14:56.006780 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:56.006713    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-150.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 14:14:56.006780 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:56.006726    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-150.ec2.internal" event="NodeHasSufficientPID"
Sep 11 14:14:56.021461 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:14:56.021443    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-150.ec2.internal\" not found" node="ip-10-0-135-150.ec2.internal"
Sep 11 14:14:56.025208 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:14:56.025192    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-150.ec2.internal\" not found" node="ip-10-0-135-150.ec2.internal"
Sep 11 14:14:56.056937 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:56.056896    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/87337b80b4395cecaecd8c7eb49f5684-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-150.ec2.internal\" (UID: \"87337b80b4395cecaecd8c7eb49f5684\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-150.ec2.internal"
Sep 11 14:14:56.057045 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:56.056939    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87337b80b4395cecaecd8c7eb49f5684-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-150.ec2.internal\" (UID: \"87337b80b4395cecaecd8c7eb49f5684\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-150.ec2.internal"
Sep 11 14:14:56.057045 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:56.056956    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9bd85cdb5fbcf445a1156f9ccc86edcd-config\") pod \"kube-apiserver-proxy-ip-10-0-135-150.ec2.internal\" (UID: \"9bd85cdb5fbcf445a1156f9ccc86edcd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-150.ec2.internal"
Sep 11 14:14:56.106056 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:14:56.105971    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-150.ec2.internal\" not found"
Sep 11 14:14:56.157517 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:56.157482    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/87337b80b4395cecaecd8c7eb49f5684-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-150.ec2.internal\" (UID: \"87337b80b4395cecaecd8c7eb49f5684\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-150.ec2.internal"
Sep 11 14:14:56.157517 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:56.157520    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87337b80b4395cecaecd8c7eb49f5684-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-150.ec2.internal\" (UID: \"87337b80b4395cecaecd8c7eb49f5684\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-150.ec2.internal"
Sep 11 14:14:56.157727 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:56.157552    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9bd85cdb5fbcf445a1156f9ccc86edcd-config\") pod \"kube-apiserver-proxy-ip-10-0-135-150.ec2.internal\" (UID: \"9bd85cdb5fbcf445a1156f9ccc86edcd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-150.ec2.internal"
Sep 11 14:14:56.157727 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:56.157594    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9bd85cdb5fbcf445a1156f9ccc86edcd-config\") pod \"kube-apiserver-proxy-ip-10-0-135-150.ec2.internal\" (UID: \"9bd85cdb5fbcf445a1156f9ccc86edcd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-150.ec2.internal"
Sep 11 14:14:56.157727 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:56.157594    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87337b80b4395cecaecd8c7eb49f5684-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-150.ec2.internal\" (UID: \"87337b80b4395cecaecd8c7eb49f5684\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-150.ec2.internal"
Sep 11 14:14:56.157727 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:56.157594    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/87337b80b4395cecaecd8c7eb49f5684-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-150.ec2.internal\" (UID: \"87337b80b4395cecaecd8c7eb49f5684\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-150.ec2.internal"
Sep 11 14:14:56.206651 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:14:56.206621    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-150.ec2.internal\" not found"
Sep 11 14:14:57.901203 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:14:57.901170    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-150.ec2.internal\" not found"
Sep 11 14:14:57.918353 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:57.918328    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-150.ec2.internal"
Sep 11 14:14:57.921436 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:57.921417    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-150.ec2.internal"
Sep 11 14:14:58.002274 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:14:58.002189    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-150.ec2.internal\" not found"
Sep 11 14:14:58.102809 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:14:58.102772    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-150.ec2.internal\" not found"
Sep 11 14:14:58.144944 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:58.144912    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 14:14:58.181894 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:58.181864    2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 11 14:14:58.182341 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:58.182030    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 14:14:58.182341 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:58.182051    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 14:14:58.203119 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:14:58.203090    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-150.ec2.internal\" not found"
Sep 11 14:14:58.286306 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:58.286212    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 14:09:57 +0000 UTC" deadline="2028-06-15 02:35:16.817570882 +0000 UTC"
Sep 11 14:14:58.286306 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:58.286247    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15420h20m18.531327478s"
Sep 11 14:14:58.303389 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:14:58.303366    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-150.ec2.internal\" not found"
Sep 11 14:14:58.342757 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:58.342731    2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 14:14:58.387170 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:58.387146    2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 14:14:58.449663 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:58.449484    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-150.ec2.internal"
Sep 11 14:14:58.460536 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:58.460511    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 14:14:58.461360 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:58.461343    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-150.ec2.internal"
Sep 11 14:14:58.485528 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:58.485501    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 14:14:58.538536 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:58.538468    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87337b80b4395cecaecd8c7eb49f5684.slice/crio-485187baec14e961f583570f9aa64df40cb4d50dce52692c190401480551ee97 WatchSource:0}: Error finding container 485187baec14e961f583570f9aa64df40cb4d50dce52692c190401480551ee97: Status 404 returned error can't find the container with id 485187baec14e961f583570f9aa64df40cb4d50dce52692c190401480551ee97
Sep 11 14:14:58.538727 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:14:58.538704    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bd85cdb5fbcf445a1156f9ccc86edcd.slice/crio-0049bbc7ff24d3207fa5f8806b8ebeb1bb8d0d5e608822c3949f1d42c1c08d83 WatchSource:0}: Error finding container 0049bbc7ff24d3207fa5f8806b8ebeb1bb8d0d5e608822c3949f1d42c1c08d83: Status 404 returned error can't find the container with id 0049bbc7ff24d3207fa5f8806b8ebeb1bb8d0d5e608822c3949f1d42c1c08d83
Sep 11 14:14:58.542234 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:58.542207    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 14:14:59.155411 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.155378    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 14:14:59.224579 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.224321    2581 apiserver.go:52] "Watching apiserver"
Sep 11 14:14:59.233349 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.233323    2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 11 14:14:59.235344 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.235317    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-pnxhk","kube-system/kube-apiserver-proxy-ip-10-0-135-150.ec2.internal","openshift-dns/node-resolver-wk6kn","openshift-multus/network-metrics-daemon-wgmt4","openshift-network-diagnostics/network-check-target-q5sxd","openshift-ovn-kubernetes/ovnkube-node-n88qj","kube-system/global-pull-secret-syncer-dqmtt","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx7nv","openshift-cluster-node-tuning-operator/tuned-b7jh5","openshift-image-registry/node-ca-qdthk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-150.ec2.internal","openshift-multus/multus-2cdd6","openshift-multus/multus-additional-cni-plugins-mn2mg","openshift-network-operator/iptables-alerter-v285k"]
Sep 11 14:14:59.237872 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.237847    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqmtt"
Sep 11 14:14:59.238002 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:14:59.237932    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqmtt" podUID="78d2df0e-635a-4123-9a80-f9898562e613"
Sep 11 14:14:59.240111 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.240066    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mn2mg"
Sep 11 14:14:59.242267 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.242246    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wk6kn"
Sep 11 14:14:59.243069 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.242906    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 11 14:14:59.243069 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.243059    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hnk2c\""
Sep 11 14:14:59.244766 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.244636    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wgmt4"
Sep 11 14:14:59.244766 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:14:59.244716    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wgmt4" podUID="d3376794-b10e-4353-b8f3-4639741019eb"
Sep 11 14:14:59.245355 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.245199    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 11 14:14:59.245355 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.245234    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 11 14:14:59.245355 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.245249    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 11 14:14:59.245355 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.245251    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 11 14:14:59.245663 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.245493    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 11 14:14:59.246805 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.246760    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-npqv2\""
Sep 11 14:14:59.246805 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.246774    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 11 14:14:59.248797 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.248773    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5sxd"
Sep 11 14:14:59.248893 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:14:59.248854    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5sxd" podUID="f58cdeac-ef49-4a6e-b459-ad21493d1497"
Sep 11 14:14:59.249376 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.249357    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 11 14:14:59.251600 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.251555    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.253592 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.253573    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-b7jh5"
Sep 11 14:14:59.254326 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.254308    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 11 14:14:59.255708 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.255388    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 11 14:14:59.255708 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.255606    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 11 14:14:59.256725 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.256706    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 11 14:14:59.256950 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.256935    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 11 14:14:59.257218 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.257202    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 11 14:14:59.257277 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.257213    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9m29t\""
Sep 11 14:14:59.257478 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.257462    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 11 14:14:59.257714 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.257667    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 11 14:14:59.257897 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.257881    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-65s69\""
Sep 11 14:14:59.259396 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.258619    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pnxhk"
Sep 11 14:14:59.260220 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.260196    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2wd7\" (UniqueName: \"kubernetes.io/projected/774c0511-8141-47cd-8718-b647be205839-kube-api-access-v2wd7\") pod \"multus-additional-cni-plugins-mn2mg\" (UID: \"774c0511-8141-47cd-8718-b647be205839\") " pod="openshift-multus/multus-additional-cni-plugins-mn2mg"
Sep 11 14:14:59.260318 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.260237    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/757db884-d01a-4ab4-8b46-f9a41a3d7369-tmp-dir\") pod \"node-resolver-wk6kn\" (UID: \"757db884-d01a-4ab4-8b46-f9a41a3d7369\") " pod="openshift-dns/node-resolver-wk6kn"
Sep 11 14:14:59.260318 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.260271    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fedc3186-f028-4964-a5e9-d6315fb07164-run-openvswitch\") pod \"ovnkube-node-n88qj\" (UID: \"fedc3186-f028-4964-a5e9-d6315fb07164\") " pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.260318 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.260294    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fedc3186-f028-4964-a5e9-d6315fb07164-host-cni-bin\") pod \"ovnkube-node-n88qj\" (UID: \"fedc3186-f028-4964-a5e9-d6315fb07164\") " pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.260318 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.260316    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fedc3186-f028-4964-a5e9-d6315fb07164-host-cni-netd\") pod \"ovnkube-node-n88qj\" (UID: \"fedc3186-f028-4964-a5e9-d6315fb07164\") " pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.260519 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.260339    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/774c0511-8141-47cd-8718-b647be205839-system-cni-dir\") pod \"multus-additional-cni-plugins-mn2mg\" (UID: \"774c0511-8141-47cd-8718-b647be205839\") " pod="openshift-multus/multus-additional-cni-plugins-mn2mg"
Sep 11 14:14:59.260519 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.260362    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/774c0511-8141-47cd-8718-b647be205839-cni-binary-copy\") pod \"multus-additional-cni-plugins-mn2mg\" (UID: \"774c0511-8141-47cd-8718-b647be205839\") " pod="openshift-multus/multus-additional-cni-plugins-mn2mg"
Sep 11 14:14:59.260519 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.260386    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/774c0511-8141-47cd-8718-b647be205839-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mn2mg\" (UID: \"774c0511-8141-47cd-8718-b647be205839\") " pod="openshift-multus/multus-additional-cni-plugins-mn2mg"
Sep 11 14:14:59.260519 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.260410    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpqcm\" (UniqueName: \"kubernetes.io/projected/757db884-d01a-4ab4-8b46-f9a41a3d7369-kube-api-access-jpqcm\") pod \"node-resolver-wk6kn\" (UID: \"757db884-d01a-4ab4-8b46-f9a41a3d7369\") " pod="openshift-dns/node-resolver-wk6kn"
Sep 11 14:14:59.260519 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.260432    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fedc3186-f028-4964-a5e9-d6315fb07164-systemd-units\") pod \"ovnkube-node-n88qj\" (UID: \"fedc3186-f028-4964-a5e9-d6315fb07164\") " pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.260519 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.260483    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fedc3186-f028-4964-a5e9-d6315fb07164-ovn-node-metrics-cert\") pod \"ovnkube-node-n88qj\" (UID: \"fedc3186-f028-4964-a5e9-d6315fb07164\") " pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.260790 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.260517    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3376794-b10e-4353-b8f3-4639741019eb-metrics-certs\") pod \"network-metrics-daemon-wgmt4\" (UID: \"d3376794-b10e-4353-b8f3-4639741019eb\") " pod="openshift-multus/network-metrics-daemon-wgmt4"
Sep 11 14:14:59.260790 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.260543    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fedc3186-f028-4964-a5e9-d6315fb07164-var-lib-openvswitch\") pod \"ovnkube-node-n88qj\" (UID: \"fedc3186-f028-4964-a5e9-d6315fb07164\") " pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.260790 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.260567    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2r68\" (UniqueName: \"kubernetes.io/projected/fedc3186-f028-4964-a5e9-d6315fb07164-kube-api-access-c2r68\") pod \"ovnkube-node-n88qj\" (UID: \"fedc3186-f028-4964-a5e9-d6315fb07164\") " pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.260790 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.260589    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fedc3186-f028-4964-a5e9-d6315fb07164-env-overrides\") pod \"ovnkube-node-n88qj\" (UID: \"fedc3186-f028-4964-a5e9-d6315fb07164\") " pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.260790 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.260611    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ffkb\" (UniqueName: \"kubernetes.io/projected/f58cdeac-ef49-4a6e-b459-ad21493d1497-kube-api-access-5ffkb\") pod \"network-check-target-q5sxd\" (UID: \"f58cdeac-ef49-4a6e-b459-ad21493d1497\") " pod="openshift-network-diagnostics/network-check-target-q5sxd"
Sep 11 14:14:59.260790 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.260634    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fedc3186-f028-4964-a5e9-d6315fb07164-host-slash\") pod \"ovnkube-node-n88qj\" (UID: \"fedc3186-f028-4964-a5e9-d6315fb07164\") " pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.260790 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.260665    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fedc3186-f028-4964-a5e9-d6315fb07164-host-run-netns\") pod \"ovnkube-node-n88qj\" (UID: \"fedc3186-f028-4964-a5e9-d6315fb07164\") " pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.260790 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.260691    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fedc3186-f028-4964-a5e9-d6315fb07164-run-ovn\") pod \"ovnkube-node-n88qj\" (UID: \"fedc3186-f028-4964-a5e9-d6315fb07164\") " pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.260790 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.260713    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fedc3186-f028-4964-a5e9-d6315fb07164-log-socket\") pod \"ovnkube-node-n88qj\" (UID: \"fedc3186-f028-4964-a5e9-d6315fb07164\") " pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.260790 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.260737    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fedc3186-f028-4964-a5e9-d6315fb07164-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n88qj\" (UID: \"fedc3186-f028-4964-a5e9-d6315fb07164\") " pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.260790 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.260761    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/78d2df0e-635a-4123-9a80-f9898562e613-dbus\") pod \"global-pull-secret-syncer-dqmtt\" (UID: \"78d2df0e-635a-4123-9a80-f9898562e613\") " pod="kube-system/global-pull-secret-syncer-dqmtt"
Sep 11 14:14:59.260790 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.260783    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fedc3186-f028-4964-a5e9-d6315fb07164-node-log\") pod \"ovnkube-node-n88qj\" (UID: \"fedc3186-f028-4964-a5e9-d6315fb07164\") " pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.261350 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.260806    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fedc3186-f028-4964-a5e9-d6315fb07164-host-run-ovn-kubernetes\") pod \"ovnkube-node-n88qj\" (UID: \"fedc3186-f028-4964-a5e9-d6315fb07164\") " pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.261350 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.260831    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/78d2df0e-635a-4123-9a80-f9898562e613-original-pull-secret\") pod \"global-pull-secret-syncer-dqmtt\" (UID: \"78d2df0e-635a-4123-9a80-f9898562e613\") " pod="kube-system/global-pull-secret-syncer-dqmtt"
Sep 11 14:14:59.261350 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.260853    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fedc3186-f028-4964-a5e9-d6315fb07164-etc-openvswitch\") pod \"ovnkube-node-n88qj\" (UID: \"fedc3186-f028-4964-a5e9-d6315fb07164\") " pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.261350 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.260876    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/78d2df0e-635a-4123-9a80-f9898562e613-kubelet-config\") pod \"global-pull-secret-syncer-dqmtt\" (UID: \"78d2df0e-635a-4123-9a80-f9898562e613\") " pod="kube-system/global-pull-secret-syncer-dqmtt"
Sep 11 14:14:59.261350 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.260909    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/774c0511-8141-47cd-8718-b647be205839-cnibin\") pod \"multus-additional-cni-plugins-mn2mg\" (UID: \"774c0511-8141-47cd-8718-b647be205839\") " pod="openshift-multus/multus-additional-cni-plugins-mn2mg"
Sep 11 14:14:59.261350 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.260931    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/774c0511-8141-47cd-8718-b647be205839-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mn2mg\" (UID: \"774c0511-8141-47cd-8718-b647be205839\") " pod="openshift-multus/multus-additional-cni-plugins-mn2mg"
Sep 11 14:14:59.261350 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.260955    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/774c0511-8141-47cd-8718-b647be205839-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mn2mg\" (UID: \"774c0511-8141-47cd-8718-b647be205839\") " pod="openshift-multus/multus-additional-cni-plugins-mn2mg"
Sep 11 14:14:59.261350 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.261009    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/757db884-d01a-4ab4-8b46-f9a41a3d7369-hosts-file\") pod \"node-resolver-wk6kn\" (UID: \"757db884-d01a-4ab4-8b46-f9a41a3d7369\") " pod="openshift-dns/node-resolver-wk6kn"
Sep 11 14:14:59.261350 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.261032    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fedc3186-f028-4964-a5e9-d6315fb07164-host-kubelet\") pod \"ovnkube-node-n88qj\" (UID: \"fedc3186-f028-4964-a5e9-d6315fb07164\") " pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.261350 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.261055    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fedc3186-f028-4964-a5e9-d6315fb07164-ovnkube-config\") pod \"ovnkube-node-n88qj\" (UID: \"fedc3186-f028-4964-a5e9-d6315fb07164\") " pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.261350 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.261076    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fedc3186-f028-4964-a5e9-d6315fb07164-ovnkube-script-lib\") pod \"ovnkube-node-n88qj\" (UID: \"fedc3186-f028-4964-a5e9-d6315fb07164\") " pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.261350 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.261103    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbp69\" (UniqueName: \"kubernetes.io/projected/d3376794-b10e-4353-b8f3-4639741019eb-kube-api-access-jbp69\") pod \"network-metrics-daemon-wgmt4\" (UID: \"d3376794-b10e-4353-b8f3-4639741019eb\") " pod="openshift-multus/network-metrics-daemon-wgmt4"
Sep 11 14:14:59.261350 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.261133    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fedc3186-f028-4964-a5e9-d6315fb07164-run-systemd\") pod \"ovnkube-node-n88qj\" (UID: \"fedc3186-f028-4964-a5e9-d6315fb07164\") " pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.261935 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.261848    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-vkn2t\""
Sep 11 14:14:59.262211 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.262196    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 11 14:14:59.262389 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.262344    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 11 14:14:59.262389 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.262383    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-v285k"
Sep 11 14:14:59.264889 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.264851    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qdthk"
Sep 11 14:14:59.265605 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.265442    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 11 14:14:59.265605 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.265550    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 11 14:14:59.265605 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.265574    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 11 14:14:59.265788 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.265662    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5tx9z\""
Sep 11 14:14:59.267710 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.267659    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx7nv"
Sep 11 14:14:59.267789 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.267716    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 11 14:14:59.267789 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.267716    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 11 14:14:59.268215 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.268151    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 11 14:14:59.268487 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.268467    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4fxn9\""
Sep 11 14:14:59.269768 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.269732    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2cdd6"
Sep 11 14:14:59.270731 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.270710    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 11 14:14:59.270962 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.270931    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 11 14:14:59.271215 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.271197    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nwt69\""
Sep 11 14:14:59.271300 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.271202    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 11 14:14:59.272299 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.272281    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-m548t\""
Sep 11 14:14:59.272518 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.272499    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 11 14:14:59.275611 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.275593    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 14:14:59.299179 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.299137    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-4mw4w"
Sep 11 14:14:59.309930 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.309910    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4mw4w"
Sep 11 14:14:59.351135 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.351112    2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 11 14:14:59.361904 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.361859    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fedc3186-f028-4964-a5e9-d6315fb07164-systemd-units\") pod \"ovnkube-node-n88qj\" (UID: \"fedc3186-f028-4964-a5e9-d6315fb07164\") " pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.361904 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.361904    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fedc3186-f028-4964-a5e9-d6315fb07164-ovn-node-metrics-cert\") pod \"ovnkube-node-n88qj\" (UID: \"fedc3186-f028-4964-a5e9-d6315fb07164\") " pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.362145 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.361935    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks678\" (UniqueName: \"kubernetes.io/projected/b86ff2d3-8990-4b7f-b243-744fe2351991-kube-api-access-ks678\") pod \"tuned-b7jh5\" (UID: \"b86ff2d3-8990-4b7f-b243-744fe2351991\") " pod="openshift-cluster-node-tuning-operator/tuned-b7jh5"
Sep 11 14:14:59.362145 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.361961    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z72bb\" (UniqueName: \"kubernetes.io/projected/2b4e57a8-379d-4375-95b4-8176b1e4e410-kube-api-access-z72bb\") pod \"node-ca-qdthk\" (UID: \"2b4e57a8-379d-4375-95b4-8176b1e4e410\") " pod="openshift-image-registry/node-ca-qdthk"
Sep 11 14:14:59.362145 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.362000    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d3509467-2ab1-4e02-9654-71fb22f361e3-cni-binary-copy\") pod \"multus-2cdd6\" (UID: \"d3509467-2ab1-4e02-9654-71fb22f361e3\") " pod="openshift-multus/multus-2cdd6"
Sep 11 14:14:59.362145 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.362022    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d3509467-2ab1-4e02-9654-71fb22f361e3-multus-daemon-config\") pod \"multus-2cdd6\" (UID: \"d3509467-2ab1-4e02-9654-71fb22f361e3\") " pod="openshift-multus/multus-2cdd6"
Sep 11 14:14:59.362145 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.362048    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3376794-b10e-4353-b8f3-4639741019eb-metrics-certs\") pod \"network-metrics-daemon-wgmt4\" (UID: \"d3376794-b10e-4353-b8f3-4639741019eb\") " pod="openshift-multus/network-metrics-daemon-wgmt4"
Sep 11 14:14:59.362145 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.362074    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fedc3186-f028-4964-a5e9-d6315fb07164-var-lib-openvswitch\") pod \"ovnkube-node-n88qj\" (UID: \"fedc3186-f028-4964-a5e9-d6315fb07164\") " pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.362145 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.362100    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2r68\" (UniqueName: \"kubernetes.io/projected/fedc3186-f028-4964-a5e9-d6315fb07164-kube-api-access-c2r68\") pod \"ovnkube-node-n88qj\" (UID: \"fedc3186-f028-4964-a5e9-d6315fb07164\") " pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.362145 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.362123    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/07633f0c-6e0a-4bc2-811f-263730bfb6a4-iptables-alerter-script\") pod \"iptables-alerter-v285k\" (UID: \"07633f0c-6e0a-4bc2-811f-263730bfb6a4\") " pod="openshift-network-operator/iptables-alerter-v285k"
Sep 11 14:14:59.362145 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.362147    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d3509467-2ab1-4e02-9654-71fb22f361e3-hostroot\") pod \"multus-2cdd6\" (UID: \"d3509467-2ab1-4e02-9654-71fb22f361e3\") " pod="openshift-multus/multus-2cdd6"
Sep 11 14:14:59.362564 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.362170    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fedc3186-f028-4964-a5e9-d6315fb07164-host-slash\") pod \"ovnkube-node-n88qj\" (UID: \"fedc3186-f028-4964-a5e9-d6315fb07164\") " pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.362564 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.362193    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fedc3186-f028-4964-a5e9-d6315fb07164-run-ovn\") pod \"ovnkube-node-n88qj\" (UID: \"fedc3186-f028-4964-a5e9-d6315fb07164\") " pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.362564 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.362215    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fedc3186-f028-4964-a5e9-d6315fb07164-log-socket\") pod \"ovnkube-node-n88qj\" (UID: \"fedc3186-f028-4964-a5e9-d6315fb07164\") " pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.362564 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.362238    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/78d2df0e-635a-4123-9a80-f9898562e613-dbus\") pod \"global-pull-secret-syncer-dqmtt\" (UID: \"78d2df0e-635a-4123-9a80-f9898562e613\") " pod="kube-system/global-pull-secret-syncer-dqmtt"
Sep 11 14:14:59.362564 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.362262    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/78d2df0e-635a-4123-9a80-f9898562e613-original-pull-secret\") pod \"global-pull-secret-syncer-dqmtt\" (UID: \"78d2df0e-635a-4123-9a80-f9898562e613\") " pod="kube-system/global-pull-secret-syncer-dqmtt"
Sep 11 14:14:59.362564 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.362286    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b86ff2d3-8990-4b7f-b243-744fe2351991-etc-systemd\") pod \"tuned-b7jh5\" (UID: \"b86ff2d3-8990-4b7f-b243-744fe2351991\") " pod="openshift-cluster-node-tuning-operator/tuned-b7jh5"
Sep 11 14:14:59.362564 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.362317    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d3509467-2ab1-4e02-9654-71fb22f361e3-multus-socket-dir-parent\") pod \"multus-2cdd6\" (UID: \"d3509467-2ab1-4e02-9654-71fb22f361e3\") " pod="openshift-multus/multus-2cdd6"
Sep 11 14:14:59.362564 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.362342    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtz8s\" (UniqueName: \"kubernetes.io/projected/d3509467-2ab1-4e02-9654-71fb22f361e3-kube-api-access-dtz8s\") pod \"multus-2cdd6\" (UID: \"d3509467-2ab1-4e02-9654-71fb22f361e3\") " pod="openshift-multus/multus-2cdd6"
Sep 11 14:14:59.362564 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.362368    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fedc3186-f028-4964-a5e9-d6315fb07164-host-run-ovn-kubernetes\") pod \"ovnkube-node-n88qj\" (UID: \"fedc3186-f028-4964-a5e9-d6315fb07164\") " pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.362564 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.362391    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9eadc143-6ef8-4518-8291-b323e6f15cb3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qx7nv\" (UID: \"9eadc143-6ef8-4518-8291-b323e6f15cb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx7nv"
Sep 11 14:14:59.362564 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.362417    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/78d2df0e-635a-4123-9a80-f9898562e613-kubelet-config\") pod \"global-pull-secret-syncer-dqmtt\" (UID: \"78d2df0e-635a-4123-9a80-f9898562e613\") " pod="kube-system/global-pull-secret-syncer-dqmtt"
Sep 11 14:14:59.362564 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.362441    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/774c0511-8141-47cd-8718-b647be205839-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mn2mg\" (UID: \"774c0511-8141-47cd-8718-b647be205839\") " pod="openshift-multus/multus-additional-cni-plugins-mn2mg"
Sep 11 14:14:59.362564 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.362467    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2b4e57a8-379d-4375-95b4-8176b1e4e410-host\") pod \"node-ca-qdthk\" (UID: \"2b4e57a8-379d-4375-95b4-8176b1e4e410\") " pod="openshift-image-registry/node-ca-qdthk"
Sep 11 14:14:59.362564 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.362498    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d3509467-2ab1-4e02-9654-71fb22f361e3-host-var-lib-cni-multus\") pod \"multus-2cdd6\" (UID: \"d3509467-2ab1-4e02-9654-71fb22f361e3\") " pod="openshift-multus/multus-2cdd6"
Sep 11 14:14:59.362564 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.362527    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fedc3186-f028-4964-a5e9-d6315fb07164-ovnkube-config\") pod \"ovnkube-node-n88qj\" (UID: \"fedc3186-f028-4964-a5e9-d6315fb07164\") " pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.362564 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.362555    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b86ff2d3-8990-4b7f-b243-744fe2351991-var-lib-kubelet\") pod \"tuned-b7jh5\" (UID: \"b86ff2d3-8990-4b7f-b243-744fe2351991\") " pod="openshift-cluster-node-tuning-operator/tuned-b7jh5"
Sep 11 14:14:59.363933 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.362579    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b86ff2d3-8990-4b7f-b243-744fe2351991-tmp\") pod \"tuned-b7jh5\" (UID: \"b86ff2d3-8990-4b7f-b243-744fe2351991\") " pod="openshift-cluster-node-tuning-operator/tuned-b7jh5"
Sep 11 14:14:59.363933 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.362576    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fedc3186-f028-4964-a5e9-d6315fb07164-systemd-units\") pod \"ovnkube-node-n88qj\" (UID: \"fedc3186-f028-4964-a5e9-d6315fb07164\") " pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.363933 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.362626    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fedc3186-f028-4964-a5e9-d6315fb07164-run-systemd\") pod \"ovnkube-node-n88qj\" (UID: \"fedc3186-f028-4964-a5e9-d6315fb07164\") " pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.363933 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.362762    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fedc3186-f028-4964-a5e9-d6315fb07164-host-slash\") pod \"ovnkube-node-n88qj\" (UID: \"fedc3186-f028-4964-a5e9-d6315fb07164\") " pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.363933 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.362758    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fedc3186-f028-4964-a5e9-d6315fb07164-var-lib-openvswitch\") pod \"ovnkube-node-n88qj\" (UID: \"fedc3186-f028-4964-a5e9-d6315fb07164\") " pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.363933 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.362787    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcxp8\" (UniqueName: \"kubernetes.io/projected/9eadc143-6ef8-4518-8291-b323e6f15cb3-kube-api-access-qcxp8\") pod \"aws-ebs-csi-driver-node-qx7nv\" (UID: \"9eadc143-6ef8-4518-8291-b323e6f15cb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx7nv"
Sep 11 14:14:59.363933 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.362942    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/78d2df0e-635a-4123-9a80-f9898562e613-dbus\") pod \"global-pull-secret-syncer-dqmtt\" (UID: \"78d2df0e-635a-4123-9a80-f9898562e613\") " pod="kube-system/global-pull-secret-syncer-dqmtt"
Sep 11 14:14:59.363933 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:14:59.362952    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 14:14:59.363933 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.362980    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fedc3186-f028-4964-a5e9-d6315fb07164-run-systemd\") pod \"ovnkube-node-n88qj\" (UID: \"fedc3186-f028-4964-a5e9-d6315fb07164\") " pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.363933 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:14:59.363047    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78d2df0e-635a-4123-9a80-f9898562e613-original-pull-secret podName:78d2df0e-635a-4123-9a80-f9898562e613 nodeName:}" failed. No retries permitted until 2026-09-11 14:14:59.863019901 +0000 UTC m=+3.013569962 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/78d2df0e-635a-4123-9a80-f9898562e613-original-pull-secret") pod "global-pull-secret-syncer-dqmtt" (UID: "78d2df0e-635a-4123-9a80-f9898562e613") : object "kube-system"/"original-pull-secret" not registered
Sep 11 14:14:59.363933 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:14:59.363124    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 14:14:59.363933 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.363216    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fedc3186-f028-4964-a5e9-d6315fb07164-run-ovn\") pod \"ovnkube-node-n88qj\" (UID: \"fedc3186-f028-4964-a5e9-d6315fb07164\") " pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.363933 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.363253    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fedc3186-f028-4964-a5e9-d6315fb07164-log-socket\") pod \"ovnkube-node-n88qj\" (UID: \"fedc3186-f028-4964-a5e9-d6315fb07164\") " pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.363933 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.363290    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fedc3186-f028-4964-a5e9-d6315fb07164-host-run-ovn-kubernetes\") pod \"ovnkube-node-n88qj\" (UID: \"fedc3186-f028-4964-a5e9-d6315fb07164\") " pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.363933 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:14:59.363313    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3376794-b10e-4353-b8f3-4639741019eb-metrics-certs podName:d3376794-b10e-4353-b8f3-4639741019eb nodeName:}" failed. No retries permitted until 2026-09-11 14:14:59.863297148 +0000 UTC m=+3.013847221 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d3376794-b10e-4353-b8f3-4639741019eb-metrics-certs") pod "network-metrics-daemon-wgmt4" (UID: "d3376794-b10e-4353-b8f3-4639741019eb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 14:14:59.363933 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.363338    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d3509467-2ab1-4e02-9654-71fb22f361e3-host-run-multus-certs\") pod \"multus-2cdd6\" (UID: \"d3509467-2ab1-4e02-9654-71fb22f361e3\") " pod="openshift-multus/multus-2cdd6"
Sep 11 14:14:59.363933 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.363376    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fedc3186-f028-4964-a5e9-d6315fb07164-run-openvswitch\") pod \"ovnkube-node-n88qj\" (UID: \"fedc3186-f028-4964-a5e9-d6315fb07164\") " pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.364740 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.363403    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/78d2df0e-635a-4123-9a80-f9898562e613-kubelet-config\") pod \"global-pull-secret-syncer-dqmtt\" (UID: \"78d2df0e-635a-4123-9a80-f9898562e613\") " pod="kube-system/global-pull-secret-syncer-dqmtt"
Sep 11 14:14:59.364740 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.363420    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fedc3186-f028-4964-a5e9-d6315fb07164-host-cni-bin\") pod \"ovnkube-node-n88qj\" (UID: \"fedc3186-f028-4964-a5e9-d6315fb07164\") " pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.364740 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.363455    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fedc3186-f028-4964-a5e9-d6315fb07164-run-openvswitch\") pod \"ovnkube-node-n88qj\" (UID: \"fedc3186-f028-4964-a5e9-d6315fb07164\") " pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.364740 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.363497    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/774c0511-8141-47cd-8718-b647be205839-cni-binary-copy\") pod \"multus-additional-cni-plugins-mn2mg\" (UID: \"774c0511-8141-47cd-8718-b647be205839\") " pod="openshift-multus/multus-additional-cni-plugins-mn2mg"
Sep 11 14:14:59.364740 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.363522    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fedc3186-f028-4964-a5e9-d6315fb07164-host-cni-bin\") pod \"ovnkube-node-n88qj\" (UID: \"fedc3186-f028-4964-a5e9-d6315fb07164\") " pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.364740 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.363523    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fedc3186-f028-4964-a5e9-d6315fb07164-ovnkube-config\") pod \"ovnkube-node-n88qj\" (UID: \"fedc3186-f028-4964-a5e9-d6315fb07164\") " pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.364740 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.363563    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/774c0511-8141-47cd-8718-b647be205839-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mn2mg\" (UID: \"774c0511-8141-47cd-8718-b647be205839\") " pod="openshift-multus/multus-additional-cni-plugins-mn2mg"
Sep 11 14:14:59.364740 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.363702    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9eadc143-6ef8-4518-8291-b323e6f15cb3-sys-fs\") pod \"aws-ebs-csi-driver-node-qx7nv\" (UID: \"9eadc143-6ef8-4518-8291-b323e6f15cb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx7nv"
Sep 11 14:14:59.364740 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.363735    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d3509467-2ab1-4e02-9654-71fb22f361e3-host-run-netns\") pod \"multus-2cdd6\" (UID: \"d3509467-2ab1-4e02-9654-71fb22f361e3\") " pod="openshift-multus/multus-2cdd6"
Sep 11 14:14:59.364740 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.363758    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3509467-2ab1-4e02-9654-71fb22f361e3-host-var-lib-kubelet\") pod \"multus-2cdd6\" (UID: \"d3509467-2ab1-4e02-9654-71fb22f361e3\") " pod="openshift-multus/multus-2cdd6"
Sep 11 14:14:59.364740 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.363788    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b86ff2d3-8990-4b7f-b243-744fe2351991-etc-sysctl-conf\") pod \"tuned-b7jh5\" (UID: \"b86ff2d3-8990-4b7f-b243-744fe2351991\") " pod="openshift-cluster-node-tuning-operator/tuned-b7jh5"
Sep 11 14:14:59.364740 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.363790    2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 11 14:14:59.364740 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.363836    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b86ff2d3-8990-4b7f-b243-744fe2351991-run\") pod \"tuned-b7jh5\" (UID: \"b86ff2d3-8990-4b7f-b243-744fe2351991\") " pod="openshift-cluster-node-tuning-operator/tuned-b7jh5"
Sep 11 14:14:59.364740 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.363858    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b86ff2d3-8990-4b7f-b243-744fe2351991-sys\") pod \"tuned-b7jh5\" (UID: \"b86ff2d3-8990-4b7f-b243-744fe2351991\") " pod="openshift-cluster-node-tuning-operator/tuned-b7jh5"
Sep 11 14:14:59.364740 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.363885    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/774c0511-8141-47cd-8718-b647be205839-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mn2mg\" (UID: \"774c0511-8141-47cd-8718-b647be205839\") " pod="openshift-multus/multus-additional-cni-plugins-mn2mg"
Sep 11 14:14:59.364740 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.363893    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d3509467-2ab1-4e02-9654-71fb22f361e3-host-run-k8s-cni-cncf-io\") pod \"multus-2cdd6\" (UID: \"d3509467-2ab1-4e02-9654-71fb22f361e3\") " pod="openshift-multus/multus-2cdd6"
Sep 11 14:14:59.364740 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.363950    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b86ff2d3-8990-4b7f-b243-744fe2351991-etc-kubernetes\") pod \"tuned-b7jh5\" (UID: \"b86ff2d3-8990-4b7f-b243-744fe2351991\") " pod="openshift-cluster-node-tuning-operator/tuned-b7jh5"
Sep 11 14:14:59.365562 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.363991    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b86ff2d3-8990-4b7f-b243-744fe2351991-host\") pod \"tuned-b7jh5\" (UID: \"b86ff2d3-8990-4b7f-b243-744fe2351991\") " pod="openshift-cluster-node-tuning-operator/tuned-b7jh5"
Sep 11 14:14:59.365562 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.364040    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/07633f0c-6e0a-4bc2-811f-263730bfb6a4-host-slash\") pod \"iptables-alerter-v285k\" (UID: \"07633f0c-6e0a-4bc2-811f-263730bfb6a4\") " pod="openshift-network-operator/iptables-alerter-v285k"
Sep 11 14:14:59.365562 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.364079    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9eadc143-6ef8-4518-8291-b323e6f15cb3-registration-dir\") pod \"aws-ebs-csi-driver-node-qx7nv\" (UID: \"9eadc143-6ef8-4518-8291-b323e6f15cb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx7nv"
Sep 11 14:14:59.365562 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.364143    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9eadc143-6ef8-4518-8291-b323e6f15cb3-device-dir\") pod \"aws-ebs-csi-driver-node-qx7nv\" (UID: \"9eadc143-6ef8-4518-8291-b323e6f15cb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx7nv"
Sep 11 14:14:59.365562 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.364306    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fedc3186-f028-4964-a5e9-d6315fb07164-env-overrides\") pod \"ovnkube-node-n88qj\" (UID: \"fedc3186-f028-4964-a5e9-d6315fb07164\") " pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.365562 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.364362    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ffkb\" (UniqueName: \"kubernetes.io/projected/f58cdeac-ef49-4a6e-b459-ad21493d1497-kube-api-access-5ffkb\") pod \"network-check-target-q5sxd\" (UID: \"f58cdeac-ef49-4a6e-b459-ad21493d1497\") " pod="openshift-network-diagnostics/network-check-target-q5sxd"
Sep 11 14:14:59.365562 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.364423    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fedc3186-f028-4964-a5e9-d6315fb07164-host-run-netns\") pod \"ovnkube-node-n88qj\" (UID: \"fedc3186-f028-4964-a5e9-d6315fb07164\") " pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.365562 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.364450    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fedc3186-f028-4964-a5e9-d6315fb07164-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n88qj\" (UID: \"fedc3186-f028-4964-a5e9-d6315fb07164\") " pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.365562 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.364514    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/774c0511-8141-47cd-8718-b647be205839-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mn2mg\" (UID: \"774c0511-8141-47cd-8718-b647be205839\") " pod="openshift-multus/multus-additional-cni-plugins-mn2mg"
Sep 11 14:14:59.365562 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.364528    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c81645fc-5b10-41bb-9908-7512097108e8-agent-certs\") pod \"konnectivity-agent-pnxhk\" (UID: \"c81645fc-5b10-41bb-9908-7512097108e8\") " pod="kube-system/konnectivity-agent-pnxhk"
Sep 11 14:14:59.365562 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.364528    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/774c0511-8141-47cd-8718-b647be205839-cni-binary-copy\") pod \"multus-additional-cni-plugins-mn2mg\" (UID: \"774c0511-8141-47cd-8718-b647be205839\") " pod="openshift-multus/multus-additional-cni-plugins-mn2mg"
Sep 11 14:14:59.365562 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.364568    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fedc3186-f028-4964-a5e9-d6315fb07164-host-run-netns\") pod \"ovnkube-node-n88qj\" (UID: \"fedc3186-f028-4964-a5e9-d6315fb07164\") " pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.365562 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.364565    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fedc3186-f028-4964-a5e9-d6315fb07164-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n88qj\" (UID: \"fedc3186-f028-4964-a5e9-d6315fb07164\") " pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.365562 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.364626    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fedc3186-f028-4964-a5e9-d6315fb07164-node-log\") pod \"ovnkube-node-n88qj\" (UID: \"fedc3186-f028-4964-a5e9-d6315fb07164\") " pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.365562 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.364660    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fedc3186-f028-4964-a5e9-d6315fb07164-etc-openvswitch\") pod \"ovnkube-node-n88qj\" (UID: \"fedc3186-f028-4964-a5e9-d6315fb07164\") " pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.365562 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.364703    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fedc3186-f028-4964-a5e9-d6315fb07164-node-log\") pod \"ovnkube-node-n88qj\" (UID: \"fedc3186-f028-4964-a5e9-d6315fb07164\") " pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.365562 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.364735    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/774c0511-8141-47cd-8718-b647be205839-cnibin\") pod \"multus-additional-cni-plugins-mn2mg\" (UID: \"774c0511-8141-47cd-8718-b647be205839\") " pod="openshift-multus/multus-additional-cni-plugins-mn2mg"
Sep 11 14:14:59.366383 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.364768    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/774c0511-8141-47cd-8718-b647be205839-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mn2mg\" (UID: \"774c0511-8141-47cd-8718-b647be205839\") " pod="openshift-multus/multus-additional-cni-plugins-mn2mg"
Sep 11 14:14:59.366383 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.364811    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fedc3186-f028-4964-a5e9-d6315fb07164-etc-openvswitch\") pod \"ovnkube-node-n88qj\" (UID: \"fedc3186-f028-4964-a5e9-d6315fb07164\") " pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.366383 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.364809    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fedc3186-f028-4964-a5e9-d6315fb07164-env-overrides\") pod \"ovnkube-node-n88qj\" (UID: \"fedc3186-f028-4964-a5e9-d6315fb07164\") " pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.366383 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.364842    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b86ff2d3-8990-4b7f-b243-744fe2351991-etc-sysconfig\") pod \"tuned-b7jh5\" (UID: \"b86ff2d3-8990-4b7f-b243-744fe2351991\") " pod="openshift-cluster-node-tuning-operator/tuned-b7jh5"
Sep 11 14:14:59.366383 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.364898    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/774c0511-8141-47cd-8718-b647be205839-cnibin\") pod \"multus-additional-cni-plugins-mn2mg\" (UID: \"774c0511-8141-47cd-8718-b647be205839\") " pod="openshift-multus/multus-additional-cni-plugins-mn2mg"
Sep 11 14:14:59.366383 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.364954    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b86ff2d3-8990-4b7f-b243-744fe2351991-etc-sysctl-d\") pod \"tuned-b7jh5\" (UID: \"b86ff2d3-8990-4b7f-b243-744fe2351991\") " pod="openshift-cluster-node-tuning-operator/tuned-b7jh5"
Sep 11 14:14:59.366383 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.364990    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/774c0511-8141-47cd-8718-b647be205839-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mn2mg\" (UID: \"774c0511-8141-47cd-8718-b647be205839\") " pod="openshift-multus/multus-additional-cni-plugins-mn2mg"
Sep 11 14:14:59.366383 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.365004    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b86ff2d3-8990-4b7f-b243-744fe2351991-etc-tuned\") pod \"tuned-b7jh5\" (UID: \"b86ff2d3-8990-4b7f-b243-744fe2351991\") " pod="openshift-cluster-node-tuning-operator/tuned-b7jh5"
Sep 11 14:14:59.366383 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.365034    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c81645fc-5b10-41bb-9908-7512097108e8-konnectivity-ca\") pod \"konnectivity-agent-pnxhk\" (UID: \"c81645fc-5b10-41bb-9908-7512097108e8\") " pod="kube-system/konnectivity-agent-pnxhk"
Sep 11 14:14:59.366383 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.365072    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ksr2\" (UniqueName: \"kubernetes.io/projected/07633f0c-6e0a-4bc2-811f-263730bfb6a4-kube-api-access-5ksr2\") pod \"iptables-alerter-v285k\" (UID: \"07633f0c-6e0a-4bc2-811f-263730bfb6a4\") " pod="openshift-network-operator/iptables-alerter-v285k"
Sep 11 14:14:59.366383 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.365111    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/757db884-d01a-4ab4-8b46-f9a41a3d7369-hosts-file\") pod \"node-resolver-wk6kn\" (UID: \"757db884-d01a-4ab4-8b46-f9a41a3d7369\") " pod="openshift-dns/node-resolver-wk6kn"
Sep 11 14:14:59.366383 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.365146    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fedc3186-f028-4964-a5e9-d6315fb07164-host-kubelet\") pod \"ovnkube-node-n88qj\" (UID: \"fedc3186-f028-4964-a5e9-d6315fb07164\") " pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.366383 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.365170    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2b4e57a8-379d-4375-95b4-8176b1e4e410-serviceca\") pod \"node-ca-qdthk\" (UID: \"2b4e57a8-379d-4375-95b4-8176b1e4e410\") " pod="openshift-image-registry/node-ca-qdthk"
Sep 11 14:14:59.366383 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.365195    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9eadc143-6ef8-4518-8291-b323e6f15cb3-socket-dir\") pod \"aws-ebs-csi-driver-node-qx7nv\" (UID: \"9eadc143-6ef8-4518-8291-b323e6f15cb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx7nv"
Sep 11 14:14:59.366383 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.365214    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/757db884-d01a-4ab4-8b46-f9a41a3d7369-hosts-file\") pod \"node-resolver-wk6kn\" (UID: \"757db884-d01a-4ab4-8b46-f9a41a3d7369\") " pod="openshift-dns/node-resolver-wk6kn"
Sep 11 14:14:59.366383 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.365226    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fedc3186-f028-4964-a5e9-d6315fb07164-host-kubelet\") pod \"ovnkube-node-n88qj\" (UID: \"fedc3186-f028-4964-a5e9-d6315fb07164\") " pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.366383 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.365256    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9eadc143-6ef8-4518-8291-b323e6f15cb3-etc-selinux\") pod \"aws-ebs-csi-driver-node-qx7nv\" (UID: \"9eadc143-6ef8-4518-8291-b323e6f15cb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx7nv"
Sep 11 14:14:59.367181 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.365281    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d3509467-2ab1-4e02-9654-71fb22f361e3-system-cni-dir\") pod \"multus-2cdd6\" (UID: \"d3509467-2ab1-4e02-9654-71fb22f361e3\") " pod="openshift-multus/multus-2cdd6"
Sep 11 14:14:59.367181 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.365305    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d3509467-2ab1-4e02-9654-71fb22f361e3-host-var-lib-cni-bin\") pod \"multus-2cdd6\" (UID: \"d3509467-2ab1-4e02-9654-71fb22f361e3\") " pod="openshift-multus/multus-2cdd6"
Sep 11 14:14:59.367181 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.365343    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fedc3186-f028-4964-a5e9-d6315fb07164-ovnkube-script-lib\") pod \"ovnkube-node-n88qj\" (UID: \"fedc3186-f028-4964-a5e9-d6315fb07164\") " pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.367181 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.365390    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbp69\" (UniqueName: \"kubernetes.io/projected/d3376794-b10e-4353-b8f3-4639741019eb-kube-api-access-jbp69\") pod \"network-metrics-daemon-wgmt4\" (UID: \"d3376794-b10e-4353-b8f3-4639741019eb\") " pod="openshift-multus/network-metrics-daemon-wgmt4"
Sep 11 14:14:59.367181 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.365422    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2wd7\" (UniqueName: \"kubernetes.io/projected/774c0511-8141-47cd-8718-b647be205839-kube-api-access-v2wd7\") pod \"multus-additional-cni-plugins-mn2mg\" (UID: \"774c0511-8141-47cd-8718-b647be205839\") " pod="openshift-multus/multus-additional-cni-plugins-mn2mg"
Sep 11 14:14:59.367181 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.365504    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b86ff2d3-8990-4b7f-b243-744fe2351991-etc-modprobe-d\") pod \"tuned-b7jh5\" (UID: \"b86ff2d3-8990-4b7f-b243-744fe2351991\") " pod="openshift-cluster-node-tuning-operator/tuned-b7jh5"
Sep 11 14:14:59.367181 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.365533    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d3509467-2ab1-4e02-9654-71fb22f361e3-multus-conf-dir\") pod \"multus-2cdd6\" (UID: \"d3509467-2ab1-4e02-9654-71fb22f361e3\") " pod="openshift-multus/multus-2cdd6"
Sep 11 14:14:59.367181 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.365573    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/757db884-d01a-4ab4-8b46-f9a41a3d7369-tmp-dir\") pod \"node-resolver-wk6kn\" (UID: \"757db884-d01a-4ab4-8b46-f9a41a3d7369\") " pod="openshift-dns/node-resolver-wk6kn"
Sep 11 14:14:59.367181 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.365597    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fedc3186-f028-4964-a5e9-d6315fb07164-host-cni-netd\") pod \"ovnkube-node-n88qj\" (UID: \"fedc3186-f028-4964-a5e9-d6315fb07164\") " pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.367181 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.365622    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/774c0511-8141-47cd-8718-b647be205839-system-cni-dir\") pod \"multus-additional-cni-plugins-mn2mg\" (UID: \"774c0511-8141-47cd-8718-b647be205839\") " pod="openshift-multus/multus-additional-cni-plugins-mn2mg"
Sep 11 14:14:59.367181 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.365660    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b86ff2d3-8990-4b7f-b243-744fe2351991-lib-modules\") pod \"tuned-b7jh5\" (UID: \"b86ff2d3-8990-4b7f-b243-744fe2351991\") " pod="openshift-cluster-node-tuning-operator/tuned-b7jh5"
Sep 11 14:14:59.367181 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.365682    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d3509467-2ab1-4e02-9654-71fb22f361e3-multus-cni-dir\") pod \"multus-2cdd6\" (UID: \"d3509467-2ab1-4e02-9654-71fb22f361e3\") " pod="openshift-multus/multus-2cdd6"
Sep 11 14:14:59.367181 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.365742    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d3509467-2ab1-4e02-9654-71fb22f361e3-cnibin\") pod \"multus-2cdd6\" (UID: \"d3509467-2ab1-4e02-9654-71fb22f361e3\") " pod="openshift-multus/multus-2cdd6"
Sep 11 14:14:59.367181 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.365773    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3509467-2ab1-4e02-9654-71fb22f361e3-etc-kubernetes\") pod \"multus-2cdd6\" (UID: \"d3509467-2ab1-4e02-9654-71fb22f361e3\") " pod="openshift-multus/multus-2cdd6"
Sep 11 14:14:59.367181 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.365800    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpqcm\" (UniqueName: \"kubernetes.io/projected/757db884-d01a-4ab4-8b46-f9a41a3d7369-kube-api-access-jpqcm\") pod \"node-resolver-wk6kn\" (UID: \"757db884-d01a-4ab4-8b46-f9a41a3d7369\") " pod="openshift-dns/node-resolver-wk6kn"
Sep 11 14:14:59.367181 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.365849    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fedc3186-f028-4964-a5e9-d6315fb07164-ovnkube-script-lib\") pod \"ovnkube-node-n88qj\" (UID: \"fedc3186-f028-4964-a5e9-d6315fb07164\") " pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.367181 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.365880    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/774c0511-8141-47cd-8718-b647be205839-system-cni-dir\") pod \"multus-additional-cni-plugins-mn2mg\" (UID: \"774c0511-8141-47cd-8718-b647be205839\") " pod="openshift-multus/multus-additional-cni-plugins-mn2mg"
Sep 11 14:14:59.368052 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.365914    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fedc3186-f028-4964-a5e9-d6315fb07164-host-cni-netd\") pod \"ovnkube-node-n88qj\" (UID: \"fedc3186-f028-4964-a5e9-d6315fb07164\") " pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.368052 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.366099    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/757db884-d01a-4ab4-8b46-f9a41a3d7369-tmp-dir\") pod \"node-resolver-wk6kn\" (UID: \"757db884-d01a-4ab4-8b46-f9a41a3d7369\") " pod="openshift-dns/node-resolver-wk6kn"
Sep 11 14:14:59.368593 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.368568    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fedc3186-f028-4964-a5e9-d6315fb07164-ovn-node-metrics-cert\") pod \"ovnkube-node-n88qj\" (UID: \"fedc3186-f028-4964-a5e9-d6315fb07164\") " pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.373573 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:14:59.373307    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 14:14:59.373573 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:14:59.373359    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 14:14:59.373573 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:14:59.373371    2581 projected.go:196] Error preparing data for projected volume kube-api-access-5ffkb for pod openshift-network-diagnostics/network-check-target-q5sxd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 14:14:59.373573 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:14:59.373426    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f58cdeac-ef49-4a6e-b459-ad21493d1497-kube-api-access-5ffkb podName:f58cdeac-ef49-4a6e-b459-ad21493d1497 nodeName:}" failed. No retries permitted until 2026-09-11 14:14:59.873409374 +0000 UTC m=+3.023959418 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5ffkb" (UniqueName: "kubernetes.io/projected/f58cdeac-ef49-4a6e-b459-ad21493d1497-kube-api-access-5ffkb") pod "network-check-target-q5sxd" (UID: "f58cdeac-ef49-4a6e-b459-ad21493d1497") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 14:14:59.374537 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.374515    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2r68\" (UniqueName: \"kubernetes.io/projected/fedc3186-f028-4964-a5e9-d6315fb07164-kube-api-access-c2r68\") pod \"ovnkube-node-n88qj\" (UID: \"fedc3186-f028-4964-a5e9-d6315fb07164\") " pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.381468 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.381428    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbp69\" (UniqueName: \"kubernetes.io/projected/d3376794-b10e-4353-b8f3-4639741019eb-kube-api-access-jbp69\") pod \"network-metrics-daemon-wgmt4\" (UID: \"d3376794-b10e-4353-b8f3-4639741019eb\") " pod="openshift-multus/network-metrics-daemon-wgmt4"
Sep 11 14:14:59.382705 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.382667    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2wd7\" (UniqueName: \"kubernetes.io/projected/774c0511-8141-47cd-8718-b647be205839-kube-api-access-v2wd7\") pod \"multus-additional-cni-plugins-mn2mg\" (UID: \"774c0511-8141-47cd-8718-b647be205839\") " pod="openshift-multus/multus-additional-cni-plugins-mn2mg"
Sep 11 14:14:59.383351 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.383327    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpqcm\" (UniqueName: \"kubernetes.io/projected/757db884-d01a-4ab4-8b46-f9a41a3d7369-kube-api-access-jpqcm\") pod \"node-resolver-wk6kn\" (UID: \"757db884-d01a-4ab4-8b46-f9a41a3d7369\") " pod="openshift-dns/node-resolver-wk6kn"
Sep 11 14:14:59.467195 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.467112    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b86ff2d3-8990-4b7f-b243-744fe2351991-etc-sysconfig\") pod \"tuned-b7jh5\" (UID: \"b86ff2d3-8990-4b7f-b243-744fe2351991\") " pod="openshift-cluster-node-tuning-operator/tuned-b7jh5"
Sep 11 14:14:59.467195 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.467152    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b86ff2d3-8990-4b7f-b243-744fe2351991-etc-sysctl-d\") pod \"tuned-b7jh5\" (UID: \"b86ff2d3-8990-4b7f-b243-744fe2351991\") " pod="openshift-cluster-node-tuning-operator/tuned-b7jh5"
Sep 11 14:14:59.467195 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.467172    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b86ff2d3-8990-4b7f-b243-744fe2351991-etc-tuned\") pod \"tuned-b7jh5\" (UID: \"b86ff2d3-8990-4b7f-b243-744fe2351991\") " pod="openshift-cluster-node-tuning-operator/tuned-b7jh5"
Sep 11 14:14:59.467195 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.467195    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c81645fc-5b10-41bb-9908-7512097108e8-konnectivity-ca\") pod \"konnectivity-agent-pnxhk\" (UID: \"c81645fc-5b10-41bb-9908-7512097108e8\") " pod="kube-system/konnectivity-agent-pnxhk"
Sep 11 14:14:59.467528 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.467219    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ksr2\" (UniqueName: \"kubernetes.io/projected/07633f0c-6e0a-4bc2-811f-263730bfb6a4-kube-api-access-5ksr2\") pod \"iptables-alerter-v285k\" (UID: \"07633f0c-6e0a-4bc2-811f-263730bfb6a4\") " pod="openshift-network-operator/iptables-alerter-v285k"
Sep 11 14:14:59.467528 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.467237    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b86ff2d3-8990-4b7f-b243-744fe2351991-etc-sysconfig\") pod \"tuned-b7jh5\" (UID: \"b86ff2d3-8990-4b7f-b243-744fe2351991\") " pod="openshift-cluster-node-tuning-operator/tuned-b7jh5"
Sep 11 14:14:59.467528 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.467243    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2b4e57a8-379d-4375-95b4-8176b1e4e410-serviceca\") pod \"node-ca-qdthk\" (UID: \"2b4e57a8-379d-4375-95b4-8176b1e4e410\") " pod="openshift-image-registry/node-ca-qdthk"
Sep 11 14:14:59.467528 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.467296    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9eadc143-6ef8-4518-8291-b323e6f15cb3-socket-dir\") pod \"aws-ebs-csi-driver-node-qx7nv\" (UID: \"9eadc143-6ef8-4518-8291-b323e6f15cb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx7nv"
Sep 11 14:14:59.467528 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.467314    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9eadc143-6ef8-4518-8291-b323e6f15cb3-etc-selinux\") pod \"aws-ebs-csi-driver-node-qx7nv\" (UID: \"9eadc143-6ef8-4518-8291-b323e6f15cb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx7nv"
Sep 11 14:14:59.467528 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.467329    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d3509467-2ab1-4e02-9654-71fb22f361e3-system-cni-dir\") pod \"multus-2cdd6\" (UID: \"d3509467-2ab1-4e02-9654-71fb22f361e3\") " pod="openshift-multus/multus-2cdd6"
Sep 11 14:14:59.467528 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.467343    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d3509467-2ab1-4e02-9654-71fb22f361e3-host-var-lib-cni-bin\") pod \"multus-2cdd6\" (UID: \"d3509467-2ab1-4e02-9654-71fb22f361e3\") " pod="openshift-multus/multus-2cdd6"
Sep 11 14:14:59.467528 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.467361    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b86ff2d3-8990-4b7f-b243-744fe2351991-etc-modprobe-d\") pod \"tuned-b7jh5\" (UID: \"b86ff2d3-8990-4b7f-b243-744fe2351991\") " pod="openshift-cluster-node-tuning-operator/tuned-b7jh5"
Sep 11 14:14:59.467528 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.467374    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d3509467-2ab1-4e02-9654-71fb22f361e3-multus-conf-dir\") pod \"multus-2cdd6\" (UID: \"d3509467-2ab1-4e02-9654-71fb22f361e3\") " pod="openshift-multus/multus-2cdd6"
Sep 11 14:14:59.467528 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.467384    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b86ff2d3-8990-4b7f-b243-744fe2351991-etc-sysctl-d\") pod \"tuned-b7jh5\" (UID: \"b86ff2d3-8990-4b7f-b243-744fe2351991\") " pod="openshift-cluster-node-tuning-operator/tuned-b7jh5"
Sep 11 14:14:59.467528 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.467393    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b86ff2d3-8990-4b7f-b243-744fe2351991-lib-modules\") pod \"tuned-b7jh5\" (UID: \"b86ff2d3-8990-4b7f-b243-744fe2351991\") " pod="openshift-cluster-node-tuning-operator/tuned-b7jh5"
Sep 11 14:14:59.467528 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.467407    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d3509467-2ab1-4e02-9654-71fb22f361e3-multus-cni-dir\") pod \"multus-2cdd6\" (UID: \"d3509467-2ab1-4e02-9654-71fb22f361e3\") " pod="openshift-multus/multus-2cdd6"
Sep 11 14:14:59.467528 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.467422    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d3509467-2ab1-4e02-9654-71fb22f361e3-cnibin\") pod \"multus-2cdd6\" (UID: \"d3509467-2ab1-4e02-9654-71fb22f361e3\") " pod="openshift-multus/multus-2cdd6"
Sep 11 14:14:59.467528 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.467436    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3509467-2ab1-4e02-9654-71fb22f361e3-etc-kubernetes\") pod \"multus-2cdd6\" (UID: \"d3509467-2ab1-4e02-9654-71fb22f361e3\") " pod="openshift-multus/multus-2cdd6"
Sep 11 14:14:59.467528 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.467453    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks678\" (UniqueName: \"kubernetes.io/projected/b86ff2d3-8990-4b7f-b243-744fe2351991-kube-api-access-ks678\") pod \"tuned-b7jh5\" (UID: \"b86ff2d3-8990-4b7f-b243-744fe2351991\") " pod="openshift-cluster-node-tuning-operator/tuned-b7jh5"
Sep 11 14:14:59.467528 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.467467    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z72bb\" (UniqueName: \"kubernetes.io/projected/2b4e57a8-379d-4375-95b4-8176b1e4e410-kube-api-access-z72bb\") pod \"node-ca-qdthk\" (UID: \"2b4e57a8-379d-4375-95b4-8176b1e4e410\") " pod="openshift-image-registry/node-ca-qdthk"
Sep 11 14:14:59.467528 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.467483    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d3509467-2ab1-4e02-9654-71fb22f361e3-cni-binary-copy\") pod \"multus-2cdd6\" (UID: \"d3509467-2ab1-4e02-9654-71fb22f361e3\") " pod="openshift-multus/multus-2cdd6"
Sep 11 14:14:59.467528 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.467496    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d3509467-2ab1-4e02-9654-71fb22f361e3-multus-daemon-config\") pod \"multus-2cdd6\" (UID: \"d3509467-2ab1-4e02-9654-71fb22f361e3\") " pod="openshift-multus/multus-2cdd6"
Sep 11 14:14:59.468407 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.467521    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/07633f0c-6e0a-4bc2-811f-263730bfb6a4-iptables-alerter-script\") pod \"iptables-alerter-v285k\" (UID: \"07633f0c-6e0a-4bc2-811f-263730bfb6a4\") " pod="openshift-network-operator/iptables-alerter-v285k"
Sep 11 14:14:59.468407 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.467536    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d3509467-2ab1-4e02-9654-71fb22f361e3-hostroot\") pod \"multus-2cdd6\" (UID: \"d3509467-2ab1-4e02-9654-71fb22f361e3\") " pod="openshift-multus/multus-2cdd6"
Sep 11 14:14:59.468407 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.467559    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b86ff2d3-8990-4b7f-b243-744fe2351991-etc-systemd\") pod \"tuned-b7jh5\" (UID: \"b86ff2d3-8990-4b7f-b243-744fe2351991\") " pod="openshift-cluster-node-tuning-operator/tuned-b7jh5"
Sep 11 14:14:59.468407 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.467574    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d3509467-2ab1-4e02-9654-71fb22f361e3-multus-socket-dir-parent\") pod \"multus-2cdd6\" (UID: \"d3509467-2ab1-4e02-9654-71fb22f361e3\") " pod="openshift-multus/multus-2cdd6"
Sep 11 14:14:59.468407 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.467589    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtz8s\" (UniqueName: \"kubernetes.io/projected/d3509467-2ab1-4e02-9654-71fb22f361e3-kube-api-access-dtz8s\") pod \"multus-2cdd6\" (UID: \"d3509467-2ab1-4e02-9654-71fb22f361e3\") " pod="openshift-multus/multus-2cdd6"
Sep 11 14:14:59.468407 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.467605    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9eadc143-6ef8-4518-8291-b323e6f15cb3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qx7nv\" (UID: \"9eadc143-6ef8-4518-8291-b323e6f15cb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx7nv"
Sep 11 14:14:59.468407 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.467621    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2b4e57a8-379d-4375-95b4-8176b1e4e410-host\") pod \"node-ca-qdthk\" (UID: \"2b4e57a8-379d-4375-95b4-8176b1e4e410\") " pod="openshift-image-registry/node-ca-qdthk"
Sep 11 14:14:59.468407 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.467635    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d3509467-2ab1-4e02-9654-71fb22f361e3-host-var-lib-cni-multus\") pod \"multus-2cdd6\" (UID: \"d3509467-2ab1-4e02-9654-71fb22f361e3\") " pod="openshift-multus/multus-2cdd6"
Sep 11 14:14:59.468407 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.467652    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b86ff2d3-8990-4b7f-b243-744fe2351991-var-lib-kubelet\") pod \"tuned-b7jh5\" (UID: \"b86ff2d3-8990-4b7f-b243-744fe2351991\") " pod="openshift-cluster-node-tuning-operator/tuned-b7jh5"
Sep 11 14:14:59.468407 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.467667    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b86ff2d3-8990-4b7f-b243-744fe2351991-tmp\") pod \"tuned-b7jh5\" (UID: \"b86ff2d3-8990-4b7f-b243-744fe2351991\") " pod="openshift-cluster-node-tuning-operator/tuned-b7jh5"
Sep 11 14:14:59.468407 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.467683    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcxp8\" (UniqueName: \"kubernetes.io/projected/9eadc143-6ef8-4518-8291-b323e6f15cb3-kube-api-access-qcxp8\") pod \"aws-ebs-csi-driver-node-qx7nv\" (UID: \"9eadc143-6ef8-4518-8291-b323e6f15cb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx7nv"
Sep 11 14:14:59.468407 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.467698    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d3509467-2ab1-4e02-9654-71fb22f361e3-host-run-multus-certs\") pod \"multus-2cdd6\" (UID: \"d3509467-2ab1-4e02-9654-71fb22f361e3\") " pod="openshift-multus/multus-2cdd6"
Sep 11 14:14:59.468407 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.467717    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9eadc143-6ef8-4518-8291-b323e6f15cb3-sys-fs\") pod \"aws-ebs-csi-driver-node-qx7nv\" (UID: \"9eadc143-6ef8-4518-8291-b323e6f15cb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx7nv"
Sep 11 14:14:59.468407 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.467732    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d3509467-2ab1-4e02-9654-71fb22f361e3-host-run-netns\") pod \"multus-2cdd6\" (UID: \"d3509467-2ab1-4e02-9654-71fb22f361e3\") " pod="openshift-multus/multus-2cdd6"
Sep 11 14:14:59.468407 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.467754    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3509467-2ab1-4e02-9654-71fb22f361e3-host-var-lib-kubelet\") pod \"multus-2cdd6\" (UID: \"d3509467-2ab1-4e02-9654-71fb22f361e3\") " pod="openshift-multus/multus-2cdd6"
Sep 11 14:14:59.468407 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.467768    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b86ff2d3-8990-4b7f-b243-744fe2351991-etc-sysctl-conf\") pod \"tuned-b7jh5\" (UID: \"b86ff2d3-8990-4b7f-b243-744fe2351991\") " pod="openshift-cluster-node-tuning-operator/tuned-b7jh5"
Sep 11 14:14:59.468407 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.467785    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b86ff2d3-8990-4b7f-b243-744fe2351991-run\") pod \"tuned-b7jh5\" (UID: \"b86ff2d3-8990-4b7f-b243-744fe2351991\") " pod="openshift-cluster-node-tuning-operator/tuned-b7jh5"
Sep 11 14:14:59.468407 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.467799    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b86ff2d3-8990-4b7f-b243-744fe2351991-sys\") pod \"tuned-b7jh5\" (UID: \"b86ff2d3-8990-4b7f-b243-744fe2351991\") " pod="openshift-cluster-node-tuning-operator/tuned-b7jh5"
Sep 11 14:14:59.469277 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.467813    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d3509467-2ab1-4e02-9654-71fb22f361e3-host-run-k8s-cni-cncf-io\") pod \"multus-2cdd6\" (UID: \"d3509467-2ab1-4e02-9654-71fb22f361e3\") " pod="openshift-multus/multus-2cdd6"
Sep 11 14:14:59.469277 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.467822    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2b4e57a8-379d-4375-95b4-8176b1e4e410-serviceca\") pod \"node-ca-qdthk\" (UID: \"2b4e57a8-379d-4375-95b4-8176b1e4e410\") " pod="openshift-image-registry/node-ca-qdthk"
Sep 11 14:14:59.469277 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.467917    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d3509467-2ab1-4e02-9654-71fb22f361e3-system-cni-dir\") pod \"multus-2cdd6\" (UID: \"d3509467-2ab1-4e02-9654-71fb22f361e3\") " pod="openshift-multus/multus-2cdd6"
Sep 11 14:14:59.469277 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.467931    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b86ff2d3-8990-4b7f-b243-744fe2351991-etc-kubernetes\") pod \"tuned-b7jh5\" (UID: \"b86ff2d3-8990-4b7f-b243-744fe2351991\") " pod="openshift-cluster-node-tuning-operator/tuned-b7jh5"
Sep 11 14:14:59.469277 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.467949    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d3509467-2ab1-4e02-9654-71fb22f361e3-host-var-lib-cni-bin\") pod \"multus-2cdd6\" (UID: \"d3509467-2ab1-4e02-9654-71fb22f361e3\") " pod="openshift-multus/multus-2cdd6"
Sep 11 14:14:59.469277 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.468046    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b86ff2d3-8990-4b7f-b243-744fe2351991-etc-modprobe-d\") pod \"tuned-b7jh5\" (UID: \"b86ff2d3-8990-4b7f-b243-744fe2351991\") " pod="openshift-cluster-node-tuning-operator/tuned-b7jh5"
Sep 11 14:14:59.469277 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.468054    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9eadc143-6ef8-4518-8291-b323e6f15cb3-socket-dir\") pod \"aws-ebs-csi-driver-node-qx7nv\" (UID: \"9eadc143-6ef8-4518-8291-b323e6f15cb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx7nv"
Sep 11 14:14:59.469277 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.468085    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2b4e57a8-379d-4375-95b4-8176b1e4e410-host\") pod \"node-ca-qdthk\" (UID: \"2b4e57a8-379d-4375-95b4-8176b1e4e410\") " pod="openshift-image-registry/node-ca-qdthk"
Sep 11 14:14:59.469277 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.468107    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d3509467-2ab1-4e02-9654-71fb22f361e3-multus-conf-dir\") pod \"multus-2cdd6\" (UID: \"d3509467-2ab1-4e02-9654-71fb22f361e3\") " pod="openshift-multus/multus-2cdd6"
Sep 11 14:14:59.469277 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.468123    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d3509467-2ab1-4e02-9654-71fb22f361e3-host-var-lib-cni-multus\") pod \"multus-2cdd6\" (UID: \"d3509467-2ab1-4e02-9654-71fb22f361e3\") " pod="openshift-multus/multus-2cdd6"
Sep 11 14:14:59.469277 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.468167    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d3509467-2ab1-4e02-9654-71fb22f361e3-cnibin\") pod \"multus-2cdd6\" (UID: \"d3509467-2ab1-4e02-9654-71fb22f361e3\") " pod="openshift-multus/multus-2cdd6"
Sep 11 14:14:59.469277 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.468168    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b86ff2d3-8990-4b7f-b243-744fe2351991-var-lib-kubelet\") pod \"tuned-b7jh5\" (UID: \"b86ff2d3-8990-4b7f-b243-744fe2351991\") " pod="openshift-cluster-node-tuning-operator/tuned-b7jh5"
Sep 11 14:14:59.469277 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.468197    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3509467-2ab1-4e02-9654-71fb22f361e3-etc-kubernetes\") pod \"multus-2cdd6\" (UID: \"d3509467-2ab1-4e02-9654-71fb22f361e3\") " pod="openshift-multus/multus-2cdd6"
Sep 11 14:14:59.469277 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.468236    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b86ff2d3-8990-4b7f-b243-744fe2351991-lib-modules\") pod \"tuned-b7jh5\" (UID: \"b86ff2d3-8990-4b7f-b243-744fe2351991\") " pod="openshift-cluster-node-tuning-operator/tuned-b7jh5"
Sep 11 14:14:59.469277 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.468369    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d3509467-2ab1-4e02-9654-71fb22f361e3-multus-cni-dir\") pod \"multus-2cdd6\" (UID: \"d3509467-2ab1-4e02-9654-71fb22f361e3\") " pod="openshift-multus/multus-2cdd6"
Sep 11 14:14:59.469277 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.467828    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b86ff2d3-8990-4b7f-b243-744fe2351991-etc-kubernetes\") pod \"tuned-b7jh5\" (UID: \"b86ff2d3-8990-4b7f-b243-744fe2351991\") " pod="openshift-cluster-node-tuning-operator/tuned-b7jh5"
Sep 11 14:14:59.469277 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.468449    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9eadc143-6ef8-4518-8291-b323e6f15cb3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qx7nv\" (UID: \"9eadc143-6ef8-4518-8291-b323e6f15cb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx7nv"
Sep 11 14:14:59.469277 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.468715    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d3509467-2ab1-4e02-9654-71fb22f361e3-host-run-multus-certs\") pod \"multus-2cdd6\" (UID: \"d3509467-2ab1-4e02-9654-71fb22f361e3\") " pod="openshift-multus/multus-2cdd6"
Sep 11 14:14:59.470142 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.468780    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9eadc143-6ef8-4518-8291-b323e6f15cb3-sys-fs\") pod \"aws-ebs-csi-driver-node-qx7nv\" (UID: \"9eadc143-6ef8-4518-8291-b323e6f15cb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx7nv"
Sep 11 14:14:59.470142 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.468825    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d3509467-2ab1-4e02-9654-71fb22f361e3-host-run-netns\") pod \"multus-2cdd6\" (UID: \"d3509467-2ab1-4e02-9654-71fb22f361e3\") " pod="openshift-multus/multus-2cdd6"
Sep 11 14:14:59.470142 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.468866    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3509467-2ab1-4e02-9654-71fb22f361e3-host-var-lib-kubelet\") pod \"multus-2cdd6\" (UID: \"d3509467-2ab1-4e02-9654-71fb22f361e3\") " pod="openshift-multus/multus-2cdd6"
Sep 11 14:14:59.470142 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.468913    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d3509467-2ab1-4e02-9654-71fb22f361e3-host-run-k8s-cni-cncf-io\") pod \"multus-2cdd6\" (UID: \"d3509467-2ab1-4e02-9654-71fb22f361e3\") " pod="openshift-multus/multus-2cdd6"
Sep 11 14:14:59.470142 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.468938    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d3509467-2ab1-4e02-9654-71fb22f361e3-cni-binary-copy\") pod \"multus-2cdd6\" (UID: \"d3509467-2ab1-4e02-9654-71fb22f361e3\") " pod="openshift-multus/multus-2cdd6"
Sep 11 14:14:59.470142 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.468962    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b86ff2d3-8990-4b7f-b243-744fe2351991-run\") pod \"tuned-b7jh5\" (UID: \"b86ff2d3-8990-4b7f-b243-744fe2351991\") " pod="openshift-cluster-node-tuning-operator/tuned-b7jh5"
Sep 11 14:14:59.470142 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.469017    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d3509467-2ab1-4e02-9654-71fb22f361e3-hostroot\") pod \"multus-2cdd6\" (UID: \"d3509467-2ab1-4e02-9654-71fb22f361e3\") " pod="openshift-multus/multus-2cdd6"
Sep 11 14:14:59.470142 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.469080    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b86ff2d3-8990-4b7f-b243-744fe2351991-sys\") pod \"tuned-b7jh5\" (UID: \"b86ff2d3-8990-4b7f-b243-744fe2351991\") " pod="openshift-cluster-node-tuning-operator/tuned-b7jh5"
Sep 11 14:14:59.470142 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.469092    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b86ff2d3-8990-4b7f-b243-744fe2351991-host\") pod \"tuned-b7jh5\" (UID: \"b86ff2d3-8990-4b7f-b243-744fe2351991\") " pod="openshift-cluster-node-tuning-operator/tuned-b7jh5"
Sep 11 14:14:59.470142 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.469106    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b86ff2d3-8990-4b7f-b243-744fe2351991-etc-sysctl-conf\") pod \"tuned-b7jh5\" (UID: \"b86ff2d3-8990-4b7f-b243-744fe2351991\") " pod="openshift-cluster-node-tuning-operator/tuned-b7jh5"
Sep 11 14:14:59.470142 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.469122    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/07633f0c-6e0a-4bc2-811f-263730bfb6a4-host-slash\") pod \"iptables-alerter-v285k\" (UID: \"07633f0c-6e0a-4bc2-811f-263730bfb6a4\") " pod="openshift-network-operator/iptables-alerter-v285k"
Sep 11 14:14:59.470142 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.469150    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9eadc143-6ef8-4518-8291-b323e6f15cb3-registration-dir\") pod \"aws-ebs-csi-driver-node-qx7nv\" (UID: \"9eadc143-6ef8-4518-8291-b323e6f15cb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx7nv"
Sep 11 14:14:59.470142 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.469164    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9eadc143-6ef8-4518-8291-b323e6f15cb3-etc-selinux\") pod \"aws-ebs-csi-driver-node-qx7nv\" (UID: \"9eadc143-6ef8-4518-8291-b323e6f15cb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx7nv"
Sep 11 14:14:59.470142 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.469193    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9eadc143-6ef8-4518-8291-b323e6f15cb3-device-dir\") pod \"aws-ebs-csi-driver-node-qx7nv\" (UID: \"9eadc143-6ef8-4518-8291-b323e6f15cb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx7nv"
Sep 11 14:14:59.470142 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.469193    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c81645fc-5b10-41bb-9908-7512097108e8-konnectivity-ca\") pod \"konnectivity-agent-pnxhk\" (UID: \"c81645fc-5b10-41bb-9908-7512097108e8\") " pod="kube-system/konnectivity-agent-pnxhk"
Sep 11 14:14:59.470142 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.469217    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d3509467-2ab1-4e02-9654-71fb22f361e3-multus-socket-dir-parent\") pod \"multus-2cdd6\" (UID: \"d3509467-2ab1-4e02-9654-71fb22f361e3\") " pod="openshift-multus/multus-2cdd6"
Sep 11 14:14:59.470142 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.469246    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9eadc143-6ef8-4518-8291-b323e6f15cb3-device-dir\") pod \"aws-ebs-csi-driver-node-qx7nv\" (UID: \"9eadc143-6ef8-4518-8291-b323e6f15cb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx7nv"
Sep 11 14:14:59.470142 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.469253    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c81645fc-5b10-41bb-9908-7512097108e8-agent-certs\") pod \"konnectivity-agent-pnxhk\" (UID: \"c81645fc-5b10-41bb-9908-7512097108e8\") " pod="kube-system/konnectivity-agent-pnxhk"
Sep 11 14:14:59.470995 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.469257    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/07633f0c-6e0a-4bc2-811f-263730bfb6a4-host-slash\") pod \"iptables-alerter-v285k\" (UID: \"07633f0c-6e0a-4bc2-811f-263730bfb6a4\") " pod="openshift-network-operator/iptables-alerter-v285k"
Sep 11 14:14:59.470995 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.469294    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b86ff2d3-8990-4b7f-b243-744fe2351991-host\") pod \"tuned-b7jh5\" (UID: \"b86ff2d3-8990-4b7f-b243-744fe2351991\") " pod="openshift-cluster-node-tuning-operator/tuned-b7jh5"
Sep 11 14:14:59.470995 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.469294    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9eadc143-6ef8-4518-8291-b323e6f15cb3-registration-dir\") pod \"aws-ebs-csi-driver-node-qx7nv\" (UID: \"9eadc143-6ef8-4518-8291-b323e6f15cb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx7nv"
Sep 11 14:14:59.470995 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.469354    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b86ff2d3-8990-4b7f-b243-744fe2351991-etc-systemd\") pod \"tuned-b7jh5\" (UID: \"b86ff2d3-8990-4b7f-b243-744fe2351991\") " pod="openshift-cluster-node-tuning-operator/tuned-b7jh5"
Sep 11 14:14:59.470995 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.469604    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/07633f0c-6e0a-4bc2-811f-263730bfb6a4-iptables-alerter-script\") pod \"iptables-alerter-v285k\" (UID: \"07633f0c-6e0a-4bc2-811f-263730bfb6a4\") " pod="openshift-network-operator/iptables-alerter-v285k"
Sep 11 14:14:59.470995 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.469648    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d3509467-2ab1-4e02-9654-71fb22f361e3-multus-daemon-config\") pod \"multus-2cdd6\" (UID: \"d3509467-2ab1-4e02-9654-71fb22f361e3\") " pod="openshift-multus/multus-2cdd6"
Sep 11 14:14:59.471274 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.471019    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b86ff2d3-8990-4b7f-b243-744fe2351991-tmp\") pod \"tuned-b7jh5\" (UID: \"b86ff2d3-8990-4b7f-b243-744fe2351991\") " pod="openshift-cluster-node-tuning-operator/tuned-b7jh5"
Sep 11 14:14:59.471470 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.471440    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b86ff2d3-8990-4b7f-b243-744fe2351991-etc-tuned\") pod \"tuned-b7jh5\" (UID: \"b86ff2d3-8990-4b7f-b243-744fe2351991\") " pod="openshift-cluster-node-tuning-operator/tuned-b7jh5"
Sep 11 14:14:59.471470 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.471461    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c81645fc-5b10-41bb-9908-7512097108e8-agent-certs\") pod \"konnectivity-agent-pnxhk\" (UID: \"c81645fc-5b10-41bb-9908-7512097108e8\") " pod="kube-system/konnectivity-agent-pnxhk"
Sep 11 14:14:59.487505 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.487420    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z72bb\" (UniqueName: \"kubernetes.io/projected/2b4e57a8-379d-4375-95b4-8176b1e4e410-kube-api-access-z72bb\") pod \"node-ca-qdthk\" (UID: \"2b4e57a8-379d-4375-95b4-8176b1e4e410\") " pod="openshift-image-registry/node-ca-qdthk"
Sep 11 14:14:59.487902 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.487876    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ksr2\" (UniqueName: \"kubernetes.io/projected/07633f0c-6e0a-4bc2-811f-263730bfb6a4-kube-api-access-5ksr2\") pod \"iptables-alerter-v285k\" (UID: \"07633f0c-6e0a-4bc2-811f-263730bfb6a4\") " pod="openshift-network-operator/iptables-alerter-v285k"
Sep 11 14:14:59.492201 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.492174    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcxp8\" (UniqueName: \"kubernetes.io/projected/9eadc143-6ef8-4518-8291-b323e6f15cb3-kube-api-access-qcxp8\") pod \"aws-ebs-csi-driver-node-qx7nv\" (UID: \"9eadc143-6ef8-4518-8291-b323e6f15cb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx7nv"
Sep 11 14:14:59.494662 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.494617    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtz8s\" (UniqueName: \"kubernetes.io/projected/d3509467-2ab1-4e02-9654-71fb22f361e3-kube-api-access-dtz8s\") pod \"multus-2cdd6\" (UID: \"d3509467-2ab1-4e02-9654-71fb22f361e3\") " pod="openshift-multus/multus-2cdd6"
Sep 11 14:14:59.497793 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.497773    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks678\" (UniqueName: \"kubernetes.io/projected/b86ff2d3-8990-4b7f-b243-744fe2351991-kube-api-access-ks678\") pod \"tuned-b7jh5\" (UID: \"b86ff2d3-8990-4b7f-b243-744fe2351991\") " pod="openshift-cluster-node-tuning-operator/tuned-b7jh5"
Sep 11 14:14:59.498251 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.498211    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-150.ec2.internal" event={"ID":"9bd85cdb5fbcf445a1156f9ccc86edcd","Type":"ContainerStarted","Data":"0049bbc7ff24d3207fa5f8806b8ebeb1bb8d0d5e608822c3949f1d42c1c08d83"}
Sep 11 14:14:59.499460 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.499439    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-150.ec2.internal" event={"ID":"87337b80b4395cecaecd8c7eb49f5684","Type":"ContainerStarted","Data":"485187baec14e961f583570f9aa64df40cb4d50dce52692c190401480551ee97"}
Sep 11 14:14:59.554305 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.554276    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mn2mg"
Sep 11 14:14:59.562549 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.562529    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wk6kn"
Sep 11 14:14:59.573421 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.573401    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:14:59.580165 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.580144    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-b7jh5"
Sep 11 14:14:59.587939 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.587920    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pnxhk"
Sep 11 14:14:59.595172 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.595149    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-v285k"
Sep 11 14:14:59.602997 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.602961    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qdthk"
Sep 11 14:14:59.610656 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.610632    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx7nv"
Sep 11 14:14:59.617874 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.617855    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2cdd6"
Sep 11 14:14:59.872359 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.872272    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3376794-b10e-4353-b8f3-4639741019eb-metrics-certs\") pod \"network-metrics-daemon-wgmt4\" (UID: \"d3376794-b10e-4353-b8f3-4639741019eb\") " pod="openshift-multus/network-metrics-daemon-wgmt4"
Sep 11 14:14:59.872359 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.872315    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/78d2df0e-635a-4123-9a80-f9898562e613-original-pull-secret\") pod \"global-pull-secret-syncer-dqmtt\" (UID: \"78d2df0e-635a-4123-9a80-f9898562e613\") " pod="kube-system/global-pull-secret-syncer-dqmtt"
Sep 11 14:14:59.872565 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:14:59.872414    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 14:14:59.872565 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:14:59.872498    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3376794-b10e-4353-b8f3-4639741019eb-metrics-certs podName:d3376794-b10e-4353-b8f3-4639741019eb nodeName:}" failed. No retries permitted until 2026-09-11 14:15:00.872473615 +0000 UTC m=+4.023023660 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d3376794-b10e-4353-b8f3-4639741019eb-metrics-certs") pod "network-metrics-daemon-wgmt4" (UID: "d3376794-b10e-4353-b8f3-4639741019eb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 14:14:59.872565 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:14:59.872414    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 14:14:59.872700 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:14:59.872571    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78d2df0e-635a-4123-9a80-f9898562e613-original-pull-secret podName:78d2df0e-635a-4123-9a80-f9898562e613 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:00.872555666 +0000 UTC m=+4.023105728 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/78d2df0e-635a-4123-9a80-f9898562e613-original-pull-secret") pod "global-pull-secret-syncer-dqmtt" (UID: "78d2df0e-635a-4123-9a80-f9898562e613") : object "kube-system"/"original-pull-secret" not registered
Sep 11 14:14:59.973470 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:14:59.973427    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ffkb\" (UniqueName: \"kubernetes.io/projected/f58cdeac-ef49-4a6e-b459-ad21493d1497-kube-api-access-5ffkb\") pod \"network-check-target-q5sxd\" (UID: \"f58cdeac-ef49-4a6e-b459-ad21493d1497\") " pod="openshift-network-diagnostics/network-check-target-q5sxd"
Sep 11 14:14:59.973628 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:14:59.973601    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 14:14:59.973628 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:14:59.973627    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 14:14:59.973724 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:14:59.973638    2581 projected.go:196] Error preparing data for projected volume kube-api-access-5ffkb for pod openshift-network-diagnostics/network-check-target-q5sxd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 14:14:59.973724 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:14:59.973706    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f58cdeac-ef49-4a6e-b459-ad21493d1497-kube-api-access-5ffkb podName:f58cdeac-ef49-4a6e-b459-ad21493d1497 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:00.973691798 +0000 UTC m=+4.124241840 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5ffkb" (UniqueName: "kubernetes.io/projected/f58cdeac-ef49-4a6e-b459-ad21493d1497-kube-api-access-5ffkb") pod "network-check-target-q5sxd" (UID: "f58cdeac-ef49-4a6e-b459-ad21493d1497") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 14:15:00.143643 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:00.143561    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 14:15:00.313217 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:00.313121    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 14:09:59 +0000 UTC" deadline="2028-04-19 07:58:44.728615515 +0000 UTC"
Sep 11 14:15:00.313217 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:00.313200    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14057h43m44.415420169s"
Sep 11 14:15:00.481630 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:15:00.481595    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3509467_2ab1_4e02_9654_71fb22f361e3.slice/crio-f906c23499fa2b0d3d7ef892457432dd3e78400f8b9b5fc96dfc8ef06621ffbb WatchSource:0}: Error finding container f906c23499fa2b0d3d7ef892457432dd3e78400f8b9b5fc96dfc8ef06621ffbb: Status 404 returned error can't find the container with id f906c23499fa2b0d3d7ef892457432dd3e78400f8b9b5fc96dfc8ef06621ffbb
Sep 11 14:15:00.483880 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:15:00.483749    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod757db884_d01a_4ab4_8b46_f9a41a3d7369.slice/crio-c147e7fce0ff0850e7d3e212f443242edb31281d639916bd80cb82360ea444a2 WatchSource:0}: Error finding container c147e7fce0ff0850e7d3e212f443242edb31281d639916bd80cb82360ea444a2: Status 404 returned error can't find the container with id c147e7fce0ff0850e7d3e212f443242edb31281d639916bd80cb82360ea444a2
Sep 11 14:15:00.488041 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:15:00.488019    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07633f0c_6e0a_4bc2_811f_263730bfb6a4.slice/crio-ffea26a120c60fecf751cfcfdbf9ddc26a6682144252bf139edf8d879c1a7487 WatchSource:0}: Error finding container ffea26a120c60fecf751cfcfdbf9ddc26a6682144252bf139edf8d879c1a7487: Status 404 returned error can't find the container with id ffea26a120c60fecf751cfcfdbf9ddc26a6682144252bf139edf8d879c1a7487
Sep 11 14:15:00.489158 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:15:00.489130    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eadc143_6ef8_4518_8291_b323e6f15cb3.slice/crio-7b75ce3ccecdb5d3d9b8ba8974a66adc2bbe4e56aafff92843557d77626c5235 WatchSource:0}: Error finding container 7b75ce3ccecdb5d3d9b8ba8974a66adc2bbe4e56aafff92843557d77626c5235: Status 404 returned error can't find the container with id 7b75ce3ccecdb5d3d9b8ba8974a66adc2bbe4e56aafff92843557d77626c5235
Sep 11 14:15:00.491026 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:15:00.490875    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b4e57a8_379d_4375_95b4_8176b1e4e410.slice/crio-c83903813ae09961c52462e6dbf01eda6ba72b412d0265a93cf82bdbe2b084cb WatchSource:0}: Error finding container c83903813ae09961c52462e6dbf01eda6ba72b412d0265a93cf82bdbe2b084cb: Status 404 returned error can't find the container with id c83903813ae09961c52462e6dbf01eda6ba72b412d0265a93cf82bdbe2b084cb
Sep 11 14:15:00.491849 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:15:00.491723    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod774c0511_8141_47cd_8718_b647be205839.slice/crio-8cf7f45363788582e85c65f16264efefb23655d4e17a3f4010a6f4176a5ca05a WatchSource:0}: Error finding container 8cf7f45363788582e85c65f16264efefb23655d4e17a3f4010a6f4176a5ca05a: Status 404 returned error can't find the container with id 8cf7f45363788582e85c65f16264efefb23655d4e17a3f4010a6f4176a5ca05a
Sep 11 14:15:00.492530 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:15:00.492404    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc81645fc_5b10_41bb_9908_7512097108e8.slice/crio-31839ca6b5678947cd485a2b640fa83c240d138671e3bf994d4d73c7d1026140 WatchSource:0}: Error finding container 31839ca6b5678947cd485a2b640fa83c240d138671e3bf994d4d73c7d1026140: Status 404 returned error can't find the container with id 31839ca6b5678947cd485a2b640fa83c240d138671e3bf994d4d73c7d1026140
Sep 11 14:15:00.493712 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:00.493283    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqmtt"
Sep 11 14:15:00.493712 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:00.493380    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqmtt" podUID="78d2df0e-635a-4123-9a80-f9898562e613"
Sep 11 14:15:00.502131 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:00.502105    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mn2mg" event={"ID":"774c0511-8141-47cd-8718-b647be205839","Type":"ContainerStarted","Data":"8cf7f45363788582e85c65f16264efefb23655d4e17a3f4010a6f4176a5ca05a"}
Sep 11 14:15:00.503498 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:00.503471    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qdthk" event={"ID":"2b4e57a8-379d-4375-95b4-8176b1e4e410","Type":"ContainerStarted","Data":"c83903813ae09961c52462e6dbf01eda6ba72b412d0265a93cf82bdbe2b084cb"}
Sep 11 14:15:00.505383 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:00.505359    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx7nv" event={"ID":"9eadc143-6ef8-4518-8291-b323e6f15cb3","Type":"ContainerStarted","Data":"7b75ce3ccecdb5d3d9b8ba8974a66adc2bbe4e56aafff92843557d77626c5235"}
Sep 11 14:15:00.506681 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:00.506662    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n88qj" event={"ID":"fedc3186-f028-4964-a5e9-d6315fb07164","Type":"ContainerStarted","Data":"b8216e3a7c20c28fa7d659a21bbe3a43c4bce00d8c7b32255bac527f2f271035"}
Sep 11 14:15:00.508328 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:00.508295    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-v285k" event={"ID":"07633f0c-6e0a-4bc2-811f-263730bfb6a4","Type":"ContainerStarted","Data":"ffea26a120c60fecf751cfcfdbf9ddc26a6682144252bf139edf8d879c1a7487"}
Sep 11 14:15:00.509415 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:00.509391    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wk6kn" event={"ID":"757db884-d01a-4ab4-8b46-f9a41a3d7369","Type":"ContainerStarted","Data":"c147e7fce0ff0850e7d3e212f443242edb31281d639916bd80cb82360ea444a2"}
Sep 11 14:15:00.510696 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:00.510665    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2cdd6" event={"ID":"d3509467-2ab1-4e02-9654-71fb22f361e3","Type":"ContainerStarted","Data":"f906c23499fa2b0d3d7ef892457432dd3e78400f8b9b5fc96dfc8ef06621ffbb"}
Sep 11 14:15:00.511948 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:00.511919    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-b7jh5" event={"ID":"b86ff2d3-8990-4b7f-b243-744fe2351991","Type":"ContainerStarted","Data":"98cb5736b91462799931b99ebc1290b3dc28635e23b921261b1460392bbd8672"}
Sep 11 14:15:00.513112 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:00.513091    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pnxhk" event={"ID":"c81645fc-5b10-41bb-9908-7512097108e8","Type":"ContainerStarted","Data":"31839ca6b5678947cd485a2b640fa83c240d138671e3bf994d4d73c7d1026140"}
Sep 11 14:15:00.880374 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:00.880340    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3376794-b10e-4353-b8f3-4639741019eb-metrics-certs\") pod \"network-metrics-daemon-wgmt4\" (UID: \"d3376794-b10e-4353-b8f3-4639741019eb\") " pod="openshift-multus/network-metrics-daemon-wgmt4"
Sep 11 14:15:00.880537 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:00.880384    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/78d2df0e-635a-4123-9a80-f9898562e613-original-pull-secret\") pod \"global-pull-secret-syncer-dqmtt\" (UID: \"78d2df0e-635a-4123-9a80-f9898562e613\") " pod="kube-system/global-pull-secret-syncer-dqmtt"
Sep 11 14:15:00.880537 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:00.880508    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 14:15:00.880537 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:00.880508    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 14:15:00.880679 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:00.880579    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3376794-b10e-4353-b8f3-4639741019eb-metrics-certs podName:d3376794-b10e-4353-b8f3-4639741019eb nodeName:}" failed. No retries permitted until 2026-09-11 14:15:02.88056109 +0000 UTC m=+6.031111144 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d3376794-b10e-4353-b8f3-4639741019eb-metrics-certs") pod "network-metrics-daemon-wgmt4" (UID: "d3376794-b10e-4353-b8f3-4639741019eb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 14:15:00.880679 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:00.880599    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78d2df0e-635a-4123-9a80-f9898562e613-original-pull-secret podName:78d2df0e-635a-4123-9a80-f9898562e613 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:02.880589257 +0000 UTC m=+6.031139300 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/78d2df0e-635a-4123-9a80-f9898562e613-original-pull-secret") pod "global-pull-secret-syncer-dqmtt" (UID: "78d2df0e-635a-4123-9a80-f9898562e613") : object "kube-system"/"original-pull-secret" not registered
Sep 11 14:15:00.981769 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:00.981692    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ffkb\" (UniqueName: \"kubernetes.io/projected/f58cdeac-ef49-4a6e-b459-ad21493d1497-kube-api-access-5ffkb\") pod \"network-check-target-q5sxd\" (UID: \"f58cdeac-ef49-4a6e-b459-ad21493d1497\") " pod="openshift-network-diagnostics/network-check-target-q5sxd"
Sep 11 14:15:00.981928 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:00.981862    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 14:15:00.981928 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:00.981883    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 14:15:00.981928 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:00.981905    2581 projected.go:196] Error preparing data for projected volume kube-api-access-5ffkb for pod openshift-network-diagnostics/network-check-target-q5sxd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 14:15:00.982108 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:00.981987    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f58cdeac-ef49-4a6e-b459-ad21493d1497-kube-api-access-5ffkb podName:f58cdeac-ef49-4a6e-b459-ad21493d1497 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:02.981944622 +0000 UTC m=+6.132494670 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5ffkb" (UniqueName: "kubernetes.io/projected/f58cdeac-ef49-4a6e-b459-ad21493d1497-kube-api-access-5ffkb") pod "network-check-target-q5sxd" (UID: "f58cdeac-ef49-4a6e-b459-ad21493d1497") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 14:15:01.314023 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:01.313914    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 14:09:59 +0000 UTC" deadline="2028-05-02 13:24:05.431982968 +0000 UTC"
Sep 11 14:15:01.314023 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:01.313958    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14375h9m4.118029298s"
Sep 11 14:15:01.493260 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:01.493230    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5sxd"
Sep 11 14:15:01.493425 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:01.493364    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5sxd" podUID="f58cdeac-ef49-4a6e-b459-ad21493d1497"
Sep 11 14:15:01.493793 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:01.493772    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wgmt4"
Sep 11 14:15:01.493918 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:01.493896    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wgmt4" podUID="d3376794-b10e-4353-b8f3-4639741019eb"
Sep 11 14:15:01.528020 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:01.527382    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-150.ec2.internal" event={"ID":"9bd85cdb5fbcf445a1156f9ccc86edcd","Type":"ContainerStarted","Data":"9f45af1066229e29b008bb04973929f9c0f6185eea7cc0d4e925377ac182200f"}
Sep 11 14:15:01.533768 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:01.533160    2581 generic.go:356] "Generic (PLEG): container finished" podID="87337b80b4395cecaecd8c7eb49f5684" containerID="4d46cad2289beabfde00523c3f862af3e10e9334a2dc0c1f9ade785b729cab15" exitCode=0
Sep 11 14:15:01.533768 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:01.533210    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-150.ec2.internal" event={"ID":"87337b80b4395cecaecd8c7eb49f5684","Type":"ContainerDied","Data":"4d46cad2289beabfde00523c3f862af3e10e9334a2dc0c1f9ade785b729cab15"}
Sep 11 14:15:01.574258 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:01.574200    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-150.ec2.internal" podStartSLOduration=3.5741762169999998 podStartE2EDuration="3.574176217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:14:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:01.550849775 +0000 UTC m=+4.701399839" watchObservedRunningTime="2026-09-11 14:15:01.574176217 +0000 UTC m=+4.724726282"
Sep 11 14:15:02.493853 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:02.493528    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqmtt"
Sep 11 14:15:02.494316 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:02.493900    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqmtt" podUID="78d2df0e-635a-4123-9a80-f9898562e613"
Sep 11 14:15:02.553190 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:02.553034    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-150.ec2.internal" event={"ID":"87337b80b4395cecaecd8c7eb49f5684","Type":"ContainerStarted","Data":"8503c27e38e76ad4671c6fea3acfdff62a602ac7b09310d886210cf56e8b74e9"}
Sep 11 14:15:02.572791 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:02.571714    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-150.ec2.internal" podStartSLOduration=4.571690584 podStartE2EDuration="4.571690584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:14:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:02.571437117 +0000 UTC m=+5.721987182" watchObservedRunningTime="2026-09-11 14:15:02.571690584 +0000 UTC m=+5.722240649"
Sep 11 14:15:02.899238 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:02.899143    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3376794-b10e-4353-b8f3-4639741019eb-metrics-certs\") pod \"network-metrics-daemon-wgmt4\" (UID: \"d3376794-b10e-4353-b8f3-4639741019eb\") " pod="openshift-multus/network-metrics-daemon-wgmt4"
Sep 11 14:15:02.899400 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:02.899199    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/78d2df0e-635a-4123-9a80-f9898562e613-original-pull-secret\") pod \"global-pull-secret-syncer-dqmtt\" (UID: \"78d2df0e-635a-4123-9a80-f9898562e613\") " pod="kube-system/global-pull-secret-syncer-dqmtt"
Sep 11 14:15:02.899454 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:02.899412    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 14:15:02.899921 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:02.899513    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78d2df0e-635a-4123-9a80-f9898562e613-original-pull-secret podName:78d2df0e-635a-4123-9a80-f9898562e613 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:06.899494075 +0000 UTC m=+10.050044133 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/78d2df0e-635a-4123-9a80-f9898562e613-original-pull-secret") pod "global-pull-secret-syncer-dqmtt" (UID: "78d2df0e-635a-4123-9a80-f9898562e613") : object "kube-system"/"original-pull-secret" not registered
Sep 11 14:15:02.900080 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:02.899942    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 14:15:02.900080 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:02.900009    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3376794-b10e-4353-b8f3-4639741019eb-metrics-certs podName:d3376794-b10e-4353-b8f3-4639741019eb nodeName:}" failed. No retries permitted until 2026-09-11 14:15:06.899992744 +0000 UTC m=+10.050542786 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d3376794-b10e-4353-b8f3-4639741019eb-metrics-certs") pod "network-metrics-daemon-wgmt4" (UID: "d3376794-b10e-4353-b8f3-4639741019eb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 14:15:02.999783 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:02.999734    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ffkb\" (UniqueName: \"kubernetes.io/projected/f58cdeac-ef49-4a6e-b459-ad21493d1497-kube-api-access-5ffkb\") pod \"network-check-target-q5sxd\" (UID: \"f58cdeac-ef49-4a6e-b459-ad21493d1497\") " pod="openshift-network-diagnostics/network-check-target-q5sxd"
Sep 11 14:15:02.999992 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:02.999944    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 14:15:02.999992 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:02.999983    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 14:15:03.000133 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:02.999997    2581 projected.go:196] Error preparing data for projected volume kube-api-access-5ffkb for pod openshift-network-diagnostics/network-check-target-q5sxd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 14:15:03.000133 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:03.000059    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f58cdeac-ef49-4a6e-b459-ad21493d1497-kube-api-access-5ffkb podName:f58cdeac-ef49-4a6e-b459-ad21493d1497 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:07.000041925 +0000 UTC m=+10.150592014 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5ffkb" (UniqueName: "kubernetes.io/projected/f58cdeac-ef49-4a6e-b459-ad21493d1497-kube-api-access-5ffkb") pod "network-check-target-q5sxd" (UID: "f58cdeac-ef49-4a6e-b459-ad21493d1497") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 14:15:03.497734 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:03.497692    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5sxd"
Sep 11 14:15:03.498198 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:03.497823    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5sxd" podUID="f58cdeac-ef49-4a6e-b459-ad21493d1497"
Sep 11 14:15:03.498198 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:03.498187    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wgmt4"
Sep 11 14:15:03.498341 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:03.498296    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wgmt4" podUID="d3376794-b10e-4353-b8f3-4639741019eb"
Sep 11 14:15:04.493824 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:04.493794    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqmtt"
Sep 11 14:15:04.494039 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:04.493922    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqmtt" podUID="78d2df0e-635a-4123-9a80-f9898562e613"
Sep 11 14:15:05.493745 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:05.493667    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wgmt4"
Sep 11 14:15:05.493745 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:05.493708    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5sxd"
Sep 11 14:15:05.494280 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:05.493809    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wgmt4" podUID="d3376794-b10e-4353-b8f3-4639741019eb"
Sep 11 14:15:05.494280 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:05.493948    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5sxd" podUID="f58cdeac-ef49-4a6e-b459-ad21493d1497"
Sep 11 14:15:06.493274 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:06.493079    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqmtt"
Sep 11 14:15:06.493274 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:06.493236    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqmtt" podUID="78d2df0e-635a-4123-9a80-f9898562e613"
Sep 11 14:15:06.935926 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:06.935705    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3376794-b10e-4353-b8f3-4639741019eb-metrics-certs\") pod \"network-metrics-daemon-wgmt4\" (UID: \"d3376794-b10e-4353-b8f3-4639741019eb\") " pod="openshift-multus/network-metrics-daemon-wgmt4"
Sep 11 14:15:06.935926 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:06.935774    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/78d2df0e-635a-4123-9a80-f9898562e613-original-pull-secret\") pod \"global-pull-secret-syncer-dqmtt\" (UID: \"78d2df0e-635a-4123-9a80-f9898562e613\") " pod="kube-system/global-pull-secret-syncer-dqmtt"
Sep 11 14:15:06.935926 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:06.935829    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 14:15:06.935926 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:06.935894    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3376794-b10e-4353-b8f3-4639741019eb-metrics-certs podName:d3376794-b10e-4353-b8f3-4639741019eb nodeName:}" failed. No retries permitted until 2026-09-11 14:15:14.935878206 +0000 UTC m=+18.086428248 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d3376794-b10e-4353-b8f3-4639741019eb-metrics-certs") pod "network-metrics-daemon-wgmt4" (UID: "d3376794-b10e-4353-b8f3-4639741019eb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 14:15:06.935926 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:06.935904    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 14:15:06.936644 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:06.935962    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78d2df0e-635a-4123-9a80-f9898562e613-original-pull-secret podName:78d2df0e-635a-4123-9a80-f9898562e613 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:14.935944726 +0000 UTC m=+18.086494788 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/78d2df0e-635a-4123-9a80-f9898562e613-original-pull-secret") pod "global-pull-secret-syncer-dqmtt" (UID: "78d2df0e-635a-4123-9a80-f9898562e613") : object "kube-system"/"original-pull-secret" not registered
Sep 11 14:15:07.036849 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:07.036817    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ffkb\" (UniqueName: \"kubernetes.io/projected/f58cdeac-ef49-4a6e-b459-ad21493d1497-kube-api-access-5ffkb\") pod \"network-check-target-q5sxd\" (UID: \"f58cdeac-ef49-4a6e-b459-ad21493d1497\") " pod="openshift-network-diagnostics/network-check-target-q5sxd"
Sep 11 14:15:07.037068 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:07.036941    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 14:15:07.037068 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:07.036954    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 14:15:07.037068 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:07.036964    2581 projected.go:196] Error preparing data for projected volume kube-api-access-5ffkb for pod openshift-network-diagnostics/network-check-target-q5sxd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 14:15:07.037068 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:07.037044    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f58cdeac-ef49-4a6e-b459-ad21493d1497-kube-api-access-5ffkb podName:f58cdeac-ef49-4a6e-b459-ad21493d1497 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:15.037025384 +0000 UTC m=+18.187575429 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5ffkb" (UniqueName: "kubernetes.io/projected/f58cdeac-ef49-4a6e-b459-ad21493d1497-kube-api-access-5ffkb") pod "network-check-target-q5sxd" (UID: "f58cdeac-ef49-4a6e-b459-ad21493d1497") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 14:15:07.495369 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:07.494633    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wgmt4"
Sep 11 14:15:07.495369 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:07.494772    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wgmt4" podUID="d3376794-b10e-4353-b8f3-4639741019eb"
Sep 11 14:15:07.495369 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:07.495221    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5sxd"
Sep 11 14:15:07.495369 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:07.495317    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5sxd" podUID="f58cdeac-ef49-4a6e-b459-ad21493d1497"
Sep 11 14:15:08.493946 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:08.493854    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqmtt"
Sep 11 14:15:08.494410 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:08.494027    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqmtt" podUID="78d2df0e-635a-4123-9a80-f9898562e613"
Sep 11 14:15:09.493624 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:09.493587    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wgmt4"
Sep 11 14:15:09.493796 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:09.493722    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wgmt4" podUID="d3376794-b10e-4353-b8f3-4639741019eb"
Sep 11 14:15:09.493796 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:09.493780    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5sxd"
Sep 11 14:15:09.493915 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:09.493893    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5sxd" podUID="f58cdeac-ef49-4a6e-b459-ad21493d1497"
Sep 11 14:15:10.493240 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:10.493202    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqmtt"
Sep 11 14:15:10.493713 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:10.493323    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqmtt" podUID="78d2df0e-635a-4123-9a80-f9898562e613"
Sep 11 14:15:11.493480 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:11.493391    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5sxd"
Sep 11 14:15:11.493895 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:11.493391    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wgmt4"
Sep 11 14:15:11.493895 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:11.493524    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5sxd" podUID="f58cdeac-ef49-4a6e-b459-ad21493d1497"
Sep 11 14:15:11.493895 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:11.493607    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wgmt4" podUID="d3376794-b10e-4353-b8f3-4639741019eb"
Sep 11 14:15:12.493713 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:12.493680    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqmtt"
Sep 11 14:15:12.494184 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:12.493791    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqmtt" podUID="78d2df0e-635a-4123-9a80-f9898562e613"
Sep 11 14:15:13.493045 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:13.493008    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wgmt4"
Sep 11 14:15:13.493239 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:13.493052    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5sxd"
Sep 11 14:15:13.493239 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:13.493141    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wgmt4" podUID="d3376794-b10e-4353-b8f3-4639741019eb"
Sep 11 14:15:13.493359 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:13.493287    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5sxd" podUID="f58cdeac-ef49-4a6e-b459-ad21493d1497"
Sep 11 14:15:14.493129 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:14.493085    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqmtt"
Sep 11 14:15:14.493529 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:14.493202    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqmtt" podUID="78d2df0e-635a-4123-9a80-f9898562e613"
Sep 11 14:15:14.994413 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:14.994377    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3376794-b10e-4353-b8f3-4639741019eb-metrics-certs\") pod \"network-metrics-daemon-wgmt4\" (UID: \"d3376794-b10e-4353-b8f3-4639741019eb\") " pod="openshift-multus/network-metrics-daemon-wgmt4"
Sep 11 14:15:14.994413 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:14.994425    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/78d2df0e-635a-4123-9a80-f9898562e613-original-pull-secret\") pod \"global-pull-secret-syncer-dqmtt\" (UID: \"78d2df0e-635a-4123-9a80-f9898562e613\") " pod="kube-system/global-pull-secret-syncer-dqmtt"
Sep 11 14:15:14.994665 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:14.994539    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 14:15:14.994665 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:14.994542    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 14:15:14.994665 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:14.994600    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78d2df0e-635a-4123-9a80-f9898562e613-original-pull-secret podName:78d2df0e-635a-4123-9a80-f9898562e613 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:30.994580812 +0000 UTC m=+34.145130875 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/78d2df0e-635a-4123-9a80-f9898562e613-original-pull-secret") pod "global-pull-secret-syncer-dqmtt" (UID: "78d2df0e-635a-4123-9a80-f9898562e613") : object "kube-system"/"original-pull-secret" not registered
Sep 11 14:15:14.994665 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:14.994623    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3376794-b10e-4353-b8f3-4639741019eb-metrics-certs podName:d3376794-b10e-4353-b8f3-4639741019eb nodeName:}" failed. No retries permitted until 2026-09-11 14:15:30.994612387 +0000 UTC m=+34.145162433 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d3376794-b10e-4353-b8f3-4639741019eb-metrics-certs") pod "network-metrics-daemon-wgmt4" (UID: "d3376794-b10e-4353-b8f3-4639741019eb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 14:15:15.095360 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:15.095327    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ffkb\" (UniqueName: \"kubernetes.io/projected/f58cdeac-ef49-4a6e-b459-ad21493d1497-kube-api-access-5ffkb\") pod \"network-check-target-q5sxd\" (UID: \"f58cdeac-ef49-4a6e-b459-ad21493d1497\") " pod="openshift-network-diagnostics/network-check-target-q5sxd"
Sep 11 14:15:15.095560 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:15.095479    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 14:15:15.095560 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:15.095499    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 14:15:15.095560 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:15.095511    2581 projected.go:196] Error preparing data for projected volume kube-api-access-5ffkb for pod openshift-network-diagnostics/network-check-target-q5sxd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 14:15:15.095714 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:15.095573    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f58cdeac-ef49-4a6e-b459-ad21493d1497-kube-api-access-5ffkb podName:f58cdeac-ef49-4a6e-b459-ad21493d1497 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:31.095556804 +0000 UTC m=+34.246106859 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5ffkb" (UniqueName: "kubernetes.io/projected/f58cdeac-ef49-4a6e-b459-ad21493d1497-kube-api-access-5ffkb") pod "network-check-target-q5sxd" (UID: "f58cdeac-ef49-4a6e-b459-ad21493d1497") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 14:15:15.493114 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:15.493035    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wgmt4"
Sep 11 14:15:15.493275 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:15.493164    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wgmt4" podUID="d3376794-b10e-4353-b8f3-4639741019eb"
Sep 11 14:15:15.493275 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:15.493226    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5sxd"
Sep 11 14:15:15.493672 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:15.493324    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5sxd" podUID="f58cdeac-ef49-4a6e-b459-ad21493d1497"
Sep 11 14:15:16.493507 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:16.493468    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqmtt"
Sep 11 14:15:16.493989 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:16.493621    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqmtt" podUID="78d2df0e-635a-4123-9a80-f9898562e613"
Sep 11 14:15:17.493910 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:17.493880    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wgmt4"
Sep 11 14:15:17.494383 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:17.493996    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wgmt4" podUID="d3376794-b10e-4353-b8f3-4639741019eb"
Sep 11 14:15:17.494383 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:17.494072    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5sxd"
Sep 11 14:15:17.494383 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:17.494158    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5sxd" podUID="f58cdeac-ef49-4a6e-b459-ad21493d1497"
Sep 11 14:15:18.493797 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:18.493771    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqmtt"
Sep 11 14:15:18.493935 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:18.493911    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqmtt" podUID="78d2df0e-635a-4123-9a80-f9898562e613"
Sep 11 14:15:18.589016 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:18.588944    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2cdd6" event={"ID":"d3509467-2ab1-4e02-9654-71fb22f361e3","Type":"ContainerStarted","Data":"f9611dac408f273ca03fe09462d899a695d601cf7c0e7e8615aa16f7a0c5baf6"}
Sep 11 14:15:18.590515 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:18.590482    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-b7jh5" event={"ID":"b86ff2d3-8990-4b7f-b243-744fe2351991","Type":"ContainerStarted","Data":"d2b9a4af592eb5bfd116a5152d3c3e3f13b52284995852416eaed88e9cfe0c95"}
Sep 11 14:15:18.593996 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:18.593149    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pnxhk" event={"ID":"c81645fc-5b10-41bb-9908-7512097108e8","Type":"ContainerStarted","Data":"0c07558fc200438b45be5342c3b72d4d031c511434f5b5f9a654f4d90c8fa728"}
Sep 11 14:15:18.620619 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:18.620340    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-b7jh5" podStartSLOduration=3.725195542 podStartE2EDuration="21.620317046s" totalImagesPullingTime="17.895121504s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:14:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:18.61969547 +0000 UTC m=+21.770245551" watchObservedRunningTime="2026-09-11 14:15:18.620317046 +0000 UTC m=+21.770867111"
Sep 11 14:15:18.651317 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:18.651231    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-pnxhk" podStartSLOduration=11.003564989 podStartE2EDuration="21.651214243s" totalImagesPullingTime="10.647649254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:14:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:18.649582982 +0000 UTC m=+21.800133046" watchObservedRunningTime="2026-09-11 14:15:18.651214243 +0000 UTC m=+21.801764307"
Sep 11 14:15:19.493984 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:19.493773    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5sxd"
Sep 11 14:15:19.494173 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:19.493773    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wgmt4"
Sep 11 14:15:19.494173 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:19.494081    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5sxd" podUID="f58cdeac-ef49-4a6e-b459-ad21493d1497"
Sep 11 14:15:19.494173 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:19.494133    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wgmt4" podUID="d3376794-b10e-4353-b8f3-4639741019eb"
Sep 11 14:15:19.598048 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:19.598014    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n88qj" event={"ID":"fedc3186-f028-4964-a5e9-d6315fb07164","Type":"ContainerStarted","Data":"a8fb1600379152ae50e92bdb292bbfc3a55fcf3dfc342a6c9a3d11af5726f369"}
Sep 11 14:15:19.598703 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:19.598097    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n88qj" event={"ID":"fedc3186-f028-4964-a5e9-d6315fb07164","Type":"ContainerStarted","Data":"34cec89ca616521965d6c3ba303477dd020079ee334f1a66045aa3d2a9c5dd04"}
Sep 11 14:15:19.598703 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:19.598132    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n88qj" event={"ID":"fedc3186-f028-4964-a5e9-d6315fb07164","Type":"ContainerStarted","Data":"d3f14d355db059846fd396830256629ae79c1ad88906c626087e4f2be87316ef"}
Sep 11 14:15:19.598703 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:19.598161    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n88qj" event={"ID":"fedc3186-f028-4964-a5e9-d6315fb07164","Type":"ContainerStarted","Data":"e821e42efb492cc83e14551f10a3aa19ad24d18d510b4242e115f7bfc7b17c12"}
Sep 11 14:15:19.598703 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:19.598188    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n88qj" event={"ID":"fedc3186-f028-4964-a5e9-d6315fb07164","Type":"ContainerStarted","Data":"642deeea64fb275b96b6e307db7cc791365828cc9e479b678222a137d9de1e5f"}
Sep 11 14:15:19.598703 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:19.598224    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n88qj" event={"ID":"fedc3186-f028-4964-a5e9-d6315fb07164","Type":"ContainerStarted","Data":"1a72796794f58fbad655bad3ca6cc477b6f5ed2911015d41c834037061924bd6"}
Sep 11 14:15:19.599503 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:19.599477    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wk6kn" event={"ID":"757db884-d01a-4ab4-8b46-f9a41a3d7369","Type":"ContainerStarted","Data":"027f7e944e644f53b89932d1fd9900ee5d558b6a96e7e37bbb83a244a52bda42"}
Sep 11 14:15:19.601095 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:19.601073    2581 generic.go:356] "Generic (PLEG): container finished" podID="774c0511-8141-47cd-8718-b647be205839" containerID="5906c836ca1b2588bb7086398d7e199f2567ddde6aaff09889a152ee68a7436a" exitCode=0
Sep 11 14:15:19.601194 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:19.601140    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mn2mg" event={"ID":"774c0511-8141-47cd-8718-b647be205839","Type":"ContainerDied","Data":"5906c836ca1b2588bb7086398d7e199f2567ddde6aaff09889a152ee68a7436a"}
Sep 11 14:15:19.602700 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:19.602677    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qdthk" event={"ID":"2b4e57a8-379d-4375-95b4-8176b1e4e410","Type":"ContainerStarted","Data":"dab3f12f688a818e1fb73c9739fd30c42c96dbd93f8ed1f7fd7464047f480bcc"}
Sep 11 14:15:19.604348 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:19.604322    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx7nv" event={"ID":"9eadc143-6ef8-4518-8291-b323e6f15cb3","Type":"ContainerStarted","Data":"24d2b475050964f604e9c924a127931bbaea801a05ae1ed9087ae1efd9025d24"}
Sep 11 14:15:19.623065 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:19.623019    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-wk6kn" podStartSLOduration=4.718877141 podStartE2EDuration="22.623004719s" totalImagesPullingTime="17.904127578s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:14:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:19.62269902 +0000 UTC m=+22.773249085" watchObservedRunningTime="2026-09-11 14:15:19.623004719 +0000 UTC m=+22.773554780"
Sep 11 14:15:19.650739 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:19.650680    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-2cdd6" podStartSLOduration=4.699444858 podStartE2EDuration="22.650660645s" totalImagesPullingTime="17.951215787s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:14:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:19.65046161 +0000 UTC m=+22.801011675" watchObservedRunningTime="2026-09-11 14:15:19.650660645 +0000 UTC m=+22.801210713"
Sep 11 14:15:19.696227 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:19.696109    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qdthk" podStartSLOduration=4.798551201 podStartE2EDuration="22.696091685s" totalImagesPullingTime="17.897540484s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:14:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:19.671558168 +0000 UTC m=+22.822108232" watchObservedRunningTime="2026-09-11 14:15:19.696091685 +0000 UTC m=+22.846641816"
Sep 11 14:15:20.289194 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:20.289169    2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 11 14:15:20.464226 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:20.464073    2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T14:15:20.289191487Z","UUID":"01667d1d-099c-4abb-94a9-f6eddeb0f79b","Handler":null,"Name":"","Endpoint":""}
Sep 11 14:15:20.465730 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:20.465706    2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 11 14:15:20.465730 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:20.465743    2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 11 14:15:20.493529 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:20.493501    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqmtt"
Sep 11 14:15:20.493693 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:20.493632    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqmtt" podUID="78d2df0e-635a-4123-9a80-f9898562e613"
Sep 11 14:15:20.608708 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:20.608672    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx7nv" event={"ID":"9eadc143-6ef8-4518-8291-b323e6f15cb3","Type":"ContainerStarted","Data":"7728fca60525dc895b74ecb3b88e6f57435e2ac8f036a9ebbb04bbbb3d308b83"}
Sep 11 14:15:20.610433 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:20.610403    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-v285k" event={"ID":"07633f0c-6e0a-4bc2-811f-263730bfb6a4","Type":"ContainerStarted","Data":"da91f52ac6bce536d2a34e01cfbcf3bbba133cdc840a7576a6666065282c5b8f"}
Sep 11 14:15:20.632216 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:20.632165    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-v285k" podStartSLOduration=5.7315401040000005 podStartE2EDuration="23.632148521s" totalImagesPullingTime="17.900608417s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:14:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:20.631665948 +0000 UTC m=+23.782216013" watchObservedRunningTime="2026-09-11 14:15:20.632148521 +0000 UTC m=+23.782698585"
Sep 11 14:15:21.493822 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:21.493787    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wgmt4"
Sep 11 14:15:21.494015 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:21.493926    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wgmt4" podUID="d3376794-b10e-4353-b8f3-4639741019eb"
Sep 11 14:15:21.494015 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:21.493808    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5sxd"
Sep 11 14:15:21.494145 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:21.494054    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5sxd" podUID="f58cdeac-ef49-4a6e-b459-ad21493d1497"
Sep 11 14:15:21.616636 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:21.616604    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n88qj" event={"ID":"fedc3186-f028-4964-a5e9-d6315fb07164","Type":"ContainerStarted","Data":"d27ab8c50a4d72f3660096f1eaf7b816c0cd018766605c0a43f13f0fff6586f3"}
Sep 11 14:15:22.493981 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:22.493936    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqmtt"
Sep 11 14:15:22.494173 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:22.494075    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqmtt" podUID="78d2df0e-635a-4123-9a80-f9898562e613"
Sep 11 14:15:22.621498 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:22.621464    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx7nv" event={"ID":"9eadc143-6ef8-4518-8291-b323e6f15cb3","Type":"ContainerStarted","Data":"558869086b7862ac7418d4948ba5967019d6e77473830df42a8bc88932032d7f"}
Sep 11 14:15:22.645103 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:22.645056    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx7nv" podStartSLOduration=4.613557513 podStartE2EDuration="25.64503962s" totalImagesPullingTime="21.031482107s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:14:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:22.645014922 +0000 UTC m=+25.795564987" watchObservedRunningTime="2026-09-11 14:15:22.64503962 +0000 UTC m=+25.795589680"
Sep 11 14:15:23.455699 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:23.455390    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-pnxhk"
Sep 11 14:15:23.456114 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:23.456091    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-pnxhk"
Sep 11 14:15:23.493496 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:23.493464    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wgmt4"
Sep 11 14:15:23.493665 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:23.493475    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5sxd"
Sep 11 14:15:23.493665 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:23.493594    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wgmt4" podUID="d3376794-b10e-4353-b8f3-4639741019eb"
Sep 11 14:15:23.493665 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:23.493626    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5sxd" podUID="f58cdeac-ef49-4a6e-b459-ad21493d1497"
Sep 11 14:15:23.629609 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:23.629253    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n88qj" event={"ID":"fedc3186-f028-4964-a5e9-d6315fb07164","Type":"ContainerStarted","Data":"45aaebb41ada54f0ef7ac907eaff0012132145b2e7f26db091daee45cea98959"}
Sep 11 14:15:23.630645 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:23.629706    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:15:23.630645 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:23.630005    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:15:23.650778 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:23.650743    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:15:23.651852 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:23.651830    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:15:23.670252 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:23.670139    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-n88qj" podStartSLOduration=8.661999701 podStartE2EDuration="26.67011763s" totalImagesPullingTime="18.008117929s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:14:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:23.66929934 +0000 UTC m=+26.819849405" watchObservedRunningTime="2026-09-11 14:15:23.67011763 +0000 UTC m=+26.820667696"
Sep 11 14:15:24.493367 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:24.493336    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqmtt"
Sep 11 14:15:24.493537 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:24.493437    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqmtt" podUID="78d2df0e-635a-4123-9a80-f9898562e613"
Sep 11 14:15:24.633464 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:24.633435    2581 generic.go:356] "Generic (PLEG): container finished" podID="774c0511-8141-47cd-8718-b647be205839" containerID="245d58d1e80126724344251cb2d3c47f2cf8daa49247f51b072934bf2cd23706" exitCode=0
Sep 11 14:15:24.634019 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:24.633522    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mn2mg" event={"ID":"774c0511-8141-47cd-8718-b647be205839","Type":"ContainerDied","Data":"245d58d1e80126724344251cb2d3c47f2cf8daa49247f51b072934bf2cd23706"}
Sep 11 14:15:24.634019 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:24.633583    2581 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 11 14:15:24.940425 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:24.940339    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-pnxhk"
Sep 11 14:15:24.940576 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:24.940445    2581 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 11 14:15:24.940868 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:24.940854    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pnxhk"
Sep 11 14:15:25.494008 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:25.493803    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wgmt4"
Sep 11 14:15:25.494165 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:25.493815    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5sxd"
Sep 11 14:15:25.494165 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:25.494110    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wgmt4" podUID="d3376794-b10e-4353-b8f3-4639741019eb"
Sep 11 14:15:25.494165 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:25.494139    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5sxd" podUID="f58cdeac-ef49-4a6e-b459-ad21493d1497"
Sep 11 14:15:25.637574 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:25.637538    2581 generic.go:356] "Generic (PLEG): container finished" podID="774c0511-8141-47cd-8718-b647be205839" containerID="535157317f9a326716828571f134419d6b74f857003aaef75ef6e8ae75d81a0b" exitCode=0
Sep 11 14:15:25.637938 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:25.637618    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mn2mg" event={"ID":"774c0511-8141-47cd-8718-b647be205839","Type":"ContainerDied","Data":"535157317f9a326716828571f134419d6b74f857003aaef75ef6e8ae75d81a0b"}
Sep 11 14:15:25.637938 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:25.637734    2581 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 11 14:15:25.704384 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:25.704345    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wgmt4"]
Sep 11 14:15:25.704544 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:25.704490    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wgmt4"
Sep 11 14:15:25.704601 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:25.704573    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wgmt4" podUID="d3376794-b10e-4353-b8f3-4639741019eb"
Sep 11 14:15:25.716544 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:25.716518    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dqmtt"]
Sep 11 14:15:25.716677 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:25.716617    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqmtt"
Sep 11 14:15:25.716716 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:25.716696    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqmtt" podUID="78d2df0e-635a-4123-9a80-f9898562e613"
Sep 11 14:15:25.722562 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:25.722536    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-q5sxd"]
Sep 11 14:15:25.722673 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:25.722631    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5sxd"
Sep 11 14:15:25.722727 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:25.722704    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5sxd" podUID="f58cdeac-ef49-4a6e-b459-ad21493d1497"
Sep 11 14:15:26.642436 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:26.642400    2581 generic.go:356] "Generic (PLEG): container finished" podID="774c0511-8141-47cd-8718-b647be205839" containerID="6ffb62dc6a9a02cd40f308fe466e09d8b8e5e5fb9bcc1dcefaf6c09574979255" exitCode=0
Sep 11 14:15:26.642777 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:26.642465    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mn2mg" event={"ID":"774c0511-8141-47cd-8718-b647be205839","Type":"ContainerDied","Data":"6ffb62dc6a9a02cd40f308fe466e09d8b8e5e5fb9bcc1dcefaf6c09574979255"}
Sep 11 14:15:27.494055 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:27.493951    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqmtt"
Sep 11 14:15:27.494055 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:27.494052    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wgmt4"
Sep 11 14:15:27.494260 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:27.494081    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5sxd"
Sep 11 14:15:27.494260 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:27.494095    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqmtt" podUID="78d2df0e-635a-4123-9a80-f9898562e613"
Sep 11 14:15:27.494260 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:27.494137    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5sxd" podUID="f58cdeac-ef49-4a6e-b459-ad21493d1497"
Sep 11 14:15:27.494260 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:27.494240    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wgmt4" podUID="d3376794-b10e-4353-b8f3-4639741019eb"
Sep 11 14:15:29.333273 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:29.333229    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:15:29.333740 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:29.333501    2581 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 11 14:15:29.349746 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:29.349692    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-n88qj" podUID="fedc3186-f028-4964-a5e9-d6315fb07164" containerName="ovnkube-controller" probeResult="failure" output=""
Sep 11 14:15:29.360898 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:29.360860    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-n88qj" podUID="fedc3186-f028-4964-a5e9-d6315fb07164" containerName="ovnkube-controller" probeResult="failure" output=""
Sep 11 14:15:29.493822 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:29.493786    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqmtt"
Sep 11 14:15:29.494007 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:29.493831    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5sxd"
Sep 11 14:15:29.494007 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:29.493786    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wgmt4"
Sep 11 14:15:29.494007 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:29.493926    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqmtt" podUID="78d2df0e-635a-4123-9a80-f9898562e613"
Sep 11 14:15:29.494184 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:29.494030    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wgmt4" podUID="d3376794-b10e-4353-b8f3-4639741019eb"
Sep 11 14:15:29.494184 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:29.494111    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5sxd" podUID="f58cdeac-ef49-4a6e-b459-ad21493d1497"
Sep 11 14:15:31.010955 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:31.010797    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3376794-b10e-4353-b8f3-4639741019eb-metrics-certs\") pod \"network-metrics-daemon-wgmt4\" (UID: \"d3376794-b10e-4353-b8f3-4639741019eb\") " pod="openshift-multus/network-metrics-daemon-wgmt4"
Sep 11 14:15:31.011328 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:31.010989    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/78d2df0e-635a-4123-9a80-f9898562e613-original-pull-secret\") pod \"global-pull-secret-syncer-dqmtt\" (UID: \"78d2df0e-635a-4123-9a80-f9898562e613\") " pod="kube-system/global-pull-secret-syncer-dqmtt"
Sep 11 14:15:31.011328 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:31.010938    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 14:15:31.011328 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:31.011076    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3376794-b10e-4353-b8f3-4639741019eb-metrics-certs podName:d3376794-b10e-4353-b8f3-4639741019eb nodeName:}" failed. No retries permitted until 2026-09-11 14:16:03.011060746 +0000 UTC m=+66.161610792 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d3376794-b10e-4353-b8f3-4639741019eb-metrics-certs") pod "network-metrics-daemon-wgmt4" (UID: "d3376794-b10e-4353-b8f3-4639741019eb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 14:15:31.011328 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:31.011083    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 14:15:31.011328 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:31.011122    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78d2df0e-635a-4123-9a80-f9898562e613-original-pull-secret podName:78d2df0e-635a-4123-9a80-f9898562e613 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:03.011109486 +0000 UTC m=+66.161659529 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/78d2df0e-635a-4123-9a80-f9898562e613-original-pull-secret") pod "global-pull-secret-syncer-dqmtt" (UID: "78d2df0e-635a-4123-9a80-f9898562e613") : object "kube-system"/"original-pull-secret" not registered
Sep 11 14:15:31.111666 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:31.111629    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ffkb\" (UniqueName: \"kubernetes.io/projected/f58cdeac-ef49-4a6e-b459-ad21493d1497-kube-api-access-5ffkb\") pod \"network-check-target-q5sxd\" (UID: \"f58cdeac-ef49-4a6e-b459-ad21493d1497\") " pod="openshift-network-diagnostics/network-check-target-q5sxd"
Sep 11 14:15:31.111857 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:31.111833    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 14:15:31.111916 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:31.111864    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 14:15:31.111916 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:31.111877    2581 projected.go:196] Error preparing data for projected volume kube-api-access-5ffkb for pod openshift-network-diagnostics/network-check-target-q5sxd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 14:15:31.112005 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:31.111947    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f58cdeac-ef49-4a6e-b459-ad21493d1497-kube-api-access-5ffkb podName:f58cdeac-ef49-4a6e-b459-ad21493d1497 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:03.111926906 +0000 UTC m=+66.262476964 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-5ffkb" (UniqueName: "kubernetes.io/projected/f58cdeac-ef49-4a6e-b459-ad21493d1497-kube-api-access-5ffkb") pod "network-check-target-q5sxd" (UID: "f58cdeac-ef49-4a6e-b459-ad21493d1497") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 14:15:31.493571 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:31.493545    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqmtt"
Sep 11 14:15:31.493744 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:31.493648    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dqmtt" podUID="78d2df0e-635a-4123-9a80-f9898562e613"
Sep 11 14:15:31.493744 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:31.493549    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wgmt4"
Sep 11 14:15:31.493861 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:31.493787    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wgmt4" podUID="d3376794-b10e-4353-b8f3-4639741019eb"
Sep 11 14:15:31.493861 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:31.493823    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5sxd"
Sep 11 14:15:31.493963 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:31.493900    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q5sxd" podUID="f58cdeac-ef49-4a6e-b459-ad21493d1497"
Sep 11 14:15:31.656880 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:31.656842    2581 generic.go:356] "Generic (PLEG): container finished" podID="774c0511-8141-47cd-8718-b647be205839" containerID="b83776f3faf6102fd56af7ef3c4a32b5949d7cdf16f8e1b94d9e5c0067199cc3" exitCode=0
Sep 11 14:15:31.657050 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:31.656891    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mn2mg" event={"ID":"774c0511-8141-47cd-8718-b647be205839","Type":"ContainerDied","Data":"b83776f3faf6102fd56af7ef3c4a32b5949d7cdf16f8e1b94d9e5c0067199cc3"}
Sep 11 14:15:32.158562 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.158489    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-150.ec2.internal" event="NodeReady"
Sep 11 14:15:32.159001 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.158606    2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 11 14:15:32.201000 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.200952    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-fbb68bfc-66pmz"]
Sep 11 14:15:32.227082 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.227043    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-fbb68bfc-66pmz"]
Sep 11 14:15:32.227082 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.227084    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-gdl2n"]
Sep 11 14:15:32.227294 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.227248    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fbb68bfc-66pmz"
Sep 11 14:15:32.232070 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.232050    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 11 14:15:32.232209 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.232100    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-92zwk\""
Sep 11 14:15:32.232385 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.232370    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 11 14:15:32.232489 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.232465    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 11 14:15:32.239135 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.239116    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 11 14:15:32.242453 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.242435    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5s8kg"]
Sep 11 14:15:32.242556 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.242539    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gdl2n"
Sep 11 14:15:32.245119 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.245098    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 11 14:15:32.245274 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.245262    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-c4f62\""
Sep 11 14:15:32.245466 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.245452    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 11 14:15:32.260501 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.260470    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5s8kg"]
Sep 11 14:15:32.260616 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.260592    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5s8kg"
Sep 11 14:15:32.260664 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.260647    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gdl2n"]
Sep 11 14:15:32.263769 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.263310    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 11 14:15:32.263769 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.263706    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 11 14:15:32.263954 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.263778    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 11 14:15:32.263954 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.263815    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-xs77v\""
Sep 11 14:15:32.321702 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.321673    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5115e3f0-399e-4db4-a892-ba00f8d8735f-trusted-ca\") pod \"image-registry-fbb68bfc-66pmz\" (UID: \"5115e3f0-399e-4db4-a892-ba00f8d8735f\") " pod="openshift-image-registry/image-registry-fbb68bfc-66pmz"
Sep 11 14:15:32.321844 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.321709    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5115e3f0-399e-4db4-a892-ba00f8d8735f-installation-pull-secrets\") pod \"image-registry-fbb68bfc-66pmz\" (UID: \"5115e3f0-399e-4db4-a892-ba00f8d8735f\") " pod="openshift-image-registry/image-registry-fbb68bfc-66pmz"
Sep 11 14:15:32.321844 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.321733    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbfh4\" (UniqueName: \"kubernetes.io/projected/5115e3f0-399e-4db4-a892-ba00f8d8735f-kube-api-access-lbfh4\") pod \"image-registry-fbb68bfc-66pmz\" (UID: \"5115e3f0-399e-4db4-a892-ba00f8d8735f\") " pod="openshift-image-registry/image-registry-fbb68bfc-66pmz"
Sep 11 14:15:32.321844 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.321758    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98689afa-2179-4240-9ad9-cbe7744a84c1-tmp-dir\") pod \"dns-default-gdl2n\" (UID: \"98689afa-2179-4240-9ad9-cbe7744a84c1\") " pod="openshift-dns/dns-default-gdl2n"
Sep 11 14:15:32.321962 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.321837    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5115e3f0-399e-4db4-a892-ba00f8d8735f-registry-tls\") pod \"image-registry-fbb68bfc-66pmz\" (UID: \"5115e3f0-399e-4db4-a892-ba00f8d8735f\") " pod="openshift-image-registry/image-registry-fbb68bfc-66pmz"
Sep 11 14:15:32.321962 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.321871    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5115e3f0-399e-4db4-a892-ba00f8d8735f-ca-trust-extracted\") pod \"image-registry-fbb68bfc-66pmz\" (UID: \"5115e3f0-399e-4db4-a892-ba00f8d8735f\") " pod="openshift-image-registry/image-registry-fbb68bfc-66pmz"
Sep 11 14:15:32.321962 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.321911    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5115e3f0-399e-4db4-a892-ba00f8d8735f-registry-certificates\") pod \"image-registry-fbb68bfc-66pmz\" (UID: \"5115e3f0-399e-4db4-a892-ba00f8d8735f\") " pod="openshift-image-registry/image-registry-fbb68bfc-66pmz"
Sep 11 14:15:32.321962 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.321951    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5115e3f0-399e-4db4-a892-ba00f8d8735f-bound-sa-token\") pod \"image-registry-fbb68bfc-66pmz\" (UID: \"5115e3f0-399e-4db4-a892-ba00f8d8735f\") " pod="openshift-image-registry/image-registry-fbb68bfc-66pmz"
Sep 11 14:15:32.322116 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.322011    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/98689afa-2179-4240-9ad9-cbe7744a84c1-config-volume\") pod \"dns-default-gdl2n\" (UID: \"98689afa-2179-4240-9ad9-cbe7744a84c1\") " pod="openshift-dns/dns-default-gdl2n"
Sep 11 14:15:32.322116 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.322065    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5115e3f0-399e-4db4-a892-ba00f8d8735f-image-registry-private-configuration\") pod \"image-registry-fbb68bfc-66pmz\" (UID: \"5115e3f0-399e-4db4-a892-ba00f8d8735f\") " pod="openshift-image-registry/image-registry-fbb68bfc-66pmz"
Sep 11 14:15:32.322116 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.322102    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c55n2\" (UniqueName: \"kubernetes.io/projected/98689afa-2179-4240-9ad9-cbe7744a84c1-kube-api-access-c55n2\") pod \"dns-default-gdl2n\" (UID: \"98689afa-2179-4240-9ad9-cbe7744a84c1\") " pod="openshift-dns/dns-default-gdl2n"
Sep 11 14:15:32.322202 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.322132    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98689afa-2179-4240-9ad9-cbe7744a84c1-metrics-tls\") pod \"dns-default-gdl2n\" (UID: \"98689afa-2179-4240-9ad9-cbe7744a84c1\") " pod="openshift-dns/dns-default-gdl2n"
Sep 11 14:15:32.423194 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.423114    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5115e3f0-399e-4db4-a892-ba00f8d8735f-trusted-ca\") pod \"image-registry-fbb68bfc-66pmz\" (UID: \"5115e3f0-399e-4db4-a892-ba00f8d8735f\") " pod="openshift-image-registry/image-registry-fbb68bfc-66pmz"
Sep 11 14:15:32.423194 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.423153    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5115e3f0-399e-4db4-a892-ba00f8d8735f-installation-pull-secrets\") pod \"image-registry-fbb68bfc-66pmz\" (UID: \"5115e3f0-399e-4db4-a892-ba00f8d8735f\") " pod="openshift-image-registry/image-registry-fbb68bfc-66pmz"
Sep 11 14:15:32.423194 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.423170    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbfh4\" (UniqueName: \"kubernetes.io/projected/5115e3f0-399e-4db4-a892-ba00f8d8735f-kube-api-access-lbfh4\") pod \"image-registry-fbb68bfc-66pmz\" (UID: \"5115e3f0-399e-4db4-a892-ba00f8d8735f\") " pod="openshift-image-registry/image-registry-fbb68bfc-66pmz"
Sep 11 14:15:32.423194 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.423187    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98689afa-2179-4240-9ad9-cbe7744a84c1-tmp-dir\") pod \"dns-default-gdl2n\" (UID: \"98689afa-2179-4240-9ad9-cbe7744a84c1\") " pod="openshift-dns/dns-default-gdl2n"
Sep 11 14:15:32.423450 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.423320    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5115e3f0-399e-4db4-a892-ba00f8d8735f-registry-tls\") pod \"image-registry-fbb68bfc-66pmz\" (UID: \"5115e3f0-399e-4db4-a892-ba00f8d8735f\") " pod="openshift-image-registry/image-registry-fbb68bfc-66pmz"
Sep 11 14:15:32.423450 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.423354    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5115e3f0-399e-4db4-a892-ba00f8d8735f-ca-trust-extracted\") pod \"image-registry-fbb68bfc-66pmz\" (UID: \"5115e3f0-399e-4db4-a892-ba00f8d8735f\") " pod="openshift-image-registry/image-registry-fbb68bfc-66pmz"
Sep 11 14:15:32.423450 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.423370    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5115e3f0-399e-4db4-a892-ba00f8d8735f-registry-certificates\") pod \"image-registry-fbb68bfc-66pmz\" (UID: \"5115e3f0-399e-4db4-a892-ba00f8d8735f\") " pod="openshift-image-registry/image-registry-fbb68bfc-66pmz"
Sep 11 14:15:32.423450 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.423410    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7aefb1ab-dcdc-451e-b710-81a121895035-cert\") pod \"ingress-canary-5s8kg\" (UID: \"7aefb1ab-dcdc-451e-b710-81a121895035\") " pod="openshift-ingress-canary/ingress-canary-5s8kg"
Sep 11 14:15:32.423450 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.423435    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5115e3f0-399e-4db4-a892-ba00f8d8735f-bound-sa-token\") pod \"image-registry-fbb68bfc-66pmz\" (UID: \"5115e3f0-399e-4db4-a892-ba00f8d8735f\") " pod="openshift-image-registry/image-registry-fbb68bfc-66pmz"
Sep 11 14:15:32.423675 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.423460    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/98689afa-2179-4240-9ad9-cbe7744a84c1-config-volume\") pod \"dns-default-gdl2n\" (UID: \"98689afa-2179-4240-9ad9-cbe7744a84c1\") " pod="openshift-dns/dns-default-gdl2n"
Sep 11 14:15:32.423675 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.423484    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6crr\" (UniqueName: \"kubernetes.io/projected/7aefb1ab-dcdc-451e-b710-81a121895035-kube-api-access-g6crr\") pod \"ingress-canary-5s8kg\" (UID: \"7aefb1ab-dcdc-451e-b710-81a121895035\") " pod="openshift-ingress-canary/ingress-canary-5s8kg"
Sep 11 14:15:32.423675 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:32.423509    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 14:15:32.423675 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:32.423532    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fbb68bfc-66pmz: secret "image-registry-tls" not found
Sep 11 14:15:32.423675 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.423538    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98689afa-2179-4240-9ad9-cbe7744a84c1-tmp-dir\") pod \"dns-default-gdl2n\" (UID: \"98689afa-2179-4240-9ad9-cbe7744a84c1\") " pod="openshift-dns/dns-default-gdl2n"
Sep 11 14:15:32.423675 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.423547    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5115e3f0-399e-4db4-a892-ba00f8d8735f-image-registry-private-configuration\") pod \"image-registry-fbb68bfc-66pmz\" (UID: \"5115e3f0-399e-4db4-a892-ba00f8d8735f\") " pod="openshift-image-registry/image-registry-fbb68bfc-66pmz"
Sep 11 14:15:32.423675 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.423602    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c55n2\" (UniqueName: \"kubernetes.io/projected/98689afa-2179-4240-9ad9-cbe7744a84c1-kube-api-access-c55n2\") pod \"dns-default-gdl2n\" (UID: \"98689afa-2179-4240-9ad9-cbe7744a84c1\") " pod="openshift-dns/dns-default-gdl2n"
Sep 11 14:15:32.423675 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:32.423627    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5115e3f0-399e-4db4-a892-ba00f8d8735f-registry-tls podName:5115e3f0-399e-4db4-a892-ba00f8d8735f nodeName:}" failed. No retries permitted until 2026-09-11 14:15:32.923605706 +0000 UTC m=+36.074155764 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5115e3f0-399e-4db4-a892-ba00f8d8735f-registry-tls") pod "image-registry-fbb68bfc-66pmz" (UID: "5115e3f0-399e-4db4-a892-ba00f8d8735f") : secret "image-registry-tls" not found
Sep 11 14:15:32.424061 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.423684    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98689afa-2179-4240-9ad9-cbe7744a84c1-metrics-tls\") pod \"dns-default-gdl2n\" (UID: \"98689afa-2179-4240-9ad9-cbe7744a84c1\") " pod="openshift-dns/dns-default-gdl2n"
Sep 11 14:15:32.424061 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.423748    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5115e3f0-399e-4db4-a892-ba00f8d8735f-ca-trust-extracted\") pod \"image-registry-fbb68bfc-66pmz\" (UID: \"5115e3f0-399e-4db4-a892-ba00f8d8735f\") " pod="openshift-image-registry/image-registry-fbb68bfc-66pmz"
Sep 11 14:15:32.424061 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:32.423780    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 14:15:32.424061 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:32.423817    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98689afa-2179-4240-9ad9-cbe7744a84c1-metrics-tls podName:98689afa-2179-4240-9ad9-cbe7744a84c1 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:32.923805087 +0000 UTC m=+36.074355153 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/98689afa-2179-4240-9ad9-cbe7744a84c1-metrics-tls") pod "dns-default-gdl2n" (UID: "98689afa-2179-4240-9ad9-cbe7744a84c1") : secret "dns-default-metrics-tls" not found
Sep 11 14:15:32.424229 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.424086    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/98689afa-2179-4240-9ad9-cbe7744a84c1-config-volume\") pod \"dns-default-gdl2n\" (UID: \"98689afa-2179-4240-9ad9-cbe7744a84c1\") " pod="openshift-dns/dns-default-gdl2n"
Sep 11 14:15:32.424229 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.424120    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5115e3f0-399e-4db4-a892-ba00f8d8735f-trusted-ca\") pod \"image-registry-fbb68bfc-66pmz\" (UID: \"5115e3f0-399e-4db4-a892-ba00f8d8735f\") " pod="openshift-image-registry/image-registry-fbb68bfc-66pmz"
Sep 11 14:15:32.424229 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.424211    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5115e3f0-399e-4db4-a892-ba00f8d8735f-registry-certificates\") pod \"image-registry-fbb68bfc-66pmz\" (UID: \"5115e3f0-399e-4db4-a892-ba00f8d8735f\") " pod="openshift-image-registry/image-registry-fbb68bfc-66pmz"
Sep 11 14:15:32.427335 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.427314    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5115e3f0-399e-4db4-a892-ba00f8d8735f-image-registry-private-configuration\") pod \"image-registry-fbb68bfc-66pmz\" (UID: \"5115e3f0-399e-4db4-a892-ba00f8d8735f\") " pod="openshift-image-registry/image-registry-fbb68bfc-66pmz"
Sep 11 14:15:32.427443 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.427319    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5115e3f0-399e-4db4-a892-ba00f8d8735f-installation-pull-secrets\") pod \"image-registry-fbb68bfc-66pmz\" (UID: \"5115e3f0-399e-4db4-a892-ba00f8d8735f\") " pod="openshift-image-registry/image-registry-fbb68bfc-66pmz"
Sep 11 14:15:32.435518 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.435492    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5115e3f0-399e-4db4-a892-ba00f8d8735f-bound-sa-token\") pod \"image-registry-fbb68bfc-66pmz\" (UID: \"5115e3f0-399e-4db4-a892-ba00f8d8735f\") " pod="openshift-image-registry/image-registry-fbb68bfc-66pmz"
Sep 11 14:15:32.435917 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.435896    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c55n2\" (UniqueName: \"kubernetes.io/projected/98689afa-2179-4240-9ad9-cbe7744a84c1-kube-api-access-c55n2\") pod \"dns-default-gdl2n\" (UID: \"98689afa-2179-4240-9ad9-cbe7744a84c1\") " pod="openshift-dns/dns-default-gdl2n"
Sep 11 14:15:32.436641 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.436619    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbfh4\" (UniqueName: \"kubernetes.io/projected/5115e3f0-399e-4db4-a892-ba00f8d8735f-kube-api-access-lbfh4\") pod \"image-registry-fbb68bfc-66pmz\" (UID: \"5115e3f0-399e-4db4-a892-ba00f8d8735f\") " pod="openshift-image-registry/image-registry-fbb68bfc-66pmz"
Sep 11 14:15:32.524410 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.524375    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7aefb1ab-dcdc-451e-b710-81a121895035-cert\") pod \"ingress-canary-5s8kg\" (UID: \"7aefb1ab-dcdc-451e-b710-81a121895035\") " pod="openshift-ingress-canary/ingress-canary-5s8kg"
Sep 11 14:15:32.524410 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.524412    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6crr\" (UniqueName: \"kubernetes.io/projected/7aefb1ab-dcdc-451e-b710-81a121895035-kube-api-access-g6crr\") pod \"ingress-canary-5s8kg\" (UID: \"7aefb1ab-dcdc-451e-b710-81a121895035\") " pod="openshift-ingress-canary/ingress-canary-5s8kg"
Sep 11 14:15:32.526706 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.526680    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7aefb1ab-dcdc-451e-b710-81a121895035-cert\") pod \"ingress-canary-5s8kg\" (UID: \"7aefb1ab-dcdc-451e-b710-81a121895035\") " pod="openshift-ingress-canary/ingress-canary-5s8kg"
Sep 11 14:15:32.535687 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.535664    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6crr\" (UniqueName: \"kubernetes.io/projected/7aefb1ab-dcdc-451e-b710-81a121895035-kube-api-access-g6crr\") pod \"ingress-canary-5s8kg\" (UID: \"7aefb1ab-dcdc-451e-b710-81a121895035\") " pod="openshift-ingress-canary/ingress-canary-5s8kg"
Sep 11 14:15:32.568833 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.568806    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5s8kg"
Sep 11 14:15:32.663347 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.663312    2581 generic.go:356] "Generic (PLEG): container finished" podID="774c0511-8141-47cd-8718-b647be205839" containerID="d01eb3e1555338ecdcf675d96deef2fabaa273ee74d405caf96bad5661918dca" exitCode=0
Sep 11 14:15:32.663527 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.663391    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mn2mg" event={"ID":"774c0511-8141-47cd-8718-b647be205839","Type":"ContainerDied","Data":"d01eb3e1555338ecdcf675d96deef2fabaa273ee74d405caf96bad5661918dca"}
Sep 11 14:15:32.901050 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.900847    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5s8kg"]
Sep 11 14:15:32.904616 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:15:32.904593    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aefb1ab_dcdc_451e_b710_81a121895035.slice/crio-93a02cca01c708c8bdca142552ff1558ae4c5571919f9234c09c25c9637ab11c WatchSource:0}: Error finding container 93a02cca01c708c8bdca142552ff1558ae4c5571919f9234c09c25c9637ab11c: Status 404 returned error can't find the container with id 93a02cca01c708c8bdca142552ff1558ae4c5571919f9234c09c25c9637ab11c
Sep 11 14:15:32.928540 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.928442    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98689afa-2179-4240-9ad9-cbe7744a84c1-metrics-tls\") pod \"dns-default-gdl2n\" (UID: \"98689afa-2179-4240-9ad9-cbe7744a84c1\") " pod="openshift-dns/dns-default-gdl2n"
Sep 11 14:15:32.928540 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:32.928487    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5115e3f0-399e-4db4-a892-ba00f8d8735f-registry-tls\") pod \"image-registry-fbb68bfc-66pmz\" (UID: \"5115e3f0-399e-4db4-a892-ba00f8d8735f\") " pod="openshift-image-registry/image-registry-fbb68bfc-66pmz"
Sep 11 14:15:32.928715 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:32.928581    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 14:15:32.928715 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:32.928615    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 14:15:32.928715 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:32.928629    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fbb68bfc-66pmz: secret "image-registry-tls" not found
Sep 11 14:15:32.928715 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:32.928638    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98689afa-2179-4240-9ad9-cbe7744a84c1-metrics-tls podName:98689afa-2179-4240-9ad9-cbe7744a84c1 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:33.928622658 +0000 UTC m=+37.079172705 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/98689afa-2179-4240-9ad9-cbe7744a84c1-metrics-tls") pod "dns-default-gdl2n" (UID: "98689afa-2179-4240-9ad9-cbe7744a84c1") : secret "dns-default-metrics-tls" not found
Sep 11 14:15:32.928715 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:32.928686    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5115e3f0-399e-4db4-a892-ba00f8d8735f-registry-tls podName:5115e3f0-399e-4db4-a892-ba00f8d8735f nodeName:}" failed. No retries permitted until 2026-09-11 14:15:33.92866988 +0000 UTC m=+37.079219922 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5115e3f0-399e-4db4-a892-ba00f8d8735f-registry-tls") pod "image-registry-fbb68bfc-66pmz" (UID: "5115e3f0-399e-4db4-a892-ba00f8d8735f") : secret "image-registry-tls" not found
Sep 11 14:15:33.497338 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:33.497307    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5sxd"
Sep 11 14:15:33.498067 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:33.497413    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqmtt"
Sep 11 14:15:33.498067 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:33.497620    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wgmt4"
Sep 11 14:15:33.500504 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:33.500486    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 14:15:33.501686 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:33.501620    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-96vl6\""
Sep 11 14:15:33.501686 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:33.501640    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 14:15:33.501686 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:33.501678    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 14:15:33.501928 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:33.501729    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 14:15:33.502006 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:33.501983    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-brmn7\""
Sep 11 14:15:33.667538 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:33.667503    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5s8kg" event={"ID":"7aefb1ab-dcdc-451e-b710-81a121895035","Type":"ContainerStarted","Data":"93a02cca01c708c8bdca142552ff1558ae4c5571919f9234c09c25c9637ab11c"}
Sep 11 14:15:33.671891 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:33.671858    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mn2mg" event={"ID":"774c0511-8141-47cd-8718-b647be205839","Type":"ContainerStarted","Data":"dbe467a3166bcc0e5f737c75c143638e050216c46110ad66661f0158cc9b1ef7"}
Sep 11 14:15:33.702434 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:33.702384    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-mn2mg" podStartSLOduration=9.189142747 podStartE2EDuration="36.7023631s" totalImagesPullingTime="27.513220353s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:14:57 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:33.70074031 +0000 UTC m=+36.851290376" watchObservedRunningTime="2026-09-11 14:15:33.7023631 +0000 UTC m=+36.852913163"
Sep 11 14:15:33.938102 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:33.938019    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98689afa-2179-4240-9ad9-cbe7744a84c1-metrics-tls\") pod \"dns-default-gdl2n\" (UID: \"98689afa-2179-4240-9ad9-cbe7744a84c1\") " pod="openshift-dns/dns-default-gdl2n"
Sep 11 14:15:33.938102 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:33.938085    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5115e3f0-399e-4db4-a892-ba00f8d8735f-registry-tls\") pod \"image-registry-fbb68bfc-66pmz\" (UID: \"5115e3f0-399e-4db4-a892-ba00f8d8735f\") " pod="openshift-image-registry/image-registry-fbb68bfc-66pmz"
Sep 11 14:15:33.938335 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:33.938237    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 14:15:33.938335 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:33.938243    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 14:15:33.938335 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:33.938264    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fbb68bfc-66pmz: secret "image-registry-tls" not found
Sep 11 14:15:33.938482 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:33.938349    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5115e3f0-399e-4db4-a892-ba00f8d8735f-registry-tls podName:5115e3f0-399e-4db4-a892-ba00f8d8735f nodeName:}" failed. No retries permitted until 2026-09-11 14:15:35.938326074 +0000 UTC m=+39.088876130 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5115e3f0-399e-4db4-a892-ba00f8d8735f-registry-tls") pod "image-registry-fbb68bfc-66pmz" (UID: "5115e3f0-399e-4db4-a892-ba00f8d8735f") : secret "image-registry-tls" not found
Sep 11 14:15:33.938482 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:33.938372    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98689afa-2179-4240-9ad9-cbe7744a84c1-metrics-tls podName:98689afa-2179-4240-9ad9-cbe7744a84c1 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:35.938361649 +0000 UTC m=+39.088911714 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/98689afa-2179-4240-9ad9-cbe7744a84c1-metrics-tls") pod "dns-default-gdl2n" (UID: "98689afa-2179-4240-9ad9-cbe7744a84c1") : secret "dns-default-metrics-tls" not found
Sep 11 14:15:35.678999 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:35.678946    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5s8kg" event={"ID":"7aefb1ab-dcdc-451e-b710-81a121895035","Type":"ContainerStarted","Data":"53ff5bb358723205a6cbc009faeddf0dd3f83ba6a836afce5f31f7346927badd"}
Sep 11 14:15:35.705879 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:35.705821    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5s8kg" podStartSLOduration=1.187084152 podStartE2EDuration="3.705806091s" totalImagesPullingTime="2.518721939s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:15:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:35.704195027 +0000 UTC m=+38.854745091" watchObservedRunningTime="2026-09-11 14:15:35.705806091 +0000 UTC m=+38.856356154"
Sep 11 14:15:35.954364 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:35.954172    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98689afa-2179-4240-9ad9-cbe7744a84c1-metrics-tls\") pod \"dns-default-gdl2n\" (UID: \"98689afa-2179-4240-9ad9-cbe7744a84c1\") " pod="openshift-dns/dns-default-gdl2n"
Sep 11 14:15:35.954571 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:35.954385    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5115e3f0-399e-4db4-a892-ba00f8d8735f-registry-tls\") pod \"image-registry-fbb68bfc-66pmz\" (UID: \"5115e3f0-399e-4db4-a892-ba00f8d8735f\") " pod="openshift-image-registry/image-registry-fbb68bfc-66pmz"
Sep 11 14:15:35.954571 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:35.954325    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 14:15:35.954571 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:35.954481    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98689afa-2179-4240-9ad9-cbe7744a84c1-metrics-tls podName:98689afa-2179-4240-9ad9-cbe7744a84c1 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:39.954465584 +0000 UTC m=+43.105015638 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/98689afa-2179-4240-9ad9-cbe7744a84c1-metrics-tls") pod "dns-default-gdl2n" (UID: "98689afa-2179-4240-9ad9-cbe7744a84c1") : secret "dns-default-metrics-tls" not found
Sep 11 14:15:35.954571 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:35.954531    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 14:15:35.954571 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:35.954544    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fbb68bfc-66pmz: secret "image-registry-tls" not found
Sep 11 14:15:35.954775 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:35.954597    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5115e3f0-399e-4db4-a892-ba00f8d8735f-registry-tls podName:5115e3f0-399e-4db4-a892-ba00f8d8735f nodeName:}" failed. No retries permitted until 2026-09-11 14:15:39.954582655 +0000 UTC m=+43.105132714 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5115e3f0-399e-4db4-a892-ba00f8d8735f-registry-tls") pod "image-registry-fbb68bfc-66pmz" (UID: "5115e3f0-399e-4db4-a892-ba00f8d8735f") : secret "image-registry-tls" not found
Sep 11 14:15:39.982420 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:39.982380    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98689afa-2179-4240-9ad9-cbe7744a84c1-metrics-tls\") pod \"dns-default-gdl2n\" (UID: \"98689afa-2179-4240-9ad9-cbe7744a84c1\") " pod="openshift-dns/dns-default-gdl2n"
Sep 11 14:15:39.982420 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:39.982426    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5115e3f0-399e-4db4-a892-ba00f8d8735f-registry-tls\") pod \"image-registry-fbb68bfc-66pmz\" (UID: \"5115e3f0-399e-4db4-a892-ba00f8d8735f\") " pod="openshift-image-registry/image-registry-fbb68bfc-66pmz"
Sep 11 14:15:39.982834 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:39.982546    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 14:15:39.982834 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:39.982563    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 14:15:39.982834 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:39.982574    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fbb68bfc-66pmz: secret "image-registry-tls" not found
Sep 11 14:15:39.982834 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:39.982607    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98689afa-2179-4240-9ad9-cbe7744a84c1-metrics-tls podName:98689afa-2179-4240-9ad9-cbe7744a84c1 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:47.982592931 +0000 UTC m=+51.133142973 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/98689afa-2179-4240-9ad9-cbe7744a84c1-metrics-tls") pod "dns-default-gdl2n" (UID: "98689afa-2179-4240-9ad9-cbe7744a84c1") : secret "dns-default-metrics-tls" not found
Sep 11 14:15:39.982834 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:39.982621    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5115e3f0-399e-4db4-a892-ba00f8d8735f-registry-tls podName:5115e3f0-399e-4db4-a892-ba00f8d8735f nodeName:}" failed. No retries permitted until 2026-09-11 14:15:47.982615025 +0000 UTC m=+51.133165066 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5115e3f0-399e-4db4-a892-ba00f8d8735f-registry-tls") pod "image-registry-fbb68bfc-66pmz" (UID: "5115e3f0-399e-4db4-a892-ba00f8d8735f") : secret "image-registry-tls" not found
Sep 11 14:15:48.036240 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:48.036196    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98689afa-2179-4240-9ad9-cbe7744a84c1-metrics-tls\") pod \"dns-default-gdl2n\" (UID: \"98689afa-2179-4240-9ad9-cbe7744a84c1\") " pod="openshift-dns/dns-default-gdl2n"
Sep 11 14:15:48.036708 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:48.036249    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5115e3f0-399e-4db4-a892-ba00f8d8735f-registry-tls\") pod \"image-registry-fbb68bfc-66pmz\" (UID: \"5115e3f0-399e-4db4-a892-ba00f8d8735f\") " pod="openshift-image-registry/image-registry-fbb68bfc-66pmz"
Sep 11 14:15:48.036708 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:48.036345    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 14:15:48.036708 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:48.036357    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fbb68bfc-66pmz: secret "image-registry-tls" not found
Sep 11 14:15:48.036708 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:48.036361    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 14:15:48.036708 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:48.036422    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5115e3f0-399e-4db4-a892-ba00f8d8735f-registry-tls podName:5115e3f0-399e-4db4-a892-ba00f8d8735f nodeName:}" failed. No retries permitted until 2026-09-11 14:16:04.036405459 +0000 UTC m=+67.186955503 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5115e3f0-399e-4db4-a892-ba00f8d8735f-registry-tls") pod "image-registry-fbb68bfc-66pmz" (UID: "5115e3f0-399e-4db4-a892-ba00f8d8735f") : secret "image-registry-tls" not found
Sep 11 14:15:48.036708 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:15:48.036435    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98689afa-2179-4240-9ad9-cbe7744a84c1-metrics-tls podName:98689afa-2179-4240-9ad9-cbe7744a84c1 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:04.036429336 +0000 UTC m=+67.186979378 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/98689afa-2179-4240-9ad9-cbe7744a84c1-metrics-tls") pod "dns-default-gdl2n" (UID: "98689afa-2179-4240-9ad9-cbe7744a84c1") : secret "dns-default-metrics-tls" not found
Sep 11 14:15:59.360088 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:15:59.360056    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n88qj"
Sep 11 14:16:03.048579 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:03.048537    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3376794-b10e-4353-b8f3-4639741019eb-metrics-certs\") pod \"network-metrics-daemon-wgmt4\" (UID: \"d3376794-b10e-4353-b8f3-4639741019eb\") " pod="openshift-multus/network-metrics-daemon-wgmt4"
Sep 11 14:16:03.048579 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:03.048579    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/78d2df0e-635a-4123-9a80-f9898562e613-original-pull-secret\") pod \"global-pull-secret-syncer-dqmtt\" (UID: \"78d2df0e-635a-4123-9a80-f9898562e613\") " pod="kube-system/global-pull-secret-syncer-dqmtt"
Sep 11 14:16:03.051955 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:03.051931    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 14:16:03.052060 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:03.051944    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 14:16:03.059693 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:16:03.059673    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 11 14:16:03.059801 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:16:03.059730    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3376794-b10e-4353-b8f3-4639741019eb-metrics-certs podName:d3376794-b10e-4353-b8f3-4639741019eb nodeName:}" failed. No retries permitted until 2026-09-11 14:17:07.059714025 +0000 UTC m=+130.210264066 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d3376794-b10e-4353-b8f3-4639741019eb-metrics-certs") pod "network-metrics-daemon-wgmt4" (UID: "d3376794-b10e-4353-b8f3-4639741019eb") : secret "metrics-daemon-secret" not found
Sep 11 14:16:03.063358 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:03.063326    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/78d2df0e-635a-4123-9a80-f9898562e613-original-pull-secret\") pod \"global-pull-secret-syncer-dqmtt\" (UID: \"78d2df0e-635a-4123-9a80-f9898562e613\") " pod="kube-system/global-pull-secret-syncer-dqmtt"
Sep 11 14:16:03.149807 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:03.149766    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ffkb\" (UniqueName: \"kubernetes.io/projected/f58cdeac-ef49-4a6e-b459-ad21493d1497-kube-api-access-5ffkb\") pod \"network-check-target-q5sxd\" (UID: \"f58cdeac-ef49-4a6e-b459-ad21493d1497\") " pod="openshift-network-diagnostics/network-check-target-q5sxd"
Sep 11 14:16:03.153033 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:03.153016    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 14:16:03.163539 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:03.163511    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 14:16:03.173567 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:03.173539    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ffkb\" (UniqueName: \"kubernetes.io/projected/f58cdeac-ef49-4a6e-b459-ad21493d1497-kube-api-access-5ffkb\") pod \"network-check-target-q5sxd\" (UID: \"f58cdeac-ef49-4a6e-b459-ad21493d1497\") " pod="openshift-network-diagnostics/network-check-target-q5sxd"
Sep 11 14:16:03.212478 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:03.212450    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-brmn7\""
Sep 11 14:16:03.217352 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:03.217336    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dqmtt"
Sep 11 14:16:03.219581 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:03.219554    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q5sxd"
Sep 11 14:16:03.514644 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:03.514612    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dqmtt"]
Sep 11 14:16:03.518128 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:03.518084    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-q5sxd"]
Sep 11 14:16:03.519516 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:16:03.519454    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78d2df0e_635a_4123_9a80_f9898562e613.slice/crio-d6835567aadce8e1306a6a3a35b41a7fe7f5359a7de11ce7d814c24434f27591 WatchSource:0}: Error finding container d6835567aadce8e1306a6a3a35b41a7fe7f5359a7de11ce7d814c24434f27591: Status 404 returned error can't find the container with id d6835567aadce8e1306a6a3a35b41a7fe7f5359a7de11ce7d814c24434f27591
Sep 11 14:16:03.521536 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:16:03.521508    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf58cdeac_ef49_4a6e_b459_ad21493d1497.slice/crio-e74437462d793b129fa3e2797574a08693f3b3c7b302fe030dd07ba4afdc9557 WatchSource:0}: Error finding container e74437462d793b129fa3e2797574a08693f3b3c7b302fe030dd07ba4afdc9557: Status 404 returned error can't find the container with id e74437462d793b129fa3e2797574a08693f3b3c7b302fe030dd07ba4afdc9557
Sep 11 14:16:03.745379 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:03.745333    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dqmtt" event={"ID":"78d2df0e-635a-4123-9a80-f9898562e613","Type":"ContainerStarted","Data":"d6835567aadce8e1306a6a3a35b41a7fe7f5359a7de11ce7d814c24434f27591"}
Sep 11 14:16:03.746400 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:03.746377    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-q5sxd" event={"ID":"f58cdeac-ef49-4a6e-b459-ad21493d1497","Type":"ContainerStarted","Data":"e74437462d793b129fa3e2797574a08693f3b3c7b302fe030dd07ba4afdc9557"}
Sep 11 14:16:04.056272 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:04.056223    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98689afa-2179-4240-9ad9-cbe7744a84c1-metrics-tls\") pod \"dns-default-gdl2n\" (UID: \"98689afa-2179-4240-9ad9-cbe7744a84c1\") " pod="openshift-dns/dns-default-gdl2n"
Sep 11 14:16:04.056641 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:04.056284    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5115e3f0-399e-4db4-a892-ba00f8d8735f-registry-tls\") pod \"image-registry-fbb68bfc-66pmz\" (UID: \"5115e3f0-399e-4db4-a892-ba00f8d8735f\") " pod="openshift-image-registry/image-registry-fbb68bfc-66pmz"
Sep 11 14:16:04.056641 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:16:04.056366    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 14:16:04.056641 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:16:04.056398    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 14:16:04.056641 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:16:04.056409    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fbb68bfc-66pmz: secret "image-registry-tls" not found
Sep 11 14:16:04.056641 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:16:04.056445    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98689afa-2179-4240-9ad9-cbe7744a84c1-metrics-tls podName:98689afa-2179-4240-9ad9-cbe7744a84c1 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:36.056428072 +0000 UTC m=+99.206978114 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/98689afa-2179-4240-9ad9-cbe7744a84c1-metrics-tls") pod "dns-default-gdl2n" (UID: "98689afa-2179-4240-9ad9-cbe7744a84c1") : secret "dns-default-metrics-tls" not found
Sep 11 14:16:04.056641 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:16:04.056460    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5115e3f0-399e-4db4-a892-ba00f8d8735f-registry-tls podName:5115e3f0-399e-4db4-a892-ba00f8d8735f nodeName:}" failed. No retries permitted until 2026-09-11 14:16:36.056454301 +0000 UTC m=+99.207004342 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5115e3f0-399e-4db4-a892-ba00f8d8735f-registry-tls") pod "image-registry-fbb68bfc-66pmz" (UID: "5115e3f0-399e-4db4-a892-ba00f8d8735f") : secret "image-registry-tls" not found
Sep 11 14:16:08.763285 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:08.763229    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-q5sxd" event={"ID":"f58cdeac-ef49-4a6e-b459-ad21493d1497","Type":"ContainerStarted","Data":"d08aae311e33ffd8a1f056d756c7329b41ed91edea6e0bef6b5bad9c2d7002e7"}
Sep 11 14:16:08.763735 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:08.763417    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-q5sxd"
Sep 11 14:16:08.764840 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:08.764804    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dqmtt" event={"ID":"78d2df0e-635a-4123-9a80-f9898562e613","Type":"ContainerStarted","Data":"98c536193878b1c639827673a5e90735cfd2b4d2745d323b53581abbfb937e4b"}
Sep 11 14:16:08.784365 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:08.784311    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-q5sxd" podStartSLOduration=66.656289148 podStartE2EDuration="1m11.784295068s" totalImagesPullingTime="5.12800592s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:14:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:08.783589533 +0000 UTC m=+71.934139612" watchObservedRunningTime="2026-09-11 14:16:08.784295068 +0000 UTC m=+71.934845131"
Sep 11 14:16:08.805279 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:08.805219    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-dqmtt" podStartSLOduration=66.675256927 podStartE2EDuration="1m11.805200419s" totalImagesPullingTime="5.129943492s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:14:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:08.804315692 +0000 UTC m=+71.954865756" watchObservedRunningTime="2026-09-11 14:16:08.805200419 +0000 UTC m=+71.955750490"
Sep 11 14:16:12.233000 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:12.232952    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-psv5r"]
Sep 11 14:16:12.235907 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:12.235890    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-psv5r"
Sep 11 14:16:12.238775 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:12.238754    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 11 14:16:12.238775 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:12.238775    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 11 14:16:12.238985 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:12.238912    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 11 14:16:12.239046 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:12.239022    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rj6zn\""
Sep 11 14:16:12.240012 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:12.239995    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 11 14:16:12.253997 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:12.253956    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-psv5r"]
Sep 11 14:16:12.319346 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:12.319311    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e83e20c-2474-4cb0-9a1c-f7025b7383c5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-psv5r\" (UID: \"0e83e20c-2474-4cb0-9a1c-f7025b7383c5\") " pod="openshift-insights/insights-runtime-extractor-psv5r"
Sep 11 14:16:12.319346 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:12.319347    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpd29\" (UniqueName: \"kubernetes.io/projected/0e83e20c-2474-4cb0-9a1c-f7025b7383c5-kube-api-access-cpd29\") pod \"insights-runtime-extractor-psv5r\" (UID: \"0e83e20c-2474-4cb0-9a1c-f7025b7383c5\") " pod="openshift-insights/insights-runtime-extractor-psv5r"
Sep 11 14:16:12.319542 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:12.319387    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0e83e20c-2474-4cb0-9a1c-f7025b7383c5-crio-socket\") pod \"insights-runtime-extractor-psv5r\" (UID: \"0e83e20c-2474-4cb0-9a1c-f7025b7383c5\") " pod="openshift-insights/insights-runtime-extractor-psv5r"
Sep 11 14:16:12.319542 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:12.319437    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0e83e20c-2474-4cb0-9a1c-f7025b7383c5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-psv5r\" (UID: \"0e83e20c-2474-4cb0-9a1c-f7025b7383c5\") " pod="openshift-insights/insights-runtime-extractor-psv5r"
Sep 11 14:16:12.319542 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:12.319472    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0e83e20c-2474-4cb0-9a1c-f7025b7383c5-data-volume\") pod \"insights-runtime-extractor-psv5r\" (UID: \"0e83e20c-2474-4cb0-9a1c-f7025b7383c5\") " pod="openshift-insights/insights-runtime-extractor-psv5r"
Sep 11 14:16:12.420441 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:12.420404    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0e83e20c-2474-4cb0-9a1c-f7025b7383c5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-psv5r\" (UID: \"0e83e20c-2474-4cb0-9a1c-f7025b7383c5\") " pod="openshift-insights/insights-runtime-extractor-psv5r"
Sep 11 14:16:12.420621 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:12.420455    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0e83e20c-2474-4cb0-9a1c-f7025b7383c5-data-volume\") pod \"insights-runtime-extractor-psv5r\" (UID: \"0e83e20c-2474-4cb0-9a1c-f7025b7383c5\") " pod="openshift-insights/insights-runtime-extractor-psv5r"
Sep 11 14:16:12.420621 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:12.420486    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e83e20c-2474-4cb0-9a1c-f7025b7383c5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-psv5r\" (UID: \"0e83e20c-2474-4cb0-9a1c-f7025b7383c5\") " pod="openshift-insights/insights-runtime-extractor-psv5r"
Sep 11 14:16:12.420621 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:12.420513    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpd29\" (UniqueName: \"kubernetes.io/projected/0e83e20c-2474-4cb0-9a1c-f7025b7383c5-kube-api-access-cpd29\") pod \"insights-runtime-extractor-psv5r\" (UID: \"0e83e20c-2474-4cb0-9a1c-f7025b7383c5\") " pod="openshift-insights/insights-runtime-extractor-psv5r"
Sep 11 14:16:12.420621 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:16:12.420538    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 14:16:12.420621 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:12.420562    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0e83e20c-2474-4cb0-9a1c-f7025b7383c5-crio-socket\") pod \"insights-runtime-extractor-psv5r\" (UID: \"0e83e20c-2474-4cb0-9a1c-f7025b7383c5\") " pod="openshift-insights/insights-runtime-extractor-psv5r"
Sep 11 14:16:12.420621 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:16:12.420621    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0e83e20c-2474-4cb0-9a1c-f7025b7383c5-kube-rbac-proxy-cm podName:0e83e20c-2474-4cb0-9a1c-f7025b7383c5 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:12.920601232 +0000 UTC m=+76.071151289 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0e83e20c-2474-4cb0-9a1c-f7025b7383c5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-psv5r" (UID: "0e83e20c-2474-4cb0-9a1c-f7025b7383c5") : configmap "kube-rbac-proxy" not found
Sep 11 14:16:12.420891 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:12.420651    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0e83e20c-2474-4cb0-9a1c-f7025b7383c5-crio-socket\") pod \"insights-runtime-extractor-psv5r\" (UID: \"0e83e20c-2474-4cb0-9a1c-f7025b7383c5\") " pod="openshift-insights/insights-runtime-extractor-psv5r"
Sep 11 14:16:12.420891 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:16:12.420720    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 14:16:12.420891 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:16:12.420791    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e83e20c-2474-4cb0-9a1c-f7025b7383c5-insights-runtime-extractor-tls podName:0e83e20c-2474-4cb0-9a1c-f7025b7383c5 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:12.920773609 +0000 UTC m=+76.071323665 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0e83e20c-2474-4cb0-9a1c-f7025b7383c5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-psv5r" (UID: "0e83e20c-2474-4cb0-9a1c-f7025b7383c5") : secret "insights-runtime-extractor-tls" not found
Sep 11 14:16:12.420891 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:12.420822    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0e83e20c-2474-4cb0-9a1c-f7025b7383c5-data-volume\") pod \"insights-runtime-extractor-psv5r\" (UID: \"0e83e20c-2474-4cb0-9a1c-f7025b7383c5\") " pod="openshift-insights/insights-runtime-extractor-psv5r"
Sep 11 14:16:12.431539 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:12.431509    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpd29\" (UniqueName: \"kubernetes.io/projected/0e83e20c-2474-4cb0-9a1c-f7025b7383c5-kube-api-access-cpd29\") pod \"insights-runtime-extractor-psv5r\" (UID: \"0e83e20c-2474-4cb0-9a1c-f7025b7383c5\") " pod="openshift-insights/insights-runtime-extractor-psv5r"
Sep 11 14:16:12.923758 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:12.923720    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e83e20c-2474-4cb0-9a1c-f7025b7383c5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-psv5r\" (UID: \"0e83e20c-2474-4cb0-9a1c-f7025b7383c5\") " pod="openshift-insights/insights-runtime-extractor-psv5r"
Sep 11 14:16:12.923990 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:12.923788    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0e83e20c-2474-4cb0-9a1c-f7025b7383c5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-psv5r\" (UID: \"0e83e20c-2474-4cb0-9a1c-f7025b7383c5\") " pod="openshift-insights/insights-runtime-extractor-psv5r"
Sep 11 14:16:12.923990 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:16:12.923860    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 14:16:12.923990 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:16:12.923881    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 14:16:12.923990 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:16:12.923926    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0e83e20c-2474-4cb0-9a1c-f7025b7383c5-kube-rbac-proxy-cm podName:0e83e20c-2474-4cb0-9a1c-f7025b7383c5 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:13.923911271 +0000 UTC m=+77.074461313 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0e83e20c-2474-4cb0-9a1c-f7025b7383c5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-psv5r" (UID: "0e83e20c-2474-4cb0-9a1c-f7025b7383c5") : configmap "kube-rbac-proxy" not found
Sep 11 14:16:12.923990 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:16:12.923940    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e83e20c-2474-4cb0-9a1c-f7025b7383c5-insights-runtime-extractor-tls podName:0e83e20c-2474-4cb0-9a1c-f7025b7383c5 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:13.923934782 +0000 UTC m=+77.074484824 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0e83e20c-2474-4cb0-9a1c-f7025b7383c5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-psv5r" (UID: "0e83e20c-2474-4cb0-9a1c-f7025b7383c5") : secret "insights-runtime-extractor-tls" not found
Sep 11 14:16:13.931414 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:13.931379    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0e83e20c-2474-4cb0-9a1c-f7025b7383c5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-psv5r\" (UID: \"0e83e20c-2474-4cb0-9a1c-f7025b7383c5\") " pod="openshift-insights/insights-runtime-extractor-psv5r"
Sep 11 14:16:13.931780 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:13.931452    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e83e20c-2474-4cb0-9a1c-f7025b7383c5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-psv5r\" (UID: \"0e83e20c-2474-4cb0-9a1c-f7025b7383c5\") " pod="openshift-insights/insights-runtime-extractor-psv5r"
Sep 11 14:16:13.931780 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:16:13.931505    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 14:16:13.931780 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:16:13.931565    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0e83e20c-2474-4cb0-9a1c-f7025b7383c5-kube-rbac-proxy-cm podName:0e83e20c-2474-4cb0-9a1c-f7025b7383c5 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:15.931551308 +0000 UTC m=+79.082101354 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0e83e20c-2474-4cb0-9a1c-f7025b7383c5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-psv5r" (UID: "0e83e20c-2474-4cb0-9a1c-f7025b7383c5") : configmap "kube-rbac-proxy" not found
Sep 11 14:16:13.931780 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:16:13.931602    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 14:16:13.931780 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:16:13.931666    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e83e20c-2474-4cb0-9a1c-f7025b7383c5-insights-runtime-extractor-tls podName:0e83e20c-2474-4cb0-9a1c-f7025b7383c5 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:15.931652379 +0000 UTC m=+79.082202427 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0e83e20c-2474-4cb0-9a1c-f7025b7383c5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-psv5r" (UID: "0e83e20c-2474-4cb0-9a1c-f7025b7383c5") : secret "insights-runtime-extractor-tls" not found
Sep 11 14:16:14.217224 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:14.217142    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-z7cxh"]
Sep 11 14:16:14.220829 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:14.220791    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-z7cxh"
Sep 11 14:16:14.223741 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:14.223717    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 11 14:16:14.223741 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:14.223717    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 11 14:16:14.223902 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:14.223752    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 11 14:16:14.224226 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:14.224209    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 11 14:16:14.225141 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:14.225122    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-s7bzc\""
Sep 11 14:16:14.232130 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:14.232111    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-z7cxh"]
Sep 11 14:16:14.334023 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:14.333984    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7dd4a673-2bd6-4dd4-a23f-f510ecb8793f-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-z7cxh\" (UID: \"7dd4a673-2bd6-4dd4-a23f-f510ecb8793f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-z7cxh"
Sep 11 14:16:14.334189 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:14.334042    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7dd4a673-2bd6-4dd4-a23f-f510ecb8793f-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-z7cxh\" (UID: \"7dd4a673-2bd6-4dd4-a23f-f510ecb8793f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-z7cxh"
Sep 11 14:16:14.334189 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:14.334107    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh5z5\" (UniqueName: \"kubernetes.io/projected/7dd4a673-2bd6-4dd4-a23f-f510ecb8793f-kube-api-access-dh5z5\") pod \"kube-storage-version-migrator-operator-6c8f84f897-z7cxh\" (UID: \"7dd4a673-2bd6-4dd4-a23f-f510ecb8793f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-z7cxh"
Sep 11 14:16:14.388676 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:14.388646    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wk6kn_757db884-d01a-4ab4-8b46-f9a41a3d7369/dns-node-resolver/0.log"
Sep 11 14:16:14.435158 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:14.435111    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh5z5\" (UniqueName: \"kubernetes.io/projected/7dd4a673-2bd6-4dd4-a23f-f510ecb8793f-kube-api-access-dh5z5\") pod \"kube-storage-version-migrator-operator-6c8f84f897-z7cxh\" (UID: \"7dd4a673-2bd6-4dd4-a23f-f510ecb8793f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-z7cxh"
Sep 11 14:16:14.435276 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:14.435164    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7dd4a673-2bd6-4dd4-a23f-f510ecb8793f-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-z7cxh\" (UID: \"7dd4a673-2bd6-4dd4-a23f-f510ecb8793f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-z7cxh"
Sep 11 14:16:14.435276 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:14.435201    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7dd4a673-2bd6-4dd4-a23f-f510ecb8793f-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-z7cxh\" (UID: \"7dd4a673-2bd6-4dd4-a23f-f510ecb8793f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-z7cxh"
Sep 11 14:16:14.435683 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:14.435662    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7dd4a673-2bd6-4dd4-a23f-f510ecb8793f-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-z7cxh\" (UID: \"7dd4a673-2bd6-4dd4-a23f-f510ecb8793f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-z7cxh"
Sep 11 14:16:14.437367 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:14.437349    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7dd4a673-2bd6-4dd4-a23f-f510ecb8793f-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-z7cxh\" (UID: \"7dd4a673-2bd6-4dd4-a23f-f510ecb8793f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-z7cxh"
Sep 11 14:16:14.445070 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:14.445051    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh5z5\" (UniqueName: \"kubernetes.io/projected/7dd4a673-2bd6-4dd4-a23f-f510ecb8793f-kube-api-access-dh5z5\") pod \"kube-storage-version-migrator-operator-6c8f84f897-z7cxh\" (UID: \"7dd4a673-2bd6-4dd4-a23f-f510ecb8793f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-z7cxh"
Sep 11 14:16:14.529934 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:14.529899    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-z7cxh"
Sep 11 14:16:14.803091 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:14.803018    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-z7cxh"]
Sep 11 14:16:14.806242 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:16:14.806212    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dd4a673_2bd6_4dd4_a23f_f510ecb8793f.slice/crio-4d74cfabcf87f06dc7baa9469454027a0ab41d70bb8d32e7b821004d4f50fea7 WatchSource:0}: Error finding container 4d74cfabcf87f06dc7baa9469454027a0ab41d70bb8d32e7b821004d4f50fea7: Status 404 returned error can't find the container with id 4d74cfabcf87f06dc7baa9469454027a0ab41d70bb8d32e7b821004d4f50fea7
Sep 11 14:16:15.192124 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:15.192052    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qdthk_2b4e57a8-379d-4375-95b4-8176b1e4e410/node-ca/0.log"
Sep 11 14:16:15.593590 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:15.593559    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5s8kg_7aefb1ab-dcdc-451e-b710-81a121895035/serve-healthcheck-canary/0.log"
Sep 11 14:16:15.783269 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:15.783229    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-z7cxh" event={"ID":"7dd4a673-2bd6-4dd4-a23f-f510ecb8793f","Type":"ContainerStarted","Data":"4d74cfabcf87f06dc7baa9469454027a0ab41d70bb8d32e7b821004d4f50fea7"}
Sep 11 14:16:15.947258 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:15.947164    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0e83e20c-2474-4cb0-9a1c-f7025b7383c5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-psv5r\" (UID: \"0e83e20c-2474-4cb0-9a1c-f7025b7383c5\") " pod="openshift-insights/insights-runtime-extractor-psv5r"
Sep 11 14:16:15.947258 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:15.947240    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e83e20c-2474-4cb0-9a1c-f7025b7383c5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-psv5r\" (UID: \"0e83e20c-2474-4cb0-9a1c-f7025b7383c5\") " pod="openshift-insights/insights-runtime-extractor-psv5r"
Sep 11 14:16:15.947471 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:16:15.947296    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 14:16:15.947471 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:16:15.947367    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0e83e20c-2474-4cb0-9a1c-f7025b7383c5-kube-rbac-proxy-cm podName:0e83e20c-2474-4cb0-9a1c-f7025b7383c5 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:19.947348363 +0000 UTC m=+83.097898421 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0e83e20c-2474-4cb0-9a1c-f7025b7383c5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-psv5r" (UID: "0e83e20c-2474-4cb0-9a1c-f7025b7383c5") : configmap "kube-rbac-proxy" not found
Sep 11 14:16:15.947471 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:16:15.947374    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 14:16:15.947471 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:16:15.947409    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e83e20c-2474-4cb0-9a1c-f7025b7383c5-insights-runtime-extractor-tls podName:0e83e20c-2474-4cb0-9a1c-f7025b7383c5 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:19.947399496 +0000 UTC m=+83.097949539 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0e83e20c-2474-4cb0-9a1c-f7025b7383c5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-psv5r" (UID: "0e83e20c-2474-4cb0-9a1c-f7025b7383c5") : secret "insights-runtime-extractor-tls" not found
Sep 11 14:16:16.786978 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:16.786933    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-z7cxh" event={"ID":"7dd4a673-2bd6-4dd4-a23f-f510ecb8793f","Type":"ContainerStarted","Data":"ecfed8196841af46930fffba2d563e2f026691f0c639050822cd1462f7aacc0a"}
Sep 11 14:16:19.979172 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:19.979032    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e83e20c-2474-4cb0-9a1c-f7025b7383c5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-psv5r\" (UID: \"0e83e20c-2474-4cb0-9a1c-f7025b7383c5\") " pod="openshift-insights/insights-runtime-extractor-psv5r"
Sep 11 14:16:19.979172 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:19.979114    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0e83e20c-2474-4cb0-9a1c-f7025b7383c5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-psv5r\" (UID: \"0e83e20c-2474-4cb0-9a1c-f7025b7383c5\") " pod="openshift-insights/insights-runtime-extractor-psv5r"
Sep 11 14:16:19.979590 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:16:19.979187    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 14:16:19.979590 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:16:19.979195    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 14:16:19.979590 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:16:19.979240    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0e83e20c-2474-4cb0-9a1c-f7025b7383c5-kube-rbac-proxy-cm podName:0e83e20c-2474-4cb0-9a1c-f7025b7383c5 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:27.979227065 +0000 UTC m=+91.129777108 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0e83e20c-2474-4cb0-9a1c-f7025b7383c5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-psv5r" (UID: "0e83e20c-2474-4cb0-9a1c-f7025b7383c5") : configmap "kube-rbac-proxy" not found
Sep 11 14:16:19.979590 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:16:19.979254    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e83e20c-2474-4cb0-9a1c-f7025b7383c5-insights-runtime-extractor-tls podName:0e83e20c-2474-4cb0-9a1c-f7025b7383c5 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:27.979247645 +0000 UTC m=+91.129797686 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0e83e20c-2474-4cb0-9a1c-f7025b7383c5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-psv5r" (UID: "0e83e20c-2474-4cb0-9a1c-f7025b7383c5") : secret "insights-runtime-extractor-tls" not found
Sep 11 14:16:23.688723 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:23.688664    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-z7cxh" podStartSLOduration=7.9308175720000005 podStartE2EDuration="9.688643845s" totalImagesPullingTime="1.757826273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:16:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:16.812130172 +0000 UTC m=+79.962680237" watchObservedRunningTime="2026-09-11 14:16:23.688643845 +0000 UTC m=+86.839193917"
Sep 11 14:16:23.689381 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:23.689362    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-x884z"]
Sep 11 14:16:23.693505 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:23.693490    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-x884z"
Sep 11 14:16:23.696423 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:23.696402    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 11 14:16:23.696423 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:23.696409    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-mw462\""
Sep 11 14:16:23.696423 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:23.696419    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 11 14:16:23.697551 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:23.697535    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 11 14:16:23.697655 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:23.697594    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 11 14:16:23.712774 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:23.712754    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-x884z"]
Sep 11 14:16:23.806276 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:23.806239    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9b5ead46-9b7f-4ceb-ab1d-bc8f6872bbed-signing-key\") pod \"service-ca-568869fdc5-x884z\" (UID: \"9b5ead46-9b7f-4ceb-ab1d-bc8f6872bbed\") " pod="openshift-service-ca/service-ca-568869fdc5-x884z"
Sep 11 14:16:23.806276 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:23.806276    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdsnc\" (UniqueName: \"kubernetes.io/projected/9b5ead46-9b7f-4ceb-ab1d-bc8f6872bbed-kube-api-access-sdsnc\") pod \"service-ca-568869fdc5-x884z\" (UID: \"9b5ead46-9b7f-4ceb-ab1d-bc8f6872bbed\") " pod="openshift-service-ca/service-ca-568869fdc5-x884z"
Sep 11 14:16:23.806490 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:23.806351    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9b5ead46-9b7f-4ceb-ab1d-bc8f6872bbed-signing-cabundle\") pod \"service-ca-568869fdc5-x884z\" (UID: \"9b5ead46-9b7f-4ceb-ab1d-bc8f6872bbed\") " pod="openshift-service-ca/service-ca-568869fdc5-x884z"
Sep 11 14:16:23.907035 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:23.906994    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9b5ead46-9b7f-4ceb-ab1d-bc8f6872bbed-signing-key\") pod \"service-ca-568869fdc5-x884z\" (UID: \"9b5ead46-9b7f-4ceb-ab1d-bc8f6872bbed\") " pod="openshift-service-ca/service-ca-568869fdc5-x884z"
Sep 11 14:16:23.907035 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:23.907035    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdsnc\" (UniqueName: \"kubernetes.io/projected/9b5ead46-9b7f-4ceb-ab1d-bc8f6872bbed-kube-api-access-sdsnc\") pod \"service-ca-568869fdc5-x884z\" (UID: \"9b5ead46-9b7f-4ceb-ab1d-bc8f6872bbed\") " pod="openshift-service-ca/service-ca-568869fdc5-x884z"
Sep 11 14:16:23.907231 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:23.907085    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9b5ead46-9b7f-4ceb-ab1d-bc8f6872bbed-signing-cabundle\") pod \"service-ca-568869fdc5-x884z\" (UID: \"9b5ead46-9b7f-4ceb-ab1d-bc8f6872bbed\") " pod="openshift-service-ca/service-ca-568869fdc5-x884z"
Sep 11 14:16:23.907839 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:23.907819    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9b5ead46-9b7f-4ceb-ab1d-bc8f6872bbed-signing-cabundle\") pod \"service-ca-568869fdc5-x884z\" (UID: \"9b5ead46-9b7f-4ceb-ab1d-bc8f6872bbed\") " pod="openshift-service-ca/service-ca-568869fdc5-x884z"
Sep 11 14:16:23.909404 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:23.909381    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9b5ead46-9b7f-4ceb-ab1d-bc8f6872bbed-signing-key\") pod \"service-ca-568869fdc5-x884z\" (UID: \"9b5ead46-9b7f-4ceb-ab1d-bc8f6872bbed\") " pod="openshift-service-ca/service-ca-568869fdc5-x884z"
Sep 11 14:16:23.918552 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:23.918523    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdsnc\" (UniqueName: \"kubernetes.io/projected/9b5ead46-9b7f-4ceb-ab1d-bc8f6872bbed-kube-api-access-sdsnc\") pod \"service-ca-568869fdc5-x884z\" (UID: \"9b5ead46-9b7f-4ceb-ab1d-bc8f6872bbed\") " pod="openshift-service-ca/service-ca-568869fdc5-x884z"
Sep 11 14:16:24.001855 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:24.001819    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-x884z"
Sep 11 14:16:24.284466 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:24.284387    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-x884z"]
Sep 11 14:16:24.287198 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:16:24.287169    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b5ead46_9b7f_4ceb_ab1d_bc8f6872bbed.slice/crio-a709c13dc1ed614709a0b5732c54f8c2d35cbb92a268fcc0b8018a1248be0457 WatchSource:0}: Error finding container a709c13dc1ed614709a0b5732c54f8c2d35cbb92a268fcc0b8018a1248be0457: Status 404 returned error can't find the container with id a709c13dc1ed614709a0b5732c54f8c2d35cbb92a268fcc0b8018a1248be0457
Sep 11 14:16:24.808346 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:24.808304    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-x884z" event={"ID":"9b5ead46-9b7f-4ceb-ab1d-bc8f6872bbed","Type":"ContainerStarted","Data":"a709c13dc1ed614709a0b5732c54f8c2d35cbb92a268fcc0b8018a1248be0457"}
Sep 11 14:16:26.815632 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:26.815596    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-x884z" event={"ID":"9b5ead46-9b7f-4ceb-ab1d-bc8f6872bbed","Type":"ContainerStarted","Data":"8f30d76e8c0aab9a453cc5c8654c29955b3cfbd43275f4750f520a08e59a0c70"}
Sep 11 14:16:26.843136 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:26.843085    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-x884z" podStartSLOduration=1.95383956 podStartE2EDuration="3.843070276s" totalImagesPullingTime="1.889230716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:16:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:26.841883043 +0000 UTC m=+89.992433108" watchObservedRunningTime="2026-09-11 14:16:26.843070276 +0000 UTC m=+89.993620343"
Sep 11 14:16:28.045651 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:28.045617    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e83e20c-2474-4cb0-9a1c-f7025b7383c5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-psv5r\" (UID: \"0e83e20c-2474-4cb0-9a1c-f7025b7383c5\") " pod="openshift-insights/insights-runtime-extractor-psv5r"
Sep 11 14:16:28.046029 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:28.045685    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0e83e20c-2474-4cb0-9a1c-f7025b7383c5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-psv5r\" (UID: \"0e83e20c-2474-4cb0-9a1c-f7025b7383c5\") " pod="openshift-insights/insights-runtime-extractor-psv5r"
Sep 11 14:16:28.046029 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:16:28.045798    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 14:16:28.046029 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:16:28.045887    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e83e20c-2474-4cb0-9a1c-f7025b7383c5-insights-runtime-extractor-tls podName:0e83e20c-2474-4cb0-9a1c-f7025b7383c5 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:44.045862377 +0000 UTC m=+107.196412434 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0e83e20c-2474-4cb0-9a1c-f7025b7383c5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-psv5r" (UID: "0e83e20c-2474-4cb0-9a1c-f7025b7383c5") : secret "insights-runtime-extractor-tls" not found
Sep 11 14:16:28.046158 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:28.046142    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0e83e20c-2474-4cb0-9a1c-f7025b7383c5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-psv5r\" (UID: \"0e83e20c-2474-4cb0-9a1c-f7025b7383c5\") " pod="openshift-insights/insights-runtime-extractor-psv5r"
Sep 11 14:16:36.109002 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:36.108944    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98689afa-2179-4240-9ad9-cbe7744a84c1-metrics-tls\") pod \"dns-default-gdl2n\" (UID: \"98689afa-2179-4240-9ad9-cbe7744a84c1\") " pod="openshift-dns/dns-default-gdl2n"
Sep 11 14:16:36.109424 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:36.109032    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5115e3f0-399e-4db4-a892-ba00f8d8735f-registry-tls\") pod \"image-registry-fbb68bfc-66pmz\" (UID: \"5115e3f0-399e-4db4-a892-ba00f8d8735f\") " pod="openshift-image-registry/image-registry-fbb68bfc-66pmz"
Sep 11 14:16:36.111507 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:36.111478    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98689afa-2179-4240-9ad9-cbe7744a84c1-metrics-tls\") pod \"dns-default-gdl2n\" (UID: \"98689afa-2179-4240-9ad9-cbe7744a84c1\") " pod="openshift-dns/dns-default-gdl2n"
Sep 11 14:16:36.111579 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:36.111559    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5115e3f0-399e-4db4-a892-ba00f8d8735f-registry-tls\") pod \"image-registry-fbb68bfc-66pmz\" (UID: \"5115e3f0-399e-4db4-a892-ba00f8d8735f\") " pod="openshift-image-registry/image-registry-fbb68bfc-66pmz"
Sep 11 14:16:36.140333 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:36.140300    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-92zwk\""
Sep 11 14:16:36.147804 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:36.147787    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fbb68bfc-66pmz"
Sep 11 14:16:36.152781 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:36.152762    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-c4f62\""
Sep 11 14:16:36.160186 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:36.160158    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gdl2n"
Sep 11 14:16:36.452050 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:36.451921    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-fbb68bfc-66pmz"]
Sep 11 14:16:36.457573 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:16:36.457546    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5115e3f0_399e_4db4_a892_ba00f8d8735f.slice/crio-be6773d92bf3e01ed724d89e27cb5288717c07a3a7d8e22c75c2718233372d7b WatchSource:0}: Error finding container be6773d92bf3e01ed724d89e27cb5288717c07a3a7d8e22c75c2718233372d7b: Status 404 returned error can't find the container with id be6773d92bf3e01ed724d89e27cb5288717c07a3a7d8e22c75c2718233372d7b
Sep 11 14:16:36.460139 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:36.460113    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gdl2n"]
Sep 11 14:16:36.462875 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:16:36.462852    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98689afa_2179_4240_9ad9_cbe7744a84c1.slice/crio-8795bc534f5029bcae4a3a2b7200896d6ee3ef5564a62e69ae66531ed75dd6d1 WatchSource:0}: Error finding container 8795bc534f5029bcae4a3a2b7200896d6ee3ef5564a62e69ae66531ed75dd6d1: Status 404 returned error can't find the container with id 8795bc534f5029bcae4a3a2b7200896d6ee3ef5564a62e69ae66531ed75dd6d1
Sep 11 14:16:36.844317 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:36.844277    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gdl2n" event={"ID":"98689afa-2179-4240-9ad9-cbe7744a84c1","Type":"ContainerStarted","Data":"8795bc534f5029bcae4a3a2b7200896d6ee3ef5564a62e69ae66531ed75dd6d1"}
Sep 11 14:16:36.846114 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:36.846083    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fbb68bfc-66pmz" event={"ID":"5115e3f0-399e-4db4-a892-ba00f8d8735f","Type":"ContainerStarted","Data":"919cd3bca193dcbcf3fe56cfe3f07d5e8beca372bad789a238a8350abd2a97cf"}
Sep 11 14:16:36.846248 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:36.846130    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fbb68bfc-66pmz" event={"ID":"5115e3f0-399e-4db4-a892-ba00f8d8735f","Type":"ContainerStarted","Data":"be6773d92bf3e01ed724d89e27cb5288717c07a3a7d8e22c75c2718233372d7b"}
Sep 11 14:16:36.846248 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:36.846228    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-fbb68bfc-66pmz"
Sep 11 14:16:36.870448 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:36.870394    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-fbb68bfc-66pmz" podStartSLOduration=98.870377865 podStartE2EDuration="1m38.870377865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:14:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:36.869545142 +0000 UTC m=+100.020095230" watchObservedRunningTime="2026-09-11 14:16:36.870377865 +0000 UTC m=+100.020927946"
Sep 11 14:16:38.855646 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:38.855606    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gdl2n" event={"ID":"98689afa-2179-4240-9ad9-cbe7744a84c1","Type":"ContainerStarted","Data":"001af247abbbcf05c5f8300227359429b05b2e4aa942e5c79316f647b71c13f5"}
Sep 11 14:16:38.856063 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:38.855703    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gdl2n" event={"ID":"98689afa-2179-4240-9ad9-cbe7744a84c1","Type":"ContainerStarted","Data":"1d11f71b0ca6f973f0f0c8c1d1fb037673664c35f005f7e916fa02807b069649"}
Sep 11 14:16:38.856063 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:38.855770    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-gdl2n"
Sep 11 14:16:38.901808 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:38.901750    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-gdl2n" podStartSLOduration=65.420990735 podStartE2EDuration="1m6.901735225s" totalImagesPullingTime="1.48074449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:15:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:38.89370389 +0000 UTC m=+102.044253955" watchObservedRunningTime="2026-09-11 14:16:38.901735225 +0000 UTC m=+102.052285286"
Sep 11 14:16:39.769514 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:39.769485    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-q5sxd"
Sep 11 14:16:44.070164 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:44.070125    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e83e20c-2474-4cb0-9a1c-f7025b7383c5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-psv5r\" (UID: \"0e83e20c-2474-4cb0-9a1c-f7025b7383c5\") " pod="openshift-insights/insights-runtime-extractor-psv5r"
Sep 11 14:16:44.072481 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:44.072460    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e83e20c-2474-4cb0-9a1c-f7025b7383c5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-psv5r\" (UID: \"0e83e20c-2474-4cb0-9a1c-f7025b7383c5\") " pod="openshift-insights/insights-runtime-extractor-psv5r"
Sep 11 14:16:44.344325 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:44.344221    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-psv5r"
Sep 11 14:16:44.621505 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:44.621346    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-psv5r"]
Sep 11 14:16:44.624036 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:16:44.624007    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e83e20c_2474_4cb0_9a1c_f7025b7383c5.slice/crio-635909c9fe56e1e3d673e09eb767d400faaa3c8aa38020710657a87a27d1a652 WatchSource:0}: Error finding container 635909c9fe56e1e3d673e09eb767d400faaa3c8aa38020710657a87a27d1a652: Status 404 returned error can't find the container with id 635909c9fe56e1e3d673e09eb767d400faaa3c8aa38020710657a87a27d1a652
Sep 11 14:16:44.879207 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:44.879107    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-psv5r" event={"ID":"0e83e20c-2474-4cb0-9a1c-f7025b7383c5","Type":"ContainerStarted","Data":"80d5c7e2b21a5eadb35a05db13ece228873f37691492139c303c360f8a6ad849"}
Sep 11 14:16:44.879207 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:44.879180    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-psv5r" event={"ID":"0e83e20c-2474-4cb0-9a1c-f7025b7383c5","Type":"ContainerStarted","Data":"635909c9fe56e1e3d673e09eb767d400faaa3c8aa38020710657a87a27d1a652"}
Sep 11 14:16:46.890196 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:46.890161    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-psv5r" event={"ID":"0e83e20c-2474-4cb0-9a1c-f7025b7383c5","Type":"ContainerStarted","Data":"dd94cb4304b77734e671e6340383e26c62926a515ea67b3956e5f6c923659e8d"}
Sep 11 14:16:47.316867 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:47.316833    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cbf875664-mb5f7"]
Sep 11 14:16:47.318960 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:47.318937    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cbf875664-mb5f7"
Sep 11 14:16:47.324039 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:47.324014    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 11 14:16:47.324163 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:47.324062    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 11 14:16:47.325628 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:47.325597    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 11 14:16:47.325628 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:47.325619    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 11 14:16:47.325769 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:47.325662    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 11 14:16:47.325769 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:47.325619    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 11 14:16:47.327177 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:47.327149    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 11 14:16:47.362549 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:47.362480    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cbf875664-mb5f7"]
Sep 11 14:16:47.396600 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:47.396558    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/6336ae86-4493-4237-b544-c86ec3b6f980-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6cbf875664-mb5f7\" (UID: \"6336ae86-4493-4237-b544-c86ec3b6f980\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cbf875664-mb5f7"
Sep 11 14:16:47.396788 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:47.396665    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/6336ae86-4493-4237-b544-c86ec3b6f980-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6cbf875664-mb5f7\" (UID: \"6336ae86-4493-4237-b544-c86ec3b6f980\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cbf875664-mb5f7"
Sep 11 14:16:47.396788 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:47.396717    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/6336ae86-4493-4237-b544-c86ec3b6f980-hub\") pod \"cluster-proxy-proxy-agent-6cbf875664-mb5f7\" (UID: \"6336ae86-4493-4237-b544-c86ec3b6f980\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cbf875664-mb5f7"
Sep 11 14:16:47.396788 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:47.396765    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6336ae86-4493-4237-b544-c86ec3b6f980-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6cbf875664-mb5f7\" (UID: \"6336ae86-4493-4237-b544-c86ec3b6f980\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cbf875664-mb5f7"
Sep 11 14:16:47.397019 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:47.396791    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/6336ae86-4493-4237-b544-c86ec3b6f980-ca\") pod \"cluster-proxy-proxy-agent-6cbf875664-mb5f7\" (UID: \"6336ae86-4493-4237-b544-c86ec3b6f980\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cbf875664-mb5f7"
Sep 11 14:16:47.397019 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:47.396813    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6q5z\" (UniqueName: \"kubernetes.io/projected/6336ae86-4493-4237-b544-c86ec3b6f980-kube-api-access-h6q5z\") pod \"cluster-proxy-proxy-agent-6cbf875664-mb5f7\" (UID: \"6336ae86-4493-4237-b544-c86ec3b6f980\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cbf875664-mb5f7"
Sep 11 14:16:47.497475 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:47.497440    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/6336ae86-4493-4237-b544-c86ec3b6f980-hub\") pod \"cluster-proxy-proxy-agent-6cbf875664-mb5f7\" (UID: \"6336ae86-4493-4237-b544-c86ec3b6f980\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cbf875664-mb5f7"
Sep 11 14:16:47.497475 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:47.497475    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6336ae86-4493-4237-b544-c86ec3b6f980-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6cbf875664-mb5f7\" (UID: \"6336ae86-4493-4237-b544-c86ec3b6f980\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cbf875664-mb5f7"
Sep 11 14:16:47.497738 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:47.497494    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/6336ae86-4493-4237-b544-c86ec3b6f980-ca\") pod \"cluster-proxy-proxy-agent-6cbf875664-mb5f7\" (UID: \"6336ae86-4493-4237-b544-c86ec3b6f980\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cbf875664-mb5f7"
Sep 11 14:16:47.497738 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:47.497566    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6q5z\" (UniqueName: \"kubernetes.io/projected/6336ae86-4493-4237-b544-c86ec3b6f980-kube-api-access-h6q5z\") pod \"cluster-proxy-proxy-agent-6cbf875664-mb5f7\" (UID: \"6336ae86-4493-4237-b544-c86ec3b6f980\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cbf875664-mb5f7"
Sep 11 14:16:47.497738 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:47.497655    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/6336ae86-4493-4237-b544-c86ec3b6f980-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6cbf875664-mb5f7\" (UID: \"6336ae86-4493-4237-b544-c86ec3b6f980\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cbf875664-mb5f7"
Sep 11 14:16:47.497738 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:47.497712    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/6336ae86-4493-4237-b544-c86ec3b6f980-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6cbf875664-mb5f7\" (UID: \"6336ae86-4493-4237-b544-c86ec3b6f980\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cbf875664-mb5f7"
Sep 11 14:16:47.498519 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:47.498466    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/6336ae86-4493-4237-b544-c86ec3b6f980-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6cbf875664-mb5f7\" (UID: \"6336ae86-4493-4237-b544-c86ec3b6f980\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cbf875664-mb5f7"
Sep 11 14:16:47.500687 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:47.500643    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/6336ae86-4493-4237-b544-c86ec3b6f980-ca\") pod \"cluster-proxy-proxy-agent-6cbf875664-mb5f7\" (UID: \"6336ae86-4493-4237-b544-c86ec3b6f980\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cbf875664-mb5f7"
Sep 11 14:16:47.500933 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:47.500883    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/6336ae86-4493-4237-b544-c86ec3b6f980-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6cbf875664-mb5f7\" (UID: \"6336ae86-4493-4237-b544-c86ec3b6f980\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cbf875664-mb5f7"
Sep 11 14:16:47.500933 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:47.500914    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6336ae86-4493-4237-b544-c86ec3b6f980-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6cbf875664-mb5f7\" (UID: \"6336ae86-4493-4237-b544-c86ec3b6f980\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cbf875664-mb5f7"
Sep 11 14:16:47.501111 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:47.500937    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/6336ae86-4493-4237-b544-c86ec3b6f980-hub\") pod \"cluster-proxy-proxy-agent-6cbf875664-mb5f7\" (UID: \"6336ae86-4493-4237-b544-c86ec3b6f980\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cbf875664-mb5f7"
Sep 11 14:16:47.522752 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:47.522718    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6q5z\" (UniqueName: \"kubernetes.io/projected/6336ae86-4493-4237-b544-c86ec3b6f980-kube-api-access-h6q5z\") pod \"cluster-proxy-proxy-agent-6cbf875664-mb5f7\" (UID: \"6336ae86-4493-4237-b544-c86ec3b6f980\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cbf875664-mb5f7"
Sep 11 14:16:47.636440 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:47.636345    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cbf875664-mb5f7"
Sep 11 14:16:47.895531 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:47.895436    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-psv5r" event={"ID":"0e83e20c-2474-4cb0-9a1c-f7025b7383c5","Type":"ContainerStarted","Data":"0f8785040956df4b6322e2d3232ccc2d7eaf93ac754fae0656e0d6c13356752e"}
Sep 11 14:16:47.934508 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:47.934455    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-psv5r" podStartSLOduration=32.965794711 podStartE2EDuration="35.934440931s" totalImagesPullingTime="2.96864622s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:16:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:47.934183261 +0000 UTC m=+111.084733325" watchObservedRunningTime="2026-09-11 14:16:47.934440931 +0000 UTC m=+111.084990994"
Sep 11 14:16:47.956604 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:47.956579    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cbf875664-mb5f7"]
Sep 11 14:16:47.958739 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:16:47.958714    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6336ae86_4493_4237_b544_c86ec3b6f980.slice/crio-cfeb0ff2b49b3f2e2fce7bcf666a40fa1eea80e349b14302c34b864c454f8dda WatchSource:0}: Error finding container cfeb0ff2b49b3f2e2fce7bcf666a40fa1eea80e349b14302c34b864c454f8dda: Status 404 returned error can't find the container with id cfeb0ff2b49b3f2e2fce7bcf666a40fa1eea80e349b14302c34b864c454f8dda
Sep 11 14:16:48.863407 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:48.863372    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gdl2n"
Sep 11 14:16:48.901917 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:48.901876    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cbf875664-mb5f7" event={"ID":"6336ae86-4493-4237-b544-c86ec3b6f980","Type":"ContainerStarted","Data":"cfeb0ff2b49b3f2e2fce7bcf666a40fa1eea80e349b14302c34b864c454f8dda"}
Sep 11 14:16:51.917554 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:51.917522    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cbf875664-mb5f7" event={"ID":"6336ae86-4493-4237-b544-c86ec3b6f980","Type":"ContainerStarted","Data":"77b0da0a80a22cf7e62b00b69ab42b581771dabe70761d36dc9a5077c8ea7372"}
Sep 11 14:16:51.917554 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:51.917562    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cbf875664-mb5f7" event={"ID":"6336ae86-4493-4237-b544-c86ec3b6f980","Type":"ContainerStarted","Data":"d447bf510ee7e3cf3097a2a841efbbafea551f9f2b597a14ccacc9200fc51f99"}
Sep 11 14:16:51.918025 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:51.917583    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cbf875664-mb5f7" event={"ID":"6336ae86-4493-4237-b544-c86ec3b6f980","Type":"ContainerStarted","Data":"ee926c9c781a07a05245d698f830ef40e49dd41ba5b47c6fe9c6f088e5a7198a"}
Sep 11 14:16:51.944893 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:51.944845    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cbf875664-mb5f7" podStartSLOduration=1.7378493769999999 podStartE2EDuration="4.944829974s" totalImagesPullingTime="3.206980597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:51.943952703 +0000 UTC m=+115.094502769" watchObservedRunningTime="2026-09-11 14:16:51.944829974 +0000 UTC m=+115.095380038"
Sep 11 14:16:56.023482 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:56.023445    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zwkxm"]
Sep 11 14:16:56.027144 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:56.027124    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zwkxm"
Sep 11 14:16:56.030179 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:56.030159    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 11 14:16:56.030313 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:56.030205    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 11 14:16:56.031273 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:56.031247    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 11 14:16:56.031372 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:56.031302    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qj5r6\""
Sep 11 14:16:56.031510 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:56.031492    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 11 14:16:56.031582 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:56.031500    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 11 14:16:56.031770 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:56.031748    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 11 14:16:56.152341 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:56.152307    2581 patch_prober.go:28] interesting pod/image-registry-fbb68bfc-66pmz container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 11 14:16:56.152502 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:56.152360    2581 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-fbb68bfc-66pmz" podUID="5115e3f0-399e-4db4-a892-ba00f8d8735f" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 14:16:56.162696 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:56.162674    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e8ad0979-ef70-493c-8e4f-8a96a151dd79-root\") pod \"node-exporter-zwkxm\" (UID: \"e8ad0979-ef70-493c-8e4f-8a96a151dd79\") " pod="openshift-monitoring/node-exporter-zwkxm"
Sep 11 14:16:56.162771 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:56.162706    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p654x\" (UniqueName: \"kubernetes.io/projected/e8ad0979-ef70-493c-8e4f-8a96a151dd79-kube-api-access-p654x\") pod \"node-exporter-zwkxm\" (UID: \"e8ad0979-ef70-493c-8e4f-8a96a151dd79\") " pod="openshift-monitoring/node-exporter-zwkxm"
Sep 11 14:16:56.162771 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:56.162723    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e8ad0979-ef70-493c-8e4f-8a96a151dd79-node-exporter-textfile\") pod \"node-exporter-zwkxm\" (UID: \"e8ad0979-ef70-493c-8e4f-8a96a151dd79\") " pod="openshift-monitoring/node-exporter-zwkxm"
Sep 11 14:16:56.162771 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:56.162740    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8ad0979-ef70-493c-8e4f-8a96a151dd79-metrics-client-ca\") pod \"node-exporter-zwkxm\" (UID: \"e8ad0979-ef70-493c-8e4f-8a96a151dd79\") " pod="openshift-monitoring/node-exporter-zwkxm"
Sep 11 14:16:56.162771 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:56.162761    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e8ad0979-ef70-493c-8e4f-8a96a151dd79-node-exporter-wtmp\") pod \"node-exporter-zwkxm\" (UID: \"e8ad0979-ef70-493c-8e4f-8a96a151dd79\") " pod="openshift-monitoring/node-exporter-zwkxm"
Sep 11 14:16:56.162908 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:56.162811    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e8ad0979-ef70-493c-8e4f-8a96a151dd79-node-exporter-tls\") pod \"node-exporter-zwkxm\" (UID: \"e8ad0979-ef70-493c-8e4f-8a96a151dd79\") " pod="openshift-monitoring/node-exporter-zwkxm"
Sep 11 14:16:56.162908 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:56.162842    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e8ad0979-ef70-493c-8e4f-8a96a151dd79-node-exporter-accelerators-collector-config\") pod \"node-exporter-zwkxm\" (UID: \"e8ad0979-ef70-493c-8e4f-8a96a151dd79\") " pod="openshift-monitoring/node-exporter-zwkxm"
Sep 11 14:16:56.162908 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:56.162873    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e8ad0979-ef70-493c-8e4f-8a96a151dd79-sys\") pod \"node-exporter-zwkxm\" (UID: \"e8ad0979-ef70-493c-8e4f-8a96a151dd79\") " pod="openshift-monitoring/node-exporter-zwkxm"
Sep 11 14:16:56.162908 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:56.162891    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e8ad0979-ef70-493c-8e4f-8a96a151dd79-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zwkxm\" (UID: \"e8ad0979-ef70-493c-8e4f-8a96a151dd79\") " pod="openshift-monitoring/node-exporter-zwkxm"
Sep 11 14:16:56.263588 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:56.263546    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e8ad0979-ef70-493c-8e4f-8a96a151dd79-node-exporter-tls\") pod \"node-exporter-zwkxm\" (UID: \"e8ad0979-ef70-493c-8e4f-8a96a151dd79\") " pod="openshift-monitoring/node-exporter-zwkxm"
Sep 11 14:16:56.263783 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:56.263592    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e8ad0979-ef70-493c-8e4f-8a96a151dd79-node-exporter-accelerators-collector-config\") pod \"node-exporter-zwkxm\" (UID: \"e8ad0979-ef70-493c-8e4f-8a96a151dd79\") " pod="openshift-monitoring/node-exporter-zwkxm"
Sep 11 14:16:56.263783 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:56.263630    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e8ad0979-ef70-493c-8e4f-8a96a151dd79-sys\") pod \"node-exporter-zwkxm\" (UID: \"e8ad0979-ef70-493c-8e4f-8a96a151dd79\") " pod="openshift-monitoring/node-exporter-zwkxm"
Sep 11 14:16:56.263783 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:56.263654    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e8ad0979-ef70-493c-8e4f-8a96a151dd79-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zwkxm\" (UID: \"e8ad0979-ef70-493c-8e4f-8a96a151dd79\") " pod="openshift-monitoring/node-exporter-zwkxm"
Sep 11 14:16:56.263783 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:56.263704    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e8ad0979-ef70-493c-8e4f-8a96a151dd79-root\") pod \"node-exporter-zwkxm\" (UID: \"e8ad0979-ef70-493c-8e4f-8a96a151dd79\") " pod="openshift-monitoring/node-exporter-zwkxm"
Sep 11 14:16:56.263783 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:56.263733    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p654x\" (UniqueName: \"kubernetes.io/projected/e8ad0979-ef70-493c-8e4f-8a96a151dd79-kube-api-access-p654x\") pod \"node-exporter-zwkxm\" (UID: \"e8ad0979-ef70-493c-8e4f-8a96a151dd79\") " pod="openshift-monitoring/node-exporter-zwkxm"
Sep 11 14:16:56.263783 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:56.263758    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e8ad0979-ef70-493c-8e4f-8a96a151dd79-node-exporter-textfile\") pod \"node-exporter-zwkxm\" (UID: \"e8ad0979-ef70-493c-8e4f-8a96a151dd79\") " pod="openshift-monitoring/node-exporter-zwkxm"
Sep 11 14:16:56.264120 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:56.263787    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8ad0979-ef70-493c-8e4f-8a96a151dd79-metrics-client-ca\") pod \"node-exporter-zwkxm\" (UID: \"e8ad0979-ef70-493c-8e4f-8a96a151dd79\") " pod="openshift-monitoring/node-exporter-zwkxm"
Sep 11 14:16:56.264120 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:56.263754    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e8ad0979-ef70-493c-8e4f-8a96a151dd79-sys\") pod \"node-exporter-zwkxm\" (UID: \"e8ad0979-ef70-493c-8e4f-8a96a151dd79\") " pod="openshift-monitoring/node-exporter-zwkxm"
Sep 11 14:16:56.264120 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:56.263818    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e8ad0979-ef70-493c-8e4f-8a96a151dd79-node-exporter-wtmp\") pod \"node-exporter-zwkxm\" (UID: \"e8ad0979-ef70-493c-8e4f-8a96a151dd79\") " pod="openshift-monitoring/node-exporter-zwkxm"
Sep 11 14:16:56.264120 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:56.263780    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e8ad0979-ef70-493c-8e4f-8a96a151dd79-root\") pod \"node-exporter-zwkxm\" (UID: \"e8ad0979-ef70-493c-8e4f-8a96a151dd79\") " pod="openshift-monitoring/node-exporter-zwkxm"
Sep 11 14:16:56.264120 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:56.264000    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e8ad0979-ef70-493c-8e4f-8a96a151dd79-node-exporter-wtmp\") pod \"node-exporter-zwkxm\" (UID: \"e8ad0979-ef70-493c-8e4f-8a96a151dd79\") " pod="openshift-monitoring/node-exporter-zwkxm"
Sep 11 14:16:56.264344 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:56.264257    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e8ad0979-ef70-493c-8e4f-8a96a151dd79-node-exporter-textfile\") pod \"node-exporter-zwkxm\" (UID: \"e8ad0979-ef70-493c-8e4f-8a96a151dd79\") " pod="openshift-monitoring/node-exporter-zwkxm"
Sep 11 14:16:56.264411 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:56.264356    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e8ad0979-ef70-493c-8e4f-8a96a151dd79-node-exporter-accelerators-collector-config\") pod \"node-exporter-zwkxm\" (UID: \"e8ad0979-ef70-493c-8e4f-8a96a151dd79\") " pod="openshift-monitoring/node-exporter-zwkxm"
Sep 11 14:16:56.264465 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:56.264409    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8ad0979-ef70-493c-8e4f-8a96a151dd79-metrics-client-ca\") pod \"node-exporter-zwkxm\" (UID: \"e8ad0979-ef70-493c-8e4f-8a96a151dd79\") " pod="openshift-monitoring/node-exporter-zwkxm"
Sep 11 14:16:56.266053 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:56.266031    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e8ad0979-ef70-493c-8e4f-8a96a151dd79-node-exporter-tls\") pod \"node-exporter-zwkxm\" (UID: \"e8ad0979-ef70-493c-8e4f-8a96a151dd79\") " pod="openshift-monitoring/node-exporter-zwkxm"
Sep 11 14:16:56.266162 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:56.266080    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e8ad0979-ef70-493c-8e4f-8a96a151dd79-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zwkxm\" (UID: \"e8ad0979-ef70-493c-8e4f-8a96a151dd79\") " pod="openshift-monitoring/node-exporter-zwkxm"
Sep 11 14:16:56.278303 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:56.278247    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p654x\" (UniqueName: \"kubernetes.io/projected/e8ad0979-ef70-493c-8e4f-8a96a151dd79-kube-api-access-p654x\") pod \"node-exporter-zwkxm\" (UID: \"e8ad0979-ef70-493c-8e4f-8a96a151dd79\") " pod="openshift-monitoring/node-exporter-zwkxm"
Sep 11 14:16:56.337241 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:56.337204    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zwkxm"
Sep 11 14:16:56.348003 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:16:56.347961    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8ad0979_ef70_493c_8e4f_8a96a151dd79.slice/crio-e96332d10ed0a8cdc481e1461f58c4390401a09aed65b62faaa292a9d502a3e7 WatchSource:0}: Error finding container e96332d10ed0a8cdc481e1461f58c4390401a09aed65b62faaa292a9d502a3e7: Status 404 returned error can't find the container with id e96332d10ed0a8cdc481e1461f58c4390401a09aed65b62faaa292a9d502a3e7
Sep 11 14:16:56.936419 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:56.936366    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zwkxm" event={"ID":"e8ad0979-ef70-493c-8e4f-8a96a151dd79","Type":"ContainerStarted","Data":"e96332d10ed0a8cdc481e1461f58c4390401a09aed65b62faaa292a9d502a3e7"}
Sep 11 14:16:57.854729 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:57.854699    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-fbb68bfc-66pmz"
Sep 11 14:16:57.941677 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:57.941635    2581 generic.go:356] "Generic (PLEG): container finished" podID="e8ad0979-ef70-493c-8e4f-8a96a151dd79" containerID="9ef66c0d501c3273bffb14e1554a4191d9200f12cf39e0dc12877856f8ef95f4" exitCode=0
Sep 11 14:16:57.941834 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:57.941717    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zwkxm" event={"ID":"e8ad0979-ef70-493c-8e4f-8a96a151dd79","Type":"ContainerDied","Data":"9ef66c0d501c3273bffb14e1554a4191d9200f12cf39e0dc12877856f8ef95f4"}
Sep 11 14:16:58.946825 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:58.946789    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zwkxm" event={"ID":"e8ad0979-ef70-493c-8e4f-8a96a151dd79","Type":"ContainerStarted","Data":"092f0457b1da839d18389bcf9e870dcfb1f18edd54363093679b78c8754fc908"}
Sep 11 14:16:58.947213 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:58.946854    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zwkxm" event={"ID":"e8ad0979-ef70-493c-8e4f-8a96a151dd79","Type":"ContainerStarted","Data":"8d944e15bd770bc332099618f2c8e83a94256c35bebbee557a98a4fc2914a1cb"}
Sep 11 14:16:59.004625 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:59.004578    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zwkxm" podStartSLOduration=2.834519429 podStartE2EDuration="4.004560977s" totalImagesPullingTime="1.108539015s" totalInitContainerRuntime="61.502533ms" isStatefulPod=true podCreationTimestamp="2026-09-11 14:16:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:59.00374443 +0000 UTC m=+122.154294504" watchObservedRunningTime="2026-09-11 14:16:59.004560977 +0000 UTC m=+122.155111022"
Sep 11 14:16:59.061492 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:59.061456    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-549f9d7d48-t7dbk"]
Sep 11 14:16:59.064887 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:59.064870    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-549f9d7d48-t7dbk"
Sep 11 14:16:59.068107 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:59.068084    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 11 14:16:59.068107 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:59.068102    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 11 14:16:59.068336 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:59.068323    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 11 14:16:59.068391 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:59.068358    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 11 14:16:59.068695 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:59.068679    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 11 14:16:59.068815 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:59.068798    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-nmn87\""
Sep 11 14:16:59.068882 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:59.068802    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-789n6olam63el\""
Sep 11 14:16:59.096175 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:59.096143    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-549f9d7d48-t7dbk"]
Sep 11 14:16:59.185241 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:59.185206    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03849958-e526-46c3-a90c-fd764025d2f2-metrics-client-ca\") pod \"thanos-querier-549f9d7d48-t7dbk\" (UID: \"03849958-e526-46c3-a90c-fd764025d2f2\") " pod="openshift-monitoring/thanos-querier-549f9d7d48-t7dbk"
Sep 11 14:16:59.185420 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:59.185312    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/03849958-e526-46c3-a90c-fd764025d2f2-secret-grpc-tls\") pod \"thanos-querier-549f9d7d48-t7dbk\" (UID: \"03849958-e526-46c3-a90c-fd764025d2f2\") " pod="openshift-monitoring/thanos-querier-549f9d7d48-t7dbk"
Sep 11 14:16:59.185420 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:59.185353    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/03849958-e526-46c3-a90c-fd764025d2f2-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-549f9d7d48-t7dbk\" (UID: \"03849958-e526-46c3-a90c-fd764025d2f2\") " pod="openshift-monitoring/thanos-querier-549f9d7d48-t7dbk"
Sep 11 14:16:59.185420 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:59.185380    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj5vd\" (UniqueName: \"kubernetes.io/projected/03849958-e526-46c3-a90c-fd764025d2f2-kube-api-access-qj5vd\") pod \"thanos-querier-549f9d7d48-t7dbk\" (UID: \"03849958-e526-46c3-a90c-fd764025d2f2\") " pod="openshift-monitoring/thanos-querier-549f9d7d48-t7dbk"
Sep 11 14:16:59.185420 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:59.185409    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/03849958-e526-46c3-a90c-fd764025d2f2-secret-thanos-querier-tls\") pod \"thanos-querier-549f9d7d48-t7dbk\" (UID: \"03849958-e526-46c3-a90c-fd764025d2f2\") " pod="openshift-monitoring/thanos-querier-549f9d7d48-t7dbk"
Sep 11 14:16:59.185550 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:59.185435    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/03849958-e526-46c3-a90c-fd764025d2f2-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-549f9d7d48-t7dbk\" (UID: \"03849958-e526-46c3-a90c-fd764025d2f2\") " pod="openshift-monitoring/thanos-querier-549f9d7d48-t7dbk"
Sep 11 14:16:59.185550 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:59.185454    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/03849958-e526-46c3-a90c-fd764025d2f2-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-549f9d7d48-t7dbk\" (UID: \"03849958-e526-46c3-a90c-fd764025d2f2\") " pod="openshift-monitoring/thanos-querier-549f9d7d48-t7dbk"
Sep 11 14:16:59.185550 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:59.185474    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/03849958-e526-46c3-a90c-fd764025d2f2-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-549f9d7d48-t7dbk\" (UID: \"03849958-e526-46c3-a90c-fd764025d2f2\") " pod="openshift-monitoring/thanos-querier-549f9d7d48-t7dbk"
Sep 11 14:16:59.286294 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:59.286259    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/03849958-e526-46c3-a90c-fd764025d2f2-secret-grpc-tls\") pod \"thanos-querier-549f9d7d48-t7dbk\" (UID: \"03849958-e526-46c3-a90c-fd764025d2f2\") " pod="openshift-monitoring/thanos-querier-549f9d7d48-t7dbk"
Sep 11 14:16:59.286466 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:59.286299    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/03849958-e526-46c3-a90c-fd764025d2f2-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-549f9d7d48-t7dbk\" (UID: \"03849958-e526-46c3-a90c-fd764025d2f2\") " pod="openshift-monitoring/thanos-querier-549f9d7d48-t7dbk"
Sep 11 14:16:59.286466 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:59.286320    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj5vd\" (UniqueName: \"kubernetes.io/projected/03849958-e526-46c3-a90c-fd764025d2f2-kube-api-access-qj5vd\") pod \"thanos-querier-549f9d7d48-t7dbk\" (UID: \"03849958-e526-46c3-a90c-fd764025d2f2\") " pod="openshift-monitoring/thanos-querier-549f9d7d48-t7dbk"
Sep 11 14:16:59.286466 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:59.286351    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/03849958-e526-46c3-a90c-fd764025d2f2-secret-thanos-querier-tls\") pod \"thanos-querier-549f9d7d48-t7dbk\" (UID: \"03849958-e526-46c3-a90c-fd764025d2f2\") " pod="openshift-monitoring/thanos-querier-549f9d7d48-t7dbk"
Sep 11 14:16:59.286466 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:59.286369    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/03849958-e526-46c3-a90c-fd764025d2f2-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-549f9d7d48-t7dbk\" (UID: \"03849958-e526-46c3-a90c-fd764025d2f2\") " pod="openshift-monitoring/thanos-querier-549f9d7d48-t7dbk"
Sep 11 14:16:59.286466 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:59.286389    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/03849958-e526-46c3-a90c-fd764025d2f2-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-549f9d7d48-t7dbk\" (UID: \"03849958-e526-46c3-a90c-fd764025d2f2\") " pod="openshift-monitoring/thanos-querier-549f9d7d48-t7dbk"
Sep 11 14:16:59.286466 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:59.286441    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/03849958-e526-46c3-a90c-fd764025d2f2-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-549f9d7d48-t7dbk\" (UID: \"03849958-e526-46c3-a90c-fd764025d2f2\") " pod="openshift-monitoring/thanos-querier-549f9d7d48-t7dbk"
Sep 11 14:16:59.286752 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:59.286479    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03849958-e526-46c3-a90c-fd764025d2f2-metrics-client-ca\") pod \"thanos-querier-549f9d7d48-t7dbk\" (UID: \"03849958-e526-46c3-a90c-fd764025d2f2\") " pod="openshift-monitoring/thanos-querier-549f9d7d48-t7dbk"
Sep 11 14:16:59.287359 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:59.287279    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03849958-e526-46c3-a90c-fd764025d2f2-metrics-client-ca\") pod \"thanos-querier-549f9d7d48-t7dbk\" (UID: \"03849958-e526-46c3-a90c-fd764025d2f2\") " pod="openshift-monitoring/thanos-querier-549f9d7d48-t7dbk"
Sep 11 14:16:59.289364 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:59.289330    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/03849958-e526-46c3-a90c-fd764025d2f2-secret-grpc-tls\") pod \"thanos-querier-549f9d7d48-t7dbk\" (UID: \"03849958-e526-46c3-a90c-fd764025d2f2\") " pod="openshift-monitoring/thanos-querier-549f9d7d48-t7dbk"
Sep 11 14:16:59.289570 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:59.289551    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/03849958-e526-46c3-a90c-fd764025d2f2-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-549f9d7d48-t7dbk\" (UID: \"03849958-e526-46c3-a90c-fd764025d2f2\") " pod="openshift-monitoring/thanos-querier-549f9d7d48-t7dbk"
Sep 11 14:16:59.289780 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:59.289747    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/03849958-e526-46c3-a90c-fd764025d2f2-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-549f9d7d48-t7dbk\" (UID: \"03849958-e526-46c3-a90c-fd764025d2f2\") " pod="openshift-monitoring/thanos-querier-549f9d7d48-t7dbk"
Sep 11 14:16:59.289986 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:59.289948    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/03849958-e526-46c3-a90c-fd764025d2f2-secret-thanos-querier-tls\") pod \"thanos-querier-549f9d7d48-t7dbk\" (UID: \"03849958-e526-46c3-a90c-fd764025d2f2\") " pod="openshift-monitoring/thanos-querier-549f9d7d48-t7dbk"
Sep 11 14:16:59.290382 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:59.290362    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/03849958-e526-46c3-a90c-fd764025d2f2-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-549f9d7d48-t7dbk\" (UID: \"03849958-e526-46c3-a90c-fd764025d2f2\") " pod="openshift-monitoring/thanos-querier-549f9d7d48-t7dbk"
Sep 11 14:16:59.290619 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:59.290598    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/03849958-e526-46c3-a90c-fd764025d2f2-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-549f9d7d48-t7dbk\" (UID: \"03849958-e526-46c3-a90c-fd764025d2f2\") " pod="openshift-monitoring/thanos-querier-549f9d7d48-t7dbk"
Sep 11 14:16:59.298401 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:59.298376    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj5vd\" (UniqueName: \"kubernetes.io/projected/03849958-e526-46c3-a90c-fd764025d2f2-kube-api-access-qj5vd\") pod \"thanos-querier-549f9d7d48-t7dbk\" (UID: \"03849958-e526-46c3-a90c-fd764025d2f2\") " pod="openshift-monitoring/thanos-querier-549f9d7d48-t7dbk"
Sep 11 14:16:59.373748 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:59.373713    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-549f9d7d48-t7dbk"
Sep 11 14:16:59.698500 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:59.698427    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-549f9d7d48-t7dbk"]
Sep 11 14:16:59.701993 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:16:59.701951    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03849958_e526_46c3_a90c_fd764025d2f2.slice/crio-0cdf147b81cd3999bb577578fce2718994e7fe48cb5ad0a30397fd293b889935 WatchSource:0}: Error finding container 0cdf147b81cd3999bb577578fce2718994e7fe48cb5ad0a30397fd293b889935: Status 404 returned error can't find the container with id 0cdf147b81cd3999bb577578fce2718994e7fe48cb5ad0a30397fd293b889935
Sep 11 14:16:59.951766 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:16:59.951677    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-549f9d7d48-t7dbk" event={"ID":"03849958-e526-46c3-a90c-fd764025d2f2","Type":"ContainerStarted","Data":"0cdf147b81cd3999bb577578fce2718994e7fe48cb5ad0a30397fd293b889935"}
Sep 11 14:17:00.522808 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:00.522772    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7b8fc55444-8rwqg"]
Sep 11 14:17:00.526782 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:00.526759    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7b8fc55444-8rwqg"
Sep 11 14:17:00.531363 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:00.531171    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 11 14:17:00.531363 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:00.531199    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-ftbn6\""
Sep 11 14:17:00.531363 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:00.531225    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 11 14:17:00.531363 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:00.531247    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 11 14:17:00.531363 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:00.531171    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 11 14:17:00.531363 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:00.531368    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-5t3lg6gil2klt\""
Sep 11 14:17:00.546897 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:00.546870    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7b8fc55444-8rwqg"]
Sep 11 14:17:00.699075 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:00.699029    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpjg6\" (UniqueName: \"kubernetes.io/projected/ceed54bd-5296-4185-9bc9-53a732bbd63f-kube-api-access-gpjg6\") pod \"metrics-server-7b8fc55444-8rwqg\" (UID: \"ceed54bd-5296-4185-9bc9-53a732bbd63f\") " pod="openshift-monitoring/metrics-server-7b8fc55444-8rwqg"
Sep 11 14:17:00.699241 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:00.699091    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ceed54bd-5296-4185-9bc9-53a732bbd63f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b8fc55444-8rwqg\" (UID: \"ceed54bd-5296-4185-9bc9-53a732bbd63f\") " pod="openshift-monitoring/metrics-server-7b8fc55444-8rwqg"
Sep 11 14:17:00.699241 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:00.699165    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ceed54bd-5296-4185-9bc9-53a732bbd63f-client-ca-bundle\") pod \"metrics-server-7b8fc55444-8rwqg\" (UID: \"ceed54bd-5296-4185-9bc9-53a732bbd63f\") " pod="openshift-monitoring/metrics-server-7b8fc55444-8rwqg"
Sep 11 14:17:00.699241 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:00.699210    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ceed54bd-5296-4185-9bc9-53a732bbd63f-audit-log\") pod \"metrics-server-7b8fc55444-8rwqg\" (UID: \"ceed54bd-5296-4185-9bc9-53a732bbd63f\") " pod="openshift-monitoring/metrics-server-7b8fc55444-8rwqg"
Sep 11 14:17:00.699412 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:00.699248    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ceed54bd-5296-4185-9bc9-53a732bbd63f-secret-metrics-server-tls\") pod \"metrics-server-7b8fc55444-8rwqg\" (UID: \"ceed54bd-5296-4185-9bc9-53a732bbd63f\") " pod="openshift-monitoring/metrics-server-7b8fc55444-8rwqg"
Sep 11 14:17:00.699412 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:00.699304    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ceed54bd-5296-4185-9bc9-53a732bbd63f-metrics-server-audit-profiles\") pod \"metrics-server-7b8fc55444-8rwqg\" (UID: \"ceed54bd-5296-4185-9bc9-53a732bbd63f\") " pod="openshift-monitoring/metrics-server-7b8fc55444-8rwqg"
Sep 11 14:17:00.699412 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:00.699370    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ceed54bd-5296-4185-9bc9-53a732bbd63f-secret-metrics-server-client-certs\") pod \"metrics-server-7b8fc55444-8rwqg\" (UID: \"ceed54bd-5296-4185-9bc9-53a732bbd63f\") " pod="openshift-monitoring/metrics-server-7b8fc55444-8rwqg"
Sep 11 14:17:00.800576 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:00.800485    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpjg6\" (UniqueName: \"kubernetes.io/projected/ceed54bd-5296-4185-9bc9-53a732bbd63f-kube-api-access-gpjg6\") pod \"metrics-server-7b8fc55444-8rwqg\" (UID: \"ceed54bd-5296-4185-9bc9-53a732bbd63f\") " pod="openshift-monitoring/metrics-server-7b8fc55444-8rwqg"
Sep 11 14:17:00.800576 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:00.800548    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ceed54bd-5296-4185-9bc9-53a732bbd63f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b8fc55444-8rwqg\" (UID: \"ceed54bd-5296-4185-9bc9-53a732bbd63f\") " pod="openshift-monitoring/metrics-server-7b8fc55444-8rwqg"
Sep 11 14:17:00.800812 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:00.800584    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ceed54bd-5296-4185-9bc9-53a732bbd63f-client-ca-bundle\") pod \"metrics-server-7b8fc55444-8rwqg\" (UID: \"ceed54bd-5296-4185-9bc9-53a732bbd63f\") " pod="openshift-monitoring/metrics-server-7b8fc55444-8rwqg"
Sep 11 14:17:00.800812 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:00.800611    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ceed54bd-5296-4185-9bc9-53a732bbd63f-audit-log\") pod \"metrics-server-7b8fc55444-8rwqg\" (UID: \"ceed54bd-5296-4185-9bc9-53a732bbd63f\") " pod="openshift-monitoring/metrics-server-7b8fc55444-8rwqg"
Sep 11 14:17:00.800812 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:00.800639    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ceed54bd-5296-4185-9bc9-53a732bbd63f-secret-metrics-server-tls\") pod \"metrics-server-7b8fc55444-8rwqg\" (UID: \"ceed54bd-5296-4185-9bc9-53a732bbd63f\") " pod="openshift-monitoring/metrics-server-7b8fc55444-8rwqg"
Sep 11 14:17:00.800812 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:00.800684    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ceed54bd-5296-4185-9bc9-53a732bbd63f-metrics-server-audit-profiles\") pod \"metrics-server-7b8fc55444-8rwqg\" (UID: \"ceed54bd-5296-4185-9bc9-53a732bbd63f\") " pod="openshift-monitoring/metrics-server-7b8fc55444-8rwqg"
Sep 11 14:17:00.800812 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:00.800718    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ceed54bd-5296-4185-9bc9-53a732bbd63f-secret-metrics-server-client-certs\") pod \"metrics-server-7b8fc55444-8rwqg\" (UID: \"ceed54bd-5296-4185-9bc9-53a732bbd63f\") " pod="openshift-monitoring/metrics-server-7b8fc55444-8rwqg"
Sep 11 14:17:00.801563 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:00.801510    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ceed54bd-5296-4185-9bc9-53a732bbd63f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b8fc55444-8rwqg\" (UID: \"ceed54bd-5296-4185-9bc9-53a732bbd63f\") " pod="openshift-monitoring/metrics-server-7b8fc55444-8rwqg"
Sep 11 14:17:00.802038 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:00.802008    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ceed54bd-5296-4185-9bc9-53a732bbd63f-audit-log\") pod \"metrics-server-7b8fc55444-8rwqg\" (UID: \"ceed54bd-5296-4185-9bc9-53a732bbd63f\") " pod="openshift-monitoring/metrics-server-7b8fc55444-8rwqg"
Sep 11 14:17:00.802169 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:00.802112    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ceed54bd-5296-4185-9bc9-53a732bbd63f-metrics-server-audit-profiles\") pod \"metrics-server-7b8fc55444-8rwqg\" (UID: \"ceed54bd-5296-4185-9bc9-53a732bbd63f\") " pod="openshift-monitoring/metrics-server-7b8fc55444-8rwqg"
Sep 11 14:17:00.803578 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:00.803547    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ceed54bd-5296-4185-9bc9-53a732bbd63f-secret-metrics-server-client-certs\") pod \"metrics-server-7b8fc55444-8rwqg\" (UID: \"ceed54bd-5296-4185-9bc9-53a732bbd63f\") " pod="openshift-monitoring/metrics-server-7b8fc55444-8rwqg"
Sep 11 14:17:00.803832 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:00.803812    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ceed54bd-5296-4185-9bc9-53a732bbd63f-secret-metrics-server-tls\") pod \"metrics-server-7b8fc55444-8rwqg\" (UID: \"ceed54bd-5296-4185-9bc9-53a732bbd63f\") " pod="openshift-monitoring/metrics-server-7b8fc55444-8rwqg"
Sep 11 14:17:00.803932 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:00.803907    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ceed54bd-5296-4185-9bc9-53a732bbd63f-client-ca-bundle\") pod \"metrics-server-7b8fc55444-8rwqg\" (UID: \"ceed54bd-5296-4185-9bc9-53a732bbd63f\") " pod="openshift-monitoring/metrics-server-7b8fc55444-8rwqg"
Sep 11 14:17:00.813609 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:00.813586    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpjg6\" (UniqueName: \"kubernetes.io/projected/ceed54bd-5296-4185-9bc9-53a732bbd63f-kube-api-access-gpjg6\") pod \"metrics-server-7b8fc55444-8rwqg\" (UID: \"ceed54bd-5296-4185-9bc9-53a732bbd63f\") " pod="openshift-monitoring/metrics-server-7b8fc55444-8rwqg"
Sep 11 14:17:00.838228 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:00.838201    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7b8fc55444-8rwqg"
Sep 11 14:17:01.130947 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:01.130865    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7b8fc55444-8rwqg"]
Sep 11 14:17:01.567919 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:17:01.567887    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceed54bd_5296_4185_9bc9_53a732bbd63f.slice/crio-b8e91bcd0ca409afd3aa1c60da929b5c72ffbac06368a068790880c9a9b18be2 WatchSource:0}: Error finding container b8e91bcd0ca409afd3aa1c60da929b5c72ffbac06368a068790880c9a9b18be2: Status 404 returned error can't find the container with id b8e91bcd0ca409afd3aa1c60da929b5c72ffbac06368a068790880c9a9b18be2
Sep 11 14:17:01.964096 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:01.963990    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-549f9d7d48-t7dbk" event={"ID":"03849958-e526-46c3-a90c-fd764025d2f2","Type":"ContainerStarted","Data":"b84530a8f67d1f2282d45707e8bcd12470d3eb5a9fe0843ff2257eac3399d823"}
Sep 11 14:17:01.964256 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:01.964096    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-549f9d7d48-t7dbk" event={"ID":"03849958-e526-46c3-a90c-fd764025d2f2","Type":"ContainerStarted","Data":"7d43036c5a226057e77eef30939423044c19bba7ce90f459474528d71dfd49cc"}
Sep 11 14:17:01.964256 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:01.964125    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-549f9d7d48-t7dbk" event={"ID":"03849958-e526-46c3-a90c-fd764025d2f2","Type":"ContainerStarted","Data":"82a3226c3df3d66a89aa2a120b71eda9926b08244f9c49cf65a1fc3eb28eac5d"}
Sep 11 14:17:01.965085 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:01.965063    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7b8fc55444-8rwqg" event={"ID":"ceed54bd-5296-4185-9bc9-53a732bbd63f","Type":"ContainerStarted","Data":"b8e91bcd0ca409afd3aa1c60da929b5c72ffbac06368a068790880c9a9b18be2"}
Sep 11 14:17:03.975221 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:03.975180    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-549f9d7d48-t7dbk" event={"ID":"03849958-e526-46c3-a90c-fd764025d2f2","Type":"ContainerStarted","Data":"863b794985e0f9721a9759d6198070d1ac9c3f60cc8fc4f4c915922b45a95488"}
Sep 11 14:17:03.975694 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:03.975262    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-549f9d7d48-t7dbk" event={"ID":"03849958-e526-46c3-a90c-fd764025d2f2","Type":"ContainerStarted","Data":"0433a702f5cbf61b0fe961a6365cbbd97daeb3c154cee657cfdf091e77fd0152"}
Sep 11 14:17:03.975694 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:03.975307    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-549f9d7d48-t7dbk" event={"ID":"03849958-e526-46c3-a90c-fd764025d2f2","Type":"ContainerStarted","Data":"fee094cef2f3adbe999b580fedf1176fdc617f3306a67ba0676d4c895c0f1dad"}
Sep 11 14:17:03.975694 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:03.975355    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-549f9d7d48-t7dbk"
Sep 11 14:17:03.976857 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:03.976833    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7b8fc55444-8rwqg" event={"ID":"ceed54bd-5296-4185-9bc9-53a732bbd63f","Type":"ContainerStarted","Data":"a1ab4166c29b61bce6ecf9acbc060e1348f55290a3e145507eeba51ecdea7c25"}
Sep 11 14:17:04.013757 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:04.013710    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-549f9d7d48-t7dbk" podStartSLOduration=1.7902335969999998 podStartE2EDuration="5.013696702s" totalImagesPullingTime="3.223463105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:16:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:17:04.013457665 +0000 UTC m=+127.164007730" watchObservedRunningTime="2026-09-11 14:17:04.013696702 +0000 UTC m=+127.164246765"
Sep 11 14:17:04.050363 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:04.050305    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7b8fc55444-8rwqg" podStartSLOduration=2.558694396 podStartE2EDuration="4.050284823s" totalImagesPullingTime="1.491590427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:17:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:17:04.049425488 +0000 UTC m=+127.199975574" watchObservedRunningTime="2026-09-11 14:17:04.050284823 +0000 UTC m=+127.200834889"
Sep 11 14:17:06.860181 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:06.860143    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6dcff7596-2d4qc"]
Sep 11 14:17:06.863047 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:06.863026    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dcff7596-2d4qc"
Sep 11 14:17:06.866335 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:06.866312    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 11 14:17:06.866469 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:06.866367    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 11 14:17:06.866469 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:06.866368    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 11 14:17:06.867335 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:06.867318    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 11 14:17:06.867677 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:06.867639    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 11 14:17:06.867677 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:06.867659    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 11 14:17:06.867864 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:06.867697    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-llnzd\""
Sep 11 14:17:06.867864 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:06.867697    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 11 14:17:06.885486 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:06.885459    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6dcff7596-2d4qc"]
Sep 11 14:17:06.940678 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:06.940642    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aaf265ac-e4ca-4242-8d94-82541dbb9938-console-serving-cert\") pod \"console-6dcff7596-2d4qc\" (UID: \"aaf265ac-e4ca-4242-8d94-82541dbb9938\") " pod="openshift-console/console-6dcff7596-2d4qc"
Sep 11 14:17:06.940862 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:06.940693    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aaf265ac-e4ca-4242-8d94-82541dbb9938-console-config\") pod \"console-6dcff7596-2d4qc\" (UID: \"aaf265ac-e4ca-4242-8d94-82541dbb9938\") " pod="openshift-console/console-6dcff7596-2d4qc"
Sep 11 14:17:06.940862 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:06.940723    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gw8h\" (UniqueName: \"kubernetes.io/projected/aaf265ac-e4ca-4242-8d94-82541dbb9938-kube-api-access-7gw8h\") pod \"console-6dcff7596-2d4qc\" (UID: \"aaf265ac-e4ca-4242-8d94-82541dbb9938\") " pod="openshift-console/console-6dcff7596-2d4qc"
Sep 11 14:17:06.940862 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:06.940788    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aaf265ac-e4ca-4242-8d94-82541dbb9938-console-oauth-config\") pod \"console-6dcff7596-2d4qc\" (UID: \"aaf265ac-e4ca-4242-8d94-82541dbb9938\") " pod="openshift-console/console-6dcff7596-2d4qc"
Sep 11 14:17:06.940862 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:06.940823    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aaf265ac-e4ca-4242-8d94-82541dbb9938-service-ca\") pod \"console-6dcff7596-2d4qc\" (UID: \"aaf265ac-e4ca-4242-8d94-82541dbb9938\") " pod="openshift-console/console-6dcff7596-2d4qc"
Sep 11 14:17:06.940862 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:06.940857    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aaf265ac-e4ca-4242-8d94-82541dbb9938-oauth-serving-cert\") pod \"console-6dcff7596-2d4qc\" (UID: \"aaf265ac-e4ca-4242-8d94-82541dbb9938\") " pod="openshift-console/console-6dcff7596-2d4qc"
Sep 11 14:17:07.041887 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:07.041845    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aaf265ac-e4ca-4242-8d94-82541dbb9938-console-serving-cert\") pod \"console-6dcff7596-2d4qc\" (UID: \"aaf265ac-e4ca-4242-8d94-82541dbb9938\") " pod="openshift-console/console-6dcff7596-2d4qc"
Sep 11 14:17:07.042104 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:07.041903    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aaf265ac-e4ca-4242-8d94-82541dbb9938-console-config\") pod \"console-6dcff7596-2d4qc\" (UID: \"aaf265ac-e4ca-4242-8d94-82541dbb9938\") " pod="openshift-console/console-6dcff7596-2d4qc"
Sep 11 14:17:07.042104 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:07.041949    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gw8h\" (UniqueName: \"kubernetes.io/projected/aaf265ac-e4ca-4242-8d94-82541dbb9938-kube-api-access-7gw8h\") pod \"console-6dcff7596-2d4qc\" (UID: \"aaf265ac-e4ca-4242-8d94-82541dbb9938\") " pod="openshift-console/console-6dcff7596-2d4qc"
Sep 11 14:17:07.042104 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:07.042000    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aaf265ac-e4ca-4242-8d94-82541dbb9938-console-oauth-config\") pod \"console-6dcff7596-2d4qc\" (UID: \"aaf265ac-e4ca-4242-8d94-82541dbb9938\") " pod="openshift-console/console-6dcff7596-2d4qc"
Sep 11 14:17:07.042104 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:07.042020    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aaf265ac-e4ca-4242-8d94-82541dbb9938-service-ca\") pod \"console-6dcff7596-2d4qc\" (UID: \"aaf265ac-e4ca-4242-8d94-82541dbb9938\") " pod="openshift-console/console-6dcff7596-2d4qc"
Sep 11 14:17:07.042104 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:07.042060    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aaf265ac-e4ca-4242-8d94-82541dbb9938-oauth-serving-cert\") pod \"console-6dcff7596-2d4qc\" (UID: \"aaf265ac-e4ca-4242-8d94-82541dbb9938\") " pod="openshift-console/console-6dcff7596-2d4qc"
Sep 11 14:17:07.042706 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:07.042678    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aaf265ac-e4ca-4242-8d94-82541dbb9938-console-config\") pod \"console-6dcff7596-2d4qc\" (UID: \"aaf265ac-e4ca-4242-8d94-82541dbb9938\") " pod="openshift-console/console-6dcff7596-2d4qc"
Sep 11 14:17:07.042801 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:07.042719    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aaf265ac-e4ca-4242-8d94-82541dbb9938-oauth-serving-cert\") pod \"console-6dcff7596-2d4qc\" (UID: \"aaf265ac-e4ca-4242-8d94-82541dbb9938\") " pod="openshift-console/console-6dcff7596-2d4qc"
Sep 11 14:17:07.042910 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:07.042888    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aaf265ac-e4ca-4242-8d94-82541dbb9938-service-ca\") pod \"console-6dcff7596-2d4qc\" (UID: \"aaf265ac-e4ca-4242-8d94-82541dbb9938\") " pod="openshift-console/console-6dcff7596-2d4qc"
Sep 11 14:17:07.044332 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:07.044314    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aaf265ac-e4ca-4242-8d94-82541dbb9938-console-oauth-config\") pod \"console-6dcff7596-2d4qc\" (UID: \"aaf265ac-e4ca-4242-8d94-82541dbb9938\") " pod="openshift-console/console-6dcff7596-2d4qc"
Sep 11 14:17:07.044427 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:07.044385    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aaf265ac-e4ca-4242-8d94-82541dbb9938-console-serving-cert\") pod \"console-6dcff7596-2d4qc\" (UID: \"aaf265ac-e4ca-4242-8d94-82541dbb9938\") " pod="openshift-console/console-6dcff7596-2d4qc"
Sep 11 14:17:07.052060 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:07.052033    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gw8h\" (UniqueName: \"kubernetes.io/projected/aaf265ac-e4ca-4242-8d94-82541dbb9938-kube-api-access-7gw8h\") pod \"console-6dcff7596-2d4qc\" (UID: \"aaf265ac-e4ca-4242-8d94-82541dbb9938\") " pod="openshift-console/console-6dcff7596-2d4qc"
Sep 11 14:17:07.143154 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:07.143055    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3376794-b10e-4353-b8f3-4639741019eb-metrics-certs\") pod \"network-metrics-daemon-wgmt4\" (UID: \"d3376794-b10e-4353-b8f3-4639741019eb\") " pod="openshift-multus/network-metrics-daemon-wgmt4"
Sep 11 14:17:07.145410 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:07.145387    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3376794-b10e-4353-b8f3-4639741019eb-metrics-certs\") pod \"network-metrics-daemon-wgmt4\" (UID: \"d3376794-b10e-4353-b8f3-4639741019eb\") " pod="openshift-multus/network-metrics-daemon-wgmt4"
Sep 11 14:17:07.173496 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:07.173465    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dcff7596-2d4qc"
Sep 11 14:17:07.425240 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:07.425175    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-96vl6\""
Sep 11 14:17:07.432837 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:07.432814    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wgmt4"
Sep 11 14:17:07.451708 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:07.451680    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6dcff7596-2d4qc"]
Sep 11 14:17:07.455680 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:17:07.453918    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaf265ac_e4ca_4242_8d94_82541dbb9938.slice/crio-dfdb35f9a8f257e1a018b2fa5fd3d4672d6ce4416fd05006074a6f1b44a61722 WatchSource:0}: Error finding container dfdb35f9a8f257e1a018b2fa5fd3d4672d6ce4416fd05006074a6f1b44a61722: Status 404 returned error can't find the container with id dfdb35f9a8f257e1a018b2fa5fd3d4672d6ce4416fd05006074a6f1b44a61722
Sep 11 14:17:07.711755 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:07.711671    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wgmt4"]
Sep 11 14:17:07.714480 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:17:07.714452    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3376794_b10e_4353_b8f3_4639741019eb.slice/crio-3ceb983ded5a9211c07b4cdee06fce12daae9face8f0b36b04f9e8b1c814d7a7 WatchSource:0}: Error finding container 3ceb983ded5a9211c07b4cdee06fce12daae9face8f0b36b04f9e8b1c814d7a7: Status 404 returned error can't find the container with id 3ceb983ded5a9211c07b4cdee06fce12daae9face8f0b36b04f9e8b1c814d7a7
Sep 11 14:17:07.991407 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:07.991367    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wgmt4" event={"ID":"d3376794-b10e-4353-b8f3-4639741019eb","Type":"ContainerStarted","Data":"3ceb983ded5a9211c07b4cdee06fce12daae9face8f0b36b04f9e8b1c814d7a7"}
Sep 11 14:17:07.992720 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:07.992688    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dcff7596-2d4qc" event={"ID":"aaf265ac-e4ca-4242-8d94-82541dbb9938","Type":"ContainerStarted","Data":"dfdb35f9a8f257e1a018b2fa5fd3d4672d6ce4416fd05006074a6f1b44a61722"}
Sep 11 14:17:09.001798 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:09.001750    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wgmt4" event={"ID":"d3376794-b10e-4353-b8f3-4639741019eb","Type":"ContainerStarted","Data":"f74ebd4766418840144862615e786b7143d8ad61fa0b0d4e16b28d5090c88894"}
Sep 11 14:17:09.002270 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:09.001839    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wgmt4" event={"ID":"d3376794-b10e-4353-b8f3-4639741019eb","Type":"ContainerStarted","Data":"e92b4a49f8fe15334e7dbbf506db0b67f89608e5bac33c2c9fb7afcfd50810b0"}
Sep 11 14:17:09.029718 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:09.029478    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-wgmt4" podStartSLOduration=131.060014353 podStartE2EDuration="2m12.029457278s" totalImagesPullingTime="969.442925ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:14:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:17:09.026229342 +0000 UTC m=+132.176779408" watchObservedRunningTime="2026-09-11 14:17:09.029457278 +0000 UTC m=+132.180007346"
Sep 11 14:17:09.516851 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:09.516818    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-fbb68bfc-66pmz"]
Sep 11 14:17:09.986561 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:09.986479    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-549f9d7d48-t7dbk"
Sep 11 14:17:11.011449 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:11.011410    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dcff7596-2d4qc" event={"ID":"aaf265ac-e4ca-4242-8d94-82541dbb9938","Type":"ContainerStarted","Data":"8d6be4a8e3d1e106946f3c09e196a6b046f5156bc02f1dec3af46c6143e47ffa"}
Sep 11 14:17:11.078763 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:11.078700    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6dcff7596-2d4qc" podStartSLOduration=2.345855312 podStartE2EDuration="5.078678627s" totalImagesPullingTime="2.732823315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:17:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:17:11.078294952 +0000 UTC m=+134.228845027" watchObservedRunningTime="2026-09-11 14:17:11.078678627 +0000 UTC m=+134.229228692"
Sep 11 14:17:17.174318 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:17.174266    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6dcff7596-2d4qc"
Sep 11 14:17:17.174720 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:17.174353    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6dcff7596-2d4qc"
Sep 11 14:17:17.179086 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:17.179064    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6dcff7596-2d4qc"
Sep 11 14:17:18.041233 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:18.041204    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6dcff7596-2d4qc"
Sep 11 14:17:20.839200 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:20.839153    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7b8fc55444-8rwqg"
Sep 11 14:17:20.839595 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:20.839228    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7b8fc55444-8rwqg"
Sep 11 14:17:28.145867 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:28.145834    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6dcff7596-2d4qc"]
Sep 11 14:17:33.092955 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:33.092918    2581 generic.go:356] "Generic (PLEG): container finished" podID="7dd4a673-2bd6-4dd4-a23f-f510ecb8793f" containerID="ecfed8196841af46930fffba2d563e2f026691f0c639050822cd1462f7aacc0a" exitCode=0
Sep 11 14:17:33.093358 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:33.092994    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-z7cxh" event={"ID":"7dd4a673-2bd6-4dd4-a23f-f510ecb8793f","Type":"ContainerDied","Data":"ecfed8196841af46930fffba2d563e2f026691f0c639050822cd1462f7aacc0a"}
Sep 11 14:17:33.093400 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:33.093383    2581 scope.go:117] "RemoveContainer" containerID="ecfed8196841af46930fffba2d563e2f026691f0c639050822cd1462f7aacc0a"
Sep 11 14:17:34.098717 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:34.098682    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-z7cxh" event={"ID":"7dd4a673-2bd6-4dd4-a23f-f510ecb8793f","Type":"ContainerStarted","Data":"10badd44a91c933e54acfc488a08dfb570ce756ab8975c10557bcc262ef733bd"}
Sep 11 14:17:34.537921 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:34.537854    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-fbb68bfc-66pmz" podUID="5115e3f0-399e-4db4-a892-ba00f8d8735f" containerName="registry" containerID="cri-o://919cd3bca193dcbcf3fe56cfe3f07d5e8beca372bad789a238a8350abd2a97cf" gracePeriod=30
Sep 11 14:17:34.776470 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:34.776442    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fbb68bfc-66pmz"
Sep 11 14:17:34.841269 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:34.841188    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5115e3f0-399e-4db4-a892-ba00f8d8735f-installation-pull-secrets\") pod \"5115e3f0-399e-4db4-a892-ba00f8d8735f\" (UID: \"5115e3f0-399e-4db4-a892-ba00f8d8735f\") "
Sep 11 14:17:34.841269 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:34.841221    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5115e3f0-399e-4db4-a892-ba00f8d8735f-bound-sa-token\") pod \"5115e3f0-399e-4db4-a892-ba00f8d8735f\" (UID: \"5115e3f0-399e-4db4-a892-ba00f8d8735f\") "
Sep 11 14:17:34.841269 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:34.841255    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5115e3f0-399e-4db4-a892-ba00f8d8735f-ca-trust-extracted\") pod \"5115e3f0-399e-4db4-a892-ba00f8d8735f\" (UID: \"5115e3f0-399e-4db4-a892-ba00f8d8735f\") "
Sep 11 14:17:34.841540 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:34.841275    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5115e3f0-399e-4db4-a892-ba00f8d8735f-image-registry-private-configuration\") pod \"5115e3f0-399e-4db4-a892-ba00f8d8735f\" (UID: \"5115e3f0-399e-4db4-a892-ba00f8d8735f\") "
Sep 11 14:17:34.841540 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:34.841365    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5115e3f0-399e-4db4-a892-ba00f8d8735f-registry-certificates\") pod \"5115e3f0-399e-4db4-a892-ba00f8d8735f\" (UID: \"5115e3f0-399e-4db4-a892-ba00f8d8735f\") "
Sep 11 14:17:34.841540 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:34.841420    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5115e3f0-399e-4db4-a892-ba00f8d8735f-trusted-ca\") pod \"5115e3f0-399e-4db4-a892-ba00f8d8735f\" (UID: \"5115e3f0-399e-4db4-a892-ba00f8d8735f\") "
Sep 11 14:17:34.841540 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:34.841471    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbfh4\" (UniqueName: \"kubernetes.io/projected/5115e3f0-399e-4db4-a892-ba00f8d8735f-kube-api-access-lbfh4\") pod \"5115e3f0-399e-4db4-a892-ba00f8d8735f\" (UID: \"5115e3f0-399e-4db4-a892-ba00f8d8735f\") "
Sep 11 14:17:34.841540 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:34.841495    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5115e3f0-399e-4db4-a892-ba00f8d8735f-registry-tls\") pod \"5115e3f0-399e-4db4-a892-ba00f8d8735f\" (UID: \"5115e3f0-399e-4db4-a892-ba00f8d8735f\") "
Sep 11 14:17:34.841848 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:34.841815    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5115e3f0-399e-4db4-a892-ba00f8d8735f-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "5115e3f0-399e-4db4-a892-ba00f8d8735f" (UID: "5115e3f0-399e-4db4-a892-ba00f8d8735f"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:17:34.842273 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:34.842230    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5115e3f0-399e-4db4-a892-ba00f8d8735f-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "5115e3f0-399e-4db4-a892-ba00f8d8735f" (UID: "5115e3f0-399e-4db4-a892-ba00f8d8735f"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:17:34.844063 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:34.844012    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5115e3f0-399e-4db4-a892-ba00f8d8735f-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "5115e3f0-399e-4db4-a892-ba00f8d8735f" (UID: "5115e3f0-399e-4db4-a892-ba00f8d8735f"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:17:34.844156 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:34.844120    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5115e3f0-399e-4db4-a892-ba00f8d8735f-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "5115e3f0-399e-4db4-a892-ba00f8d8735f" (UID: "5115e3f0-399e-4db4-a892-ba00f8d8735f"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:17:34.844211 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:34.844161    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5115e3f0-399e-4db4-a892-ba00f8d8735f-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "5115e3f0-399e-4db4-a892-ba00f8d8735f" (UID: "5115e3f0-399e-4db4-a892-ba00f8d8735f"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:17:34.844247 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:34.844227    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5115e3f0-399e-4db4-a892-ba00f8d8735f-kube-api-access-lbfh4" (OuterVolumeSpecName: "kube-api-access-lbfh4") pod "5115e3f0-399e-4db4-a892-ba00f8d8735f" (UID: "5115e3f0-399e-4db4-a892-ba00f8d8735f"). InnerVolumeSpecName "kube-api-access-lbfh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:17:34.844247 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:34.844227    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5115e3f0-399e-4db4-a892-ba00f8d8735f-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "5115e3f0-399e-4db4-a892-ba00f8d8735f" (UID: "5115e3f0-399e-4db4-a892-ba00f8d8735f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:17:34.849785 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:34.849756    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5115e3f0-399e-4db4-a892-ba00f8d8735f-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "5115e3f0-399e-4db4-a892-ba00f8d8735f" (UID: "5115e3f0-399e-4db4-a892-ba00f8d8735f"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:17:34.942763 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:34.942708    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbfh4\" (UniqueName: \"kubernetes.io/projected/5115e3f0-399e-4db4-a892-ba00f8d8735f-kube-api-access-lbfh4\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:17:34.942763 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:34.942758    2581 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5115e3f0-399e-4db4-a892-ba00f8d8735f-registry-tls\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:17:34.942763 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:34.942768    2581 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5115e3f0-399e-4db4-a892-ba00f8d8735f-installation-pull-secrets\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:17:34.942763 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:34.942777    2581 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5115e3f0-399e-4db4-a892-ba00f8d8735f-bound-sa-token\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:17:34.943045 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:34.942785    2581 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5115e3f0-399e-4db4-a892-ba00f8d8735f-ca-trust-extracted\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:17:34.943045 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:34.942795    2581 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5115e3f0-399e-4db4-a892-ba00f8d8735f-image-registry-private-configuration\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:17:34.943045 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:34.942804    2581 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5115e3f0-399e-4db4-a892-ba00f8d8735f-registry-certificates\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:17:34.943045 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:34.942812    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5115e3f0-399e-4db4-a892-ba00f8d8735f-trusted-ca\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:17:35.104291 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:35.104192    2581 generic.go:356] "Generic (PLEG): container finished" podID="5115e3f0-399e-4db4-a892-ba00f8d8735f" containerID="919cd3bca193dcbcf3fe56cfe3f07d5e8beca372bad789a238a8350abd2a97cf" exitCode=0
Sep 11 14:17:35.104291 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:35.104248    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fbb68bfc-66pmz"
Sep 11 14:17:35.104291 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:35.104265    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fbb68bfc-66pmz" event={"ID":"5115e3f0-399e-4db4-a892-ba00f8d8735f","Type":"ContainerDied","Data":"919cd3bca193dcbcf3fe56cfe3f07d5e8beca372bad789a238a8350abd2a97cf"}
Sep 11 14:17:35.104876 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:35.104331    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fbb68bfc-66pmz" event={"ID":"5115e3f0-399e-4db4-a892-ba00f8d8735f","Type":"ContainerDied","Data":"be6773d92bf3e01ed724d89e27cb5288717c07a3a7d8e22c75c2718233372d7b"}
Sep 11 14:17:35.104876 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:35.104351    2581 scope.go:117] "RemoveContainer" containerID="919cd3bca193dcbcf3fe56cfe3f07d5e8beca372bad789a238a8350abd2a97cf"
Sep 11 14:17:35.113305 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:35.113287    2581 scope.go:117] "RemoveContainer" containerID="919cd3bca193dcbcf3fe56cfe3f07d5e8beca372bad789a238a8350abd2a97cf"
Sep 11 14:17:35.113581 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:17:35.113560    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"919cd3bca193dcbcf3fe56cfe3f07d5e8beca372bad789a238a8350abd2a97cf\": container with ID starting with 919cd3bca193dcbcf3fe56cfe3f07d5e8beca372bad789a238a8350abd2a97cf not found: ID does not exist" containerID="919cd3bca193dcbcf3fe56cfe3f07d5e8beca372bad789a238a8350abd2a97cf"
Sep 11 14:17:35.113628 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:35.113590    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"919cd3bca193dcbcf3fe56cfe3f07d5e8beca372bad789a238a8350abd2a97cf"} err="failed to get container status \"919cd3bca193dcbcf3fe56cfe3f07d5e8beca372bad789a238a8350abd2a97cf\": rpc error: code = NotFound desc = could not find container \"919cd3bca193dcbcf3fe56cfe3f07d5e8beca372bad789a238a8350abd2a97cf\": container with ID starting with 919cd3bca193dcbcf3fe56cfe3f07d5e8beca372bad789a238a8350abd2a97cf not found: ID does not exist"
Sep 11 14:17:35.129656 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:35.129625    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-fbb68bfc-66pmz"]
Sep 11 14:17:35.132612 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:35.132583    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-fbb68bfc-66pmz"]
Sep 11 14:17:35.498251 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:35.498219    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5115e3f0-399e-4db4-a892-ba00f8d8735f" path="/var/lib/kubelet/pods/5115e3f0-399e-4db4-a892-ba00f8d8735f/volumes"
Sep 11 14:17:40.844316 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:40.844285    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7b8fc55444-8rwqg"
Sep 11 14:17:40.848088 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:40.848067    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7b8fc55444-8rwqg"
Sep 11 14:17:53.164924 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:53.164835    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6dcff7596-2d4qc" podUID="aaf265ac-e4ca-4242-8d94-82541dbb9938" containerName="console" containerID="cri-o://8d6be4a8e3d1e106946f3c09e196a6b046f5156bc02f1dec3af46c6143e47ffa" gracePeriod=15
Sep 11 14:17:53.400665 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:53.400640    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dcff7596-2d4qc"
Sep 11 14:17:53.475120 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:53.475034    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aaf265ac-e4ca-4242-8d94-82541dbb9938-console-serving-cert\") pod \"aaf265ac-e4ca-4242-8d94-82541dbb9938\" (UID: \"aaf265ac-e4ca-4242-8d94-82541dbb9938\") "
Sep 11 14:17:53.475120 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:53.475081    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gw8h\" (UniqueName: \"kubernetes.io/projected/aaf265ac-e4ca-4242-8d94-82541dbb9938-kube-api-access-7gw8h\") pod \"aaf265ac-e4ca-4242-8d94-82541dbb9938\" (UID: \"aaf265ac-e4ca-4242-8d94-82541dbb9938\") "
Sep 11 14:17:53.475335 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:53.475122    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aaf265ac-e4ca-4242-8d94-82541dbb9938-console-oauth-config\") pod \"aaf265ac-e4ca-4242-8d94-82541dbb9938\" (UID: \"aaf265ac-e4ca-4242-8d94-82541dbb9938\") "
Sep 11 14:17:53.475335 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:53.475153    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aaf265ac-e4ca-4242-8d94-82541dbb9938-oauth-serving-cert\") pod \"aaf265ac-e4ca-4242-8d94-82541dbb9938\" (UID: \"aaf265ac-e4ca-4242-8d94-82541dbb9938\") "
Sep 11 14:17:53.475335 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:53.475182    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aaf265ac-e4ca-4242-8d94-82541dbb9938-service-ca\") pod \"aaf265ac-e4ca-4242-8d94-82541dbb9938\" (UID: \"aaf265ac-e4ca-4242-8d94-82541dbb9938\") "
Sep 11 14:17:53.475335 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:53.475214    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aaf265ac-e4ca-4242-8d94-82541dbb9938-console-config\") pod \"aaf265ac-e4ca-4242-8d94-82541dbb9938\" (UID: \"aaf265ac-e4ca-4242-8d94-82541dbb9938\") "
Sep 11 14:17:53.475605 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:53.475579    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaf265ac-e4ca-4242-8d94-82541dbb9938-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "aaf265ac-e4ca-4242-8d94-82541dbb9938" (UID: "aaf265ac-e4ca-4242-8d94-82541dbb9938"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:17:53.475669 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:53.475597    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaf265ac-e4ca-4242-8d94-82541dbb9938-service-ca" (OuterVolumeSpecName: "service-ca") pod "aaf265ac-e4ca-4242-8d94-82541dbb9938" (UID: "aaf265ac-e4ca-4242-8d94-82541dbb9938"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:17:53.475725 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:53.475674    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaf265ac-e4ca-4242-8d94-82541dbb9938-console-config" (OuterVolumeSpecName: "console-config") pod "aaf265ac-e4ca-4242-8d94-82541dbb9938" (UID: "aaf265ac-e4ca-4242-8d94-82541dbb9938"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:17:53.477422 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:53.477395    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaf265ac-e4ca-4242-8d94-82541dbb9938-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "aaf265ac-e4ca-4242-8d94-82541dbb9938" (UID: "aaf265ac-e4ca-4242-8d94-82541dbb9938"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:17:53.477534 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:53.477449    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaf265ac-e4ca-4242-8d94-82541dbb9938-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "aaf265ac-e4ca-4242-8d94-82541dbb9938" (UID: "aaf265ac-e4ca-4242-8d94-82541dbb9938"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:17:53.477534 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:53.477480    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaf265ac-e4ca-4242-8d94-82541dbb9938-kube-api-access-7gw8h" (OuterVolumeSpecName: "kube-api-access-7gw8h") pod "aaf265ac-e4ca-4242-8d94-82541dbb9938" (UID: "aaf265ac-e4ca-4242-8d94-82541dbb9938"). InnerVolumeSpecName "kube-api-access-7gw8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:17:53.576365 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:53.576330    2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aaf265ac-e4ca-4242-8d94-82541dbb9938-console-oauth-config\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:17:53.576365 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:53.576360    2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aaf265ac-e4ca-4242-8d94-82541dbb9938-oauth-serving-cert\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:17:53.576365 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:53.576370    2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aaf265ac-e4ca-4242-8d94-82541dbb9938-service-ca\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:17:53.576590 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:53.576379    2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aaf265ac-e4ca-4242-8d94-82541dbb9938-console-config\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:17:53.576590 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:53.576387    2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aaf265ac-e4ca-4242-8d94-82541dbb9938-console-serving-cert\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:17:53.576590 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:53.576394    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gw8h\" (UniqueName: \"kubernetes.io/projected/aaf265ac-e4ca-4242-8d94-82541dbb9938-kube-api-access-7gw8h\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:17:54.174307 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:54.174266    2581 generic.go:356] "Generic (PLEG): container finished" podID="aaf265ac-e4ca-4242-8d94-82541dbb9938" containerID="8d6be4a8e3d1e106946f3c09e196a6b046f5156bc02f1dec3af46c6143e47ffa" exitCode=0
Sep 11 14:17:54.174764 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:54.174317    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dcff7596-2d4qc"
Sep 11 14:17:54.174764 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:54.174352    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dcff7596-2d4qc" event={"ID":"aaf265ac-e4ca-4242-8d94-82541dbb9938","Type":"ContainerDied","Data":"8d6be4a8e3d1e106946f3c09e196a6b046f5156bc02f1dec3af46c6143e47ffa"}
Sep 11 14:17:54.174764 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:54.174409    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dcff7596-2d4qc" event={"ID":"aaf265ac-e4ca-4242-8d94-82541dbb9938","Type":"ContainerDied","Data":"dfdb35f9a8f257e1a018b2fa5fd3d4672d6ce4416fd05006074a6f1b44a61722"}
Sep 11 14:17:54.174764 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:54.174433    2581 scope.go:117] "RemoveContainer" containerID="8d6be4a8e3d1e106946f3c09e196a6b046f5156bc02f1dec3af46c6143e47ffa"
Sep 11 14:17:54.182367 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:54.182347    2581 scope.go:117] "RemoveContainer" containerID="8d6be4a8e3d1e106946f3c09e196a6b046f5156bc02f1dec3af46c6143e47ffa"
Sep 11 14:17:54.182636 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:17:54.182616    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d6be4a8e3d1e106946f3c09e196a6b046f5156bc02f1dec3af46c6143e47ffa\": container with ID starting with 8d6be4a8e3d1e106946f3c09e196a6b046f5156bc02f1dec3af46c6143e47ffa not found: ID does not exist" containerID="8d6be4a8e3d1e106946f3c09e196a6b046f5156bc02f1dec3af46c6143e47ffa"
Sep 11 14:17:54.182711 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:54.182642    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d6be4a8e3d1e106946f3c09e196a6b046f5156bc02f1dec3af46c6143e47ffa"} err="failed to get container status \"8d6be4a8e3d1e106946f3c09e196a6b046f5156bc02f1dec3af46c6143e47ffa\": rpc error: code = NotFound desc = could not find container \"8d6be4a8e3d1e106946f3c09e196a6b046f5156bc02f1dec3af46c6143e47ffa\": container with ID starting with 8d6be4a8e3d1e106946f3c09e196a6b046f5156bc02f1dec3af46c6143e47ffa not found: ID does not exist"
Sep 11 14:17:54.203127 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:54.203096    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6dcff7596-2d4qc"]
Sep 11 14:17:54.215074 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:54.215046    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6dcff7596-2d4qc"]
Sep 11 14:17:55.497788 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:17:55.497750    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aaf265ac-e4ca-4242-8d94-82541dbb9938" path="/var/lib/kubelet/pods/aaf265ac-e4ca-4242-8d94-82541dbb9938/volumes"
Sep 11 14:18:23.497237 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:18:23.497209    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5979f7b764-9gqmk"]
Sep 11 14:18:23.497704 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:18:23.497444    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5115e3f0-399e-4db4-a892-ba00f8d8735f" containerName="registry"
Sep 11 14:18:23.497704 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:18:23.497457    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5115e3f0-399e-4db4-a892-ba00f8d8735f" containerName="registry"
Sep 11 14:18:23.497704 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:18:23.497468    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aaf265ac-e4ca-4242-8d94-82541dbb9938" containerName="console"
Sep 11 14:18:23.497704 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:18:23.497473    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="aaf265ac-e4ca-4242-8d94-82541dbb9938" containerName="console"
Sep 11 14:18:23.497704 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:18:23.497518    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5115e3f0-399e-4db4-a892-ba00f8d8735f" containerName="registry"
Sep 11 14:18:23.497704 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:18:23.497525    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="aaf265ac-e4ca-4242-8d94-82541dbb9938" containerName="console"
Sep 11 14:18:23.500271 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:18:23.500255    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5979f7b764-9gqmk"
Sep 11 14:18:23.504136 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:18:23.504115    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 11 14:18:23.505561 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:18:23.505539    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 11 14:18:23.505848 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:18:23.505831    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 11 14:18:23.506367 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:18:23.506348    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 11 14:18:23.506455 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:18:23.506365    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 11 14:18:23.507192 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:18:23.507172    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-llnzd\""
Sep 11 14:18:23.507564 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:18:23.507542    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 11 14:18:23.507664 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:18:23.507546    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 11 14:18:23.511540 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:18:23.511521    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 11 14:18:23.515376 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:18:23.515355    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5979f7b764-9gqmk"]
Sep 11 14:18:23.597416 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:18:23.597375    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5c790b9e-f83a-4d04-9c6c-b35e8c224c8a-console-oauth-config\") pod \"console-5979f7b764-9gqmk\" (UID: \"5c790b9e-f83a-4d04-9c6c-b35e8c224c8a\") " pod="openshift-console/console-5979f7b764-9gqmk"
Sep 11 14:18:23.597416 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:18:23.597417    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5c790b9e-f83a-4d04-9c6c-b35e8c224c8a-console-config\") pod \"console-5979f7b764-9gqmk\" (UID: \"5c790b9e-f83a-4d04-9c6c-b35e8c224c8a\") " pod="openshift-console/console-5979f7b764-9gqmk"
Sep 11 14:18:23.597627 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:18:23.597481    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5c790b9e-f83a-4d04-9c6c-b35e8c224c8a-service-ca\") pod \"console-5979f7b764-9gqmk\" (UID: \"5c790b9e-f83a-4d04-9c6c-b35e8c224c8a\") " pod="openshift-console/console-5979f7b764-9gqmk"
Sep 11 14:18:23.597627 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:18:23.597526    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5c790b9e-f83a-4d04-9c6c-b35e8c224c8a-console-serving-cert\") pod \"console-5979f7b764-9gqmk\" (UID: \"5c790b9e-f83a-4d04-9c6c-b35e8c224c8a\") " pod="openshift-console/console-5979f7b764-9gqmk"
Sep 11 14:18:23.597627 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:18:23.597547    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p4gh\" (UniqueName: \"kubernetes.io/projected/5c790b9e-f83a-4d04-9c6c-b35e8c224c8a-kube-api-access-8p4gh\") pod \"console-5979f7b764-9gqmk\" (UID: \"5c790b9e-f83a-4d04-9c6c-b35e8c224c8a\") " pod="openshift-console/console-5979f7b764-9gqmk"
Sep 11 14:18:23.597723 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:18:23.597636    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c790b9e-f83a-4d04-9c6c-b35e8c224c8a-trusted-ca-bundle\") pod \"console-5979f7b764-9gqmk\" (UID: \"5c790b9e-f83a-4d04-9c6c-b35e8c224c8a\") " pod="openshift-console/console-5979f7b764-9gqmk"
Sep 11 14:18:23.597723 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:18:23.597665    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5c790b9e-f83a-4d04-9c6c-b35e8c224c8a-oauth-serving-cert\") pod \"console-5979f7b764-9gqmk\" (UID: \"5c790b9e-f83a-4d04-9c6c-b35e8c224c8a\") " pod="openshift-console/console-5979f7b764-9gqmk"
Sep 11 14:18:23.698313 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:18:23.698267    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c790b9e-f83a-4d04-9c6c-b35e8c224c8a-trusted-ca-bundle\") pod \"console-5979f7b764-9gqmk\" (UID: \"5c790b9e-f83a-4d04-9c6c-b35e8c224c8a\") " pod="openshift-console/console-5979f7b764-9gqmk"
Sep 11 14:18:23.698313 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:18:23.698316    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5c790b9e-f83a-4d04-9c6c-b35e8c224c8a-oauth-serving-cert\") pod \"console-5979f7b764-9gqmk\" (UID: \"5c790b9e-f83a-4d04-9c6c-b35e8c224c8a\") " pod="openshift-console/console-5979f7b764-9gqmk"
Sep 11 14:18:23.698561 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:18:23.698344    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5c790b9e-f83a-4d04-9c6c-b35e8c224c8a-console-oauth-config\") pod \"console-5979f7b764-9gqmk\" (UID: \"5c790b9e-f83a-4d04-9c6c-b35e8c224c8a\") " pod="openshift-console/console-5979f7b764-9gqmk"
Sep 11 14:18:23.698561 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:18:23.698367    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5c790b9e-f83a-4d04-9c6c-b35e8c224c8a-console-config\") pod \"console-5979f7b764-9gqmk\" (UID: \"5c790b9e-f83a-4d04-9c6c-b35e8c224c8a\") " pod="openshift-console/console-5979f7b764-9gqmk"
Sep 11 14:18:23.698561 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:18:23.698406    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5c790b9e-f83a-4d04-9c6c-b35e8c224c8a-service-ca\") pod \"console-5979f7b764-9gqmk\" (UID: \"5c790b9e-f83a-4d04-9c6c-b35e8c224c8a\") " pod="openshift-console/console-5979f7b764-9gqmk"
Sep 11 14:18:23.698561 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:18:23.698446    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5c790b9e-f83a-4d04-9c6c-b35e8c224c8a-console-serving-cert\") pod \"console-5979f7b764-9gqmk\" (UID: \"5c790b9e-f83a-4d04-9c6c-b35e8c224c8a\") " pod="openshift-console/console-5979f7b764-9gqmk"
Sep 11 14:18:23.698561 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:18:23.698475    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p4gh\" (UniqueName: \"kubernetes.io/projected/5c790b9e-f83a-4d04-9c6c-b35e8c224c8a-kube-api-access-8p4gh\") pod \"console-5979f7b764-9gqmk\" (UID: \"5c790b9e-f83a-4d04-9c6c-b35e8c224c8a\") " pod="openshift-console/console-5979f7b764-9gqmk"
Sep 11 14:18:23.699200 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:18:23.699167    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5c790b9e-f83a-4d04-9c6c-b35e8c224c8a-service-ca\") pod \"console-5979f7b764-9gqmk\" (UID: \"5c790b9e-f83a-4d04-9c6c-b35e8c224c8a\") " pod="openshift-console/console-5979f7b764-9gqmk"
Sep 11 14:18:23.699354 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:18:23.699176    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5c790b9e-f83a-4d04-9c6c-b35e8c224c8a-oauth-serving-cert\") pod \"console-5979f7b764-9gqmk\" (UID: \"5c790b9e-f83a-4d04-9c6c-b35e8c224c8a\") " pod="openshift-console/console-5979f7b764-9gqmk"
Sep 11 14:18:23.699354 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:18:23.699262    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5c790b9e-f83a-4d04-9c6c-b35e8c224c8a-console-config\") pod \"console-5979f7b764-9gqmk\" (UID: \"5c790b9e-f83a-4d04-9c6c-b35e8c224c8a\") " pod="openshift-console/console-5979f7b764-9gqmk"
Sep 11 14:18:23.699456 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:18:23.699358    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c790b9e-f83a-4d04-9c6c-b35e8c224c8a-trusted-ca-bundle\") pod \"console-5979f7b764-9gqmk\" (UID: \"5c790b9e-f83a-4d04-9c6c-b35e8c224c8a\") " pod="openshift-console/console-5979f7b764-9gqmk"
Sep 11 14:18:23.700891 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:18:23.700865    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5c790b9e-f83a-4d04-9c6c-b35e8c224c8a-console-serving-cert\") pod \"console-5979f7b764-9gqmk\" (UID: \"5c790b9e-f83a-4d04-9c6c-b35e8c224c8a\") " pod="openshift-console/console-5979f7b764-9gqmk"
Sep 11 14:18:23.700891 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:18:23.700886    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5c790b9e-f83a-4d04-9c6c-b35e8c224c8a-console-oauth-config\") pod \"console-5979f7b764-9gqmk\" (UID: \"5c790b9e-f83a-4d04-9c6c-b35e8c224c8a\") " pod="openshift-console/console-5979f7b764-9gqmk"
Sep 11 14:18:23.713253 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:18:23.713227    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p4gh\" (UniqueName: \"kubernetes.io/projected/5c790b9e-f83a-4d04-9c6c-b35e8c224c8a-kube-api-access-8p4gh\") pod \"console-5979f7b764-9gqmk\" (UID: \"5c790b9e-f83a-4d04-9c6c-b35e8c224c8a\") " pod="openshift-console/console-5979f7b764-9gqmk"
Sep 11 14:18:23.810002 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:18:23.809890    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5979f7b764-9gqmk"
Sep 11 14:18:24.107994 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:18:24.107903    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5979f7b764-9gqmk"]
Sep 11 14:18:24.111384 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:18:24.111355    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c790b9e_f83a_4d04_9c6c_b35e8c224c8a.slice/crio-cf973f12b02112582a422b1872ed4053267cc44bf5bf21dbba9bdb33cc807871 WatchSource:0}: Error finding container cf973f12b02112582a422b1872ed4053267cc44bf5bf21dbba9bdb33cc807871: Status 404 returned error can't find the container with id cf973f12b02112582a422b1872ed4053267cc44bf5bf21dbba9bdb33cc807871
Sep 11 14:18:24.278867 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:18:24.278830    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5979f7b764-9gqmk" event={"ID":"5c790b9e-f83a-4d04-9c6c-b35e8c224c8a","Type":"ContainerStarted","Data":"59f24d71e99de823ec0b3f32b0f3a316e3080aa3e2676e889603ae3021c8ed6a"}
Sep 11 14:18:24.279043 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:18:24.278886    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5979f7b764-9gqmk" event={"ID":"5c790b9e-f83a-4d04-9c6c-b35e8c224c8a","Type":"ContainerStarted","Data":"cf973f12b02112582a422b1872ed4053267cc44bf5bf21dbba9bdb33cc807871"}
Sep 11 14:18:24.305203 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:18:24.305155    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5979f7b764-9gqmk" podStartSLOduration=1.3051375680000001 podStartE2EDuration="1.305137568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:18:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:18:24.304458093 +0000 UTC m=+207.455008159" watchObservedRunningTime="2026-09-11 14:18:24.305137568 +0000 UTC m=+207.455687632"
Sep 11 14:18:33.810666 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:18:33.810627    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5979f7b764-9gqmk"
Sep 11 14:18:33.811210 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:18:33.810703    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5979f7b764-9gqmk"
Sep 11 14:18:33.815617 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:18:33.815593    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5979f7b764-9gqmk"
Sep 11 14:18:34.318282 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:18:34.318253    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5979f7b764-9gqmk"
Sep 11 14:19:33.213356 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:19:33.213274    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-65c7b69c6c-nlhdj"]
Sep 11 14:19:33.216351 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:19:33.216333    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65c7b69c6c-nlhdj"
Sep 11 14:19:33.246828 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:19:33.246794    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65c7b69c6c-nlhdj"]
Sep 11 14:19:33.316809 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:19:33.316779    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/af59576c-6739-4472-89e1-796775e0aa3a-console-config\") pod \"console-65c7b69c6c-nlhdj\" (UID: \"af59576c-6739-4472-89e1-796775e0aa3a\") " pod="openshift-console/console-65c7b69c6c-nlhdj"
Sep 11 14:19:33.317010 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:19:33.316817    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/af59576c-6739-4472-89e1-796775e0aa3a-console-oauth-config\") pod \"console-65c7b69c6c-nlhdj\" (UID: \"af59576c-6739-4472-89e1-796775e0aa3a\") " pod="openshift-console/console-65c7b69c6c-nlhdj"
Sep 11 14:19:33.317010 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:19:33.316835    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/af59576c-6739-4472-89e1-796775e0aa3a-oauth-serving-cert\") pod \"console-65c7b69c6c-nlhdj\" (UID: \"af59576c-6739-4472-89e1-796775e0aa3a\") " pod="openshift-console/console-65c7b69c6c-nlhdj"
Sep 11 14:19:33.317010 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:19:33.316858    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98fbt\" (UniqueName: \"kubernetes.io/projected/af59576c-6739-4472-89e1-796775e0aa3a-kube-api-access-98fbt\") pod \"console-65c7b69c6c-nlhdj\" (UID: \"af59576c-6739-4472-89e1-796775e0aa3a\") " pod="openshift-console/console-65c7b69c6c-nlhdj"
Sep 11 14:19:33.317010 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:19:33.316946    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/af59576c-6739-4472-89e1-796775e0aa3a-service-ca\") pod \"console-65c7b69c6c-nlhdj\" (UID: \"af59576c-6739-4472-89e1-796775e0aa3a\") " pod="openshift-console/console-65c7b69c6c-nlhdj"
Sep 11 14:19:33.317158 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:19:33.317006    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/af59576c-6739-4472-89e1-796775e0aa3a-console-serving-cert\") pod \"console-65c7b69c6c-nlhdj\" (UID: \"af59576c-6739-4472-89e1-796775e0aa3a\") " pod="openshift-console/console-65c7b69c6c-nlhdj"
Sep 11 14:19:33.317158 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:19:33.317029    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af59576c-6739-4472-89e1-796775e0aa3a-trusted-ca-bundle\") pod \"console-65c7b69c6c-nlhdj\" (UID: \"af59576c-6739-4472-89e1-796775e0aa3a\") " pod="openshift-console/console-65c7b69c6c-nlhdj"
Sep 11 14:19:33.418029 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:19:33.417962    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98fbt\" (UniqueName: \"kubernetes.io/projected/af59576c-6739-4472-89e1-796775e0aa3a-kube-api-access-98fbt\") pod \"console-65c7b69c6c-nlhdj\" (UID: \"af59576c-6739-4472-89e1-796775e0aa3a\") " pod="openshift-console/console-65c7b69c6c-nlhdj"
Sep 11 14:19:33.418029 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:19:33.418032    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/af59576c-6739-4472-89e1-796775e0aa3a-service-ca\") pod \"console-65c7b69c6c-nlhdj\" (UID: \"af59576c-6739-4472-89e1-796775e0aa3a\") " pod="openshift-console/console-65c7b69c6c-nlhdj"
Sep 11 14:19:33.418237 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:19:33.418059    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/af59576c-6739-4472-89e1-796775e0aa3a-console-serving-cert\") pod \"console-65c7b69c6c-nlhdj\" (UID: \"af59576c-6739-4472-89e1-796775e0aa3a\") " pod="openshift-console/console-65c7b69c6c-nlhdj"
Sep 11 14:19:33.418237 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:19:33.418078    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af59576c-6739-4472-89e1-796775e0aa3a-trusted-ca-bundle\") pod \"console-65c7b69c6c-nlhdj\" (UID: \"af59576c-6739-4472-89e1-796775e0aa3a\") " pod="openshift-console/console-65c7b69c6c-nlhdj"
Sep 11 14:19:33.418237 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:19:33.418134    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/af59576c-6739-4472-89e1-796775e0aa3a-console-config\") pod \"console-65c7b69c6c-nlhdj\" (UID: \"af59576c-6739-4472-89e1-796775e0aa3a\") " pod="openshift-console/console-65c7b69c6c-nlhdj"
Sep 11 14:19:33.418237 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:19:33.418164    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/af59576c-6739-4472-89e1-796775e0aa3a-console-oauth-config\") pod \"console-65c7b69c6c-nlhdj\" (UID: \"af59576c-6739-4472-89e1-796775e0aa3a\") " pod="openshift-console/console-65c7b69c6c-nlhdj"
Sep 11 14:19:33.418237 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:19:33.418186    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/af59576c-6739-4472-89e1-796775e0aa3a-oauth-serving-cert\") pod \"console-65c7b69c6c-nlhdj\" (UID: \"af59576c-6739-4472-89e1-796775e0aa3a\") " pod="openshift-console/console-65c7b69c6c-nlhdj"
Sep 11 14:19:33.418868 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:19:33.418841    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/af59576c-6739-4472-89e1-796775e0aa3a-service-ca\") pod \"console-65c7b69c6c-nlhdj\" (UID: \"af59576c-6739-4472-89e1-796775e0aa3a\") " pod="openshift-console/console-65c7b69c6c-nlhdj"
Sep 11 14:19:33.418868 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:19:33.418859    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/af59576c-6739-4472-89e1-796775e0aa3a-console-config\") pod \"console-65c7b69c6c-nlhdj\" (UID: \"af59576c-6739-4472-89e1-796775e0aa3a\") " pod="openshift-console/console-65c7b69c6c-nlhdj"
Sep 11 14:19:33.419051 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:19:33.419005    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/af59576c-6739-4472-89e1-796775e0aa3a-oauth-serving-cert\") pod \"console-65c7b69c6c-nlhdj\" (UID: \"af59576c-6739-4472-89e1-796775e0aa3a\") " pod="openshift-console/console-65c7b69c6c-nlhdj"
Sep 11 14:19:33.419234 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:19:33.419213    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af59576c-6739-4472-89e1-796775e0aa3a-trusted-ca-bundle\") pod \"console-65c7b69c6c-nlhdj\" (UID: \"af59576c-6739-4472-89e1-796775e0aa3a\") " pod="openshift-console/console-65c7b69c6c-nlhdj"
Sep 11 14:19:33.420660 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:19:33.420638    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/af59576c-6739-4472-89e1-796775e0aa3a-console-serving-cert\") pod \"console-65c7b69c6c-nlhdj\" (UID: \"af59576c-6739-4472-89e1-796775e0aa3a\") " pod="openshift-console/console-65c7b69c6c-nlhdj"
Sep 11 14:19:33.420752 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:19:33.420673    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/af59576c-6739-4472-89e1-796775e0aa3a-console-oauth-config\") pod \"console-65c7b69c6c-nlhdj\" (UID: \"af59576c-6739-4472-89e1-796775e0aa3a\") " pod="openshift-console/console-65c7b69c6c-nlhdj"
Sep 11 14:19:33.440072 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:19:33.440052    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98fbt\" (UniqueName: \"kubernetes.io/projected/af59576c-6739-4472-89e1-796775e0aa3a-kube-api-access-98fbt\") pod \"console-65c7b69c6c-nlhdj\" (UID: \"af59576c-6739-4472-89e1-796775e0aa3a\") " pod="openshift-console/console-65c7b69c6c-nlhdj"
Sep 11 14:19:33.525146 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:19:33.525110    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65c7b69c6c-nlhdj"
Sep 11 14:19:33.805388 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:19:33.805330    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65c7b69c6c-nlhdj"]
Sep 11 14:19:33.807478 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:19:33.807449    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf59576c_6739_4472_89e1_796775e0aa3a.slice/crio-dc7b1b61a692f93896274f6d1795e0bc3410dc6897d6a8021e3ea003ad0fb37e WatchSource:0}: Error finding container dc7b1b61a692f93896274f6d1795e0bc3410dc6897d6a8021e3ea003ad0fb37e: Status 404 returned error can't find the container with id dc7b1b61a692f93896274f6d1795e0bc3410dc6897d6a8021e3ea003ad0fb37e
Sep 11 14:19:34.520461 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:19:34.520424    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65c7b69c6c-nlhdj" event={"ID":"af59576c-6739-4472-89e1-796775e0aa3a","Type":"ContainerStarted","Data":"f6aae31e758caef7472f056f3594fc62b10991a83e8c8a8dd51ca5ad05055b92"}
Sep 11 14:19:34.520833 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:19:34.520475    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65c7b69c6c-nlhdj" event={"ID":"af59576c-6739-4472-89e1-796775e0aa3a","Type":"ContainerStarted","Data":"dc7b1b61a692f93896274f6d1795e0bc3410dc6897d6a8021e3ea003ad0fb37e"}
Sep 11 14:19:34.553633 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:19:34.553581    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-65c7b69c6c-nlhdj" podStartSLOduration=1.5535633770000001 podStartE2EDuration="1.553563377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:19:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:19:34.551997746 +0000 UTC m=+277.702547810" watchObservedRunningTime="2026-09-11 14:19:34.553563377 +0000 UTC m=+277.704113440"
Sep 11 14:19:43.525667 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:19:43.525631    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-65c7b69c6c-nlhdj"
Sep 11 14:19:43.526268 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:19:43.525695    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-65c7b69c6c-nlhdj"
Sep 11 14:19:43.530424 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:19:43.530401    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-65c7b69c6c-nlhdj"
Sep 11 14:19:43.554505 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:19:43.554479    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-65c7b69c6c-nlhdj"
Sep 11 14:19:43.663707 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:19:43.663673    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5979f7b764-9gqmk"]
Sep 11 14:19:57.266472 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:19:57.266443    2581 kubelet.go:1648] "Image garbage collection succeeded"
Sep 11 14:20:08.682829 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:20:08.682781    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5979f7b764-9gqmk" podUID="5c790b9e-f83a-4d04-9c6c-b35e8c224c8a" containerName="console" containerID="cri-o://59f24d71e99de823ec0b3f32b0f3a316e3080aa3e2676e889603ae3021c8ed6a" gracePeriod=15
Sep 11 14:20:08.919270 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:20:08.919245    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5979f7b764-9gqmk"
Sep 11 14:20:08.992399 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:20:08.992367    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8p4gh\" (UniqueName: \"kubernetes.io/projected/5c790b9e-f83a-4d04-9c6c-b35e8c224c8a-kube-api-access-8p4gh\") pod \"5c790b9e-f83a-4d04-9c6c-b35e8c224c8a\" (UID: \"5c790b9e-f83a-4d04-9c6c-b35e8c224c8a\") "
Sep 11 14:20:08.992583 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:20:08.992407    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5c790b9e-f83a-4d04-9c6c-b35e8c224c8a-console-serving-cert\") pod \"5c790b9e-f83a-4d04-9c6c-b35e8c224c8a\" (UID: \"5c790b9e-f83a-4d04-9c6c-b35e8c224c8a\") "
Sep 11 14:20:08.992583 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:20:08.992446    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5c790b9e-f83a-4d04-9c6c-b35e8c224c8a-console-oauth-config\") pod \"5c790b9e-f83a-4d04-9c6c-b35e8c224c8a\" (UID: \"5c790b9e-f83a-4d04-9c6c-b35e8c224c8a\") "
Sep 11 14:20:08.992583 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:20:08.992474    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5c790b9e-f83a-4d04-9c6c-b35e8c224c8a-service-ca\") pod \"5c790b9e-f83a-4d04-9c6c-b35e8c224c8a\" (UID: \"5c790b9e-f83a-4d04-9c6c-b35e8c224c8a\") "
Sep 11 14:20:08.992583 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:20:08.992542    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5c790b9e-f83a-4d04-9c6c-b35e8c224c8a-console-config\") pod \"5c790b9e-f83a-4d04-9c6c-b35e8c224c8a\" (UID: \"5c790b9e-f83a-4d04-9c6c-b35e8c224c8a\") "
Sep 11 14:20:08.992583 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:20:08.992564    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5c790b9e-f83a-4d04-9c6c-b35e8c224c8a-oauth-serving-cert\") pod \"5c790b9e-f83a-4d04-9c6c-b35e8c224c8a\" (UID: \"5c790b9e-f83a-4d04-9c6c-b35e8c224c8a\") "
Sep 11 14:20:08.992816 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:20:08.992588    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c790b9e-f83a-4d04-9c6c-b35e8c224c8a-trusted-ca-bundle\") pod \"5c790b9e-f83a-4d04-9c6c-b35e8c224c8a\" (UID: \"5c790b9e-f83a-4d04-9c6c-b35e8c224c8a\") "
Sep 11 14:20:08.993095 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:20:08.993056    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c790b9e-f83a-4d04-9c6c-b35e8c224c8a-console-config" (OuterVolumeSpecName: "console-config") pod "5c790b9e-f83a-4d04-9c6c-b35e8c224c8a" (UID: "5c790b9e-f83a-4d04-9c6c-b35e8c224c8a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:20:08.993095 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:20:08.993069    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c790b9e-f83a-4d04-9c6c-b35e8c224c8a-service-ca" (OuterVolumeSpecName: "service-ca") pod "5c790b9e-f83a-4d04-9c6c-b35e8c224c8a" (UID: "5c790b9e-f83a-4d04-9c6c-b35e8c224c8a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:20:08.993287 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:20:08.993085    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c790b9e-f83a-4d04-9c6c-b35e8c224c8a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5c790b9e-f83a-4d04-9c6c-b35e8c224c8a" (UID: "5c790b9e-f83a-4d04-9c6c-b35e8c224c8a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:20:08.993287 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:20:08.993215    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c790b9e-f83a-4d04-9c6c-b35e8c224c8a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "5c790b9e-f83a-4d04-9c6c-b35e8c224c8a" (UID: "5c790b9e-f83a-4d04-9c6c-b35e8c224c8a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:20:08.994686 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:20:08.994660    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c790b9e-f83a-4d04-9c6c-b35e8c224c8a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5c790b9e-f83a-4d04-9c6c-b35e8c224c8a" (UID: "5c790b9e-f83a-4d04-9c6c-b35e8c224c8a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:20:08.994764 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:20:08.994705    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c790b9e-f83a-4d04-9c6c-b35e8c224c8a-kube-api-access-8p4gh" (OuterVolumeSpecName: "kube-api-access-8p4gh") pod "5c790b9e-f83a-4d04-9c6c-b35e8c224c8a" (UID: "5c790b9e-f83a-4d04-9c6c-b35e8c224c8a"). InnerVolumeSpecName "kube-api-access-8p4gh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:20:08.994802 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:20:08.994748    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c790b9e-f83a-4d04-9c6c-b35e8c224c8a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5c790b9e-f83a-4d04-9c6c-b35e8c224c8a" (UID: "5c790b9e-f83a-4d04-9c6c-b35e8c224c8a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:20:09.093150 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:20:09.093120    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p4gh\" (UniqueName: \"kubernetes.io/projected/5c790b9e-f83a-4d04-9c6c-b35e8c224c8a-kube-api-access-8p4gh\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:20:09.093150 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:20:09.093146    2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5c790b9e-f83a-4d04-9c6c-b35e8c224c8a-console-serving-cert\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:20:09.093150 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:20:09.093157    2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5c790b9e-f83a-4d04-9c6c-b35e8c224c8a-console-oauth-config\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:20:09.093379 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:20:09.093165    2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5c790b9e-f83a-4d04-9c6c-b35e8c224c8a-service-ca\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:20:09.093379 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:20:09.093176    2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5c790b9e-f83a-4d04-9c6c-b35e8c224c8a-console-config\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:20:09.093379 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:20:09.093188    2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5c790b9e-f83a-4d04-9c6c-b35e8c224c8a-oauth-serving-cert\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:20:09.093379 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:20:09.093197    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c790b9e-f83a-4d04-9c6c-b35e8c224c8a-trusted-ca-bundle\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:20:09.639953 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:20:09.639915    2581 generic.go:356] "Generic (PLEG): container finished" podID="5c790b9e-f83a-4d04-9c6c-b35e8c224c8a" containerID="59f24d71e99de823ec0b3f32b0f3a316e3080aa3e2676e889603ae3021c8ed6a" exitCode=0
Sep 11 14:20:09.639953 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:20:09.639959    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5979f7b764-9gqmk"
Sep 11 14:20:09.640206 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:20:09.640005    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5979f7b764-9gqmk" event={"ID":"5c790b9e-f83a-4d04-9c6c-b35e8c224c8a","Type":"ContainerDied","Data":"59f24d71e99de823ec0b3f32b0f3a316e3080aa3e2676e889603ae3021c8ed6a"}
Sep 11 14:20:09.640206 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:20:09.640057    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5979f7b764-9gqmk" event={"ID":"5c790b9e-f83a-4d04-9c6c-b35e8c224c8a","Type":"ContainerDied","Data":"cf973f12b02112582a422b1872ed4053267cc44bf5bf21dbba9bdb33cc807871"}
Sep 11 14:20:09.640206 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:20:09.640083    2581 scope.go:117] "RemoveContainer" containerID="59f24d71e99de823ec0b3f32b0f3a316e3080aa3e2676e889603ae3021c8ed6a"
Sep 11 14:20:09.648192 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:20:09.648170    2581 scope.go:117] "RemoveContainer" containerID="59f24d71e99de823ec0b3f32b0f3a316e3080aa3e2676e889603ae3021c8ed6a"
Sep 11 14:20:09.648481 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:20:09.648461    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59f24d71e99de823ec0b3f32b0f3a316e3080aa3e2676e889603ae3021c8ed6a\": container with ID starting with 59f24d71e99de823ec0b3f32b0f3a316e3080aa3e2676e889603ae3021c8ed6a not found: ID does not exist" containerID="59f24d71e99de823ec0b3f32b0f3a316e3080aa3e2676e889603ae3021c8ed6a"
Sep 11 14:20:09.648549 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:20:09.648496    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59f24d71e99de823ec0b3f32b0f3a316e3080aa3e2676e889603ae3021c8ed6a"} err="failed to get container status \"59f24d71e99de823ec0b3f32b0f3a316e3080aa3e2676e889603ae3021c8ed6a\": rpc error: code = NotFound desc = could not find container \"59f24d71e99de823ec0b3f32b0f3a316e3080aa3e2676e889603ae3021c8ed6a\": container with ID starting with 59f24d71e99de823ec0b3f32b0f3a316e3080aa3e2676e889603ae3021c8ed6a not found: ID does not exist"
Sep 11 14:20:09.663357 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:20:09.663332    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5979f7b764-9gqmk"]
Sep 11 14:20:09.669639 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:20:09.669617    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5979f7b764-9gqmk"]
Sep 11 14:20:11.497482 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:20:11.497447    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c790b9e-f83a-4d04-9c6c-b35e8c224c8a" path="/var/lib/kubelet/pods/5c790b9e-f83a-4d04-9c6c-b35e8c224c8a/volumes"
Sep 11 14:20:55.969131 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:20:55.969049    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lrt7g"]
Sep 11 14:20:55.969602 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:20:55.969302    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c790b9e-f83a-4d04-9c6c-b35e8c224c8a" containerName="console"
Sep 11 14:20:55.969602 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:20:55.969313    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c790b9e-f83a-4d04-9c6c-b35e8c224c8a" containerName="console"
Sep 11 14:20:55.969602 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:20:55.969355    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c790b9e-f83a-4d04-9c6c-b35e8c224c8a" containerName="console"
Sep 11 14:20:55.972077 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:20:55.972060    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lrt7g"
Sep 11 14:20:55.975116 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:20:55.975095    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 11 14:20:55.975250 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:20:55.975208    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-8j4kg\""
Sep 11 14:20:55.975502 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:20:55.975484    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 11 14:20:56.008685 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:20:56.008651    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lrt7g"]
Sep 11 14:20:56.028503 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:20:56.028465    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4db28b58-2462-408e-85c7-b6818900977c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-lrt7g\" (UID: \"4db28b58-2462-408e-85c7-b6818900977c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lrt7g"
Sep 11 14:20:56.028675 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:20:56.028520    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnwzp\" (UniqueName: \"kubernetes.io/projected/4db28b58-2462-408e-85c7-b6818900977c-kube-api-access-lnwzp\") pod \"cert-manager-operator-controller-manager-55c444bf86-lrt7g\" (UID: \"4db28b58-2462-408e-85c7-b6818900977c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lrt7g"
Sep 11 14:20:56.128952 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:20:56.128909    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnwzp\" (UniqueName: \"kubernetes.io/projected/4db28b58-2462-408e-85c7-b6818900977c-kube-api-access-lnwzp\") pod \"cert-manager-operator-controller-manager-55c444bf86-lrt7g\" (UID: \"4db28b58-2462-408e-85c7-b6818900977c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lrt7g"
Sep 11 14:20:56.129175 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:20:56.129004    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4db28b58-2462-408e-85c7-b6818900977c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-lrt7g\" (UID: \"4db28b58-2462-408e-85c7-b6818900977c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lrt7g"
Sep 11 14:20:56.129357 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:20:56.129339    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4db28b58-2462-408e-85c7-b6818900977c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-lrt7g\" (UID: \"4db28b58-2462-408e-85c7-b6818900977c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lrt7g"
Sep 11 14:20:56.144529 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:20:56.144495    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnwzp\" (UniqueName: \"kubernetes.io/projected/4db28b58-2462-408e-85c7-b6818900977c-kube-api-access-lnwzp\") pod \"cert-manager-operator-controller-manager-55c444bf86-lrt7g\" (UID: \"4db28b58-2462-408e-85c7-b6818900977c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lrt7g"
Sep 11 14:20:56.280591 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:20:56.280558    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lrt7g"
Sep 11 14:20:56.591132 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:20:56.591092    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lrt7g"]
Sep 11 14:20:56.593760 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:20:56.593727    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4db28b58_2462_408e_85c7_b6818900977c.slice/crio-4d0e08b4c067735de75668495360b100187e37eefd4c4c98dfacc774642b685a WatchSource:0}: Error finding container 4d0e08b4c067735de75668495360b100187e37eefd4c4c98dfacc774642b685a: Status 404 returned error can't find the container with id 4d0e08b4c067735de75668495360b100187e37eefd4c4c98dfacc774642b685a
Sep 11 14:20:56.596398 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:20:56.596379    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 14:20:56.805185 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:20:56.805151    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lrt7g" event={"ID":"4db28b58-2462-408e-85c7-b6818900977c","Type":"ContainerStarted","Data":"4d0e08b4c067735de75668495360b100187e37eefd4c4c98dfacc774642b685a"}
Sep 11 14:20:59.817933 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:20:59.817889    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lrt7g" event={"ID":"4db28b58-2462-408e-85c7-b6818900977c","Type":"ContainerStarted","Data":"1d0bf3b9e4ca901840d6d99b7d942a0a0460af4395c1afdb0da47bdb263c7894"}
Sep 11 14:20:59.847261 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:20:59.847196    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lrt7g" podStartSLOduration=2.647374415 podStartE2EDuration="4.847174383s" totalImagesPullingTime="2.199799968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:20:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:20:59.845414424 +0000 UTC m=+362.995964512" watchObservedRunningTime="2026-09-11 14:20:59.847174383 +0000 UTC m=+362.997724448"
Sep 11 14:21:09.926773 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:09.926734    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-vm264"]
Sep 11 14:21:09.930007 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:09.929989    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-vm264"
Sep 11 14:21:09.933084 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:09.933057    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 11 14:21:09.933199 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:09.933084    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-q5g2f\""
Sep 11 14:21:09.933199 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:09.933089    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 11 14:21:09.949417 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:09.949390    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-vm264"]
Sep 11 14:21:10.039134 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:10.039094    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sbbh\" (UniqueName: \"kubernetes.io/projected/7aaeccef-8a46-415b-8592-c14d66acc290-kube-api-access-4sbbh\") pod \"cert-manager-55b4d954d5-vm264\" (UID: \"7aaeccef-8a46-415b-8592-c14d66acc290\") " pod="cert-manager/cert-manager-55b4d954d5-vm264"
Sep 11 14:21:10.039300 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:10.039149    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7aaeccef-8a46-415b-8592-c14d66acc290-bound-sa-token\") pod \"cert-manager-55b4d954d5-vm264\" (UID: \"7aaeccef-8a46-415b-8592-c14d66acc290\") " pod="cert-manager/cert-manager-55b4d954d5-vm264"
Sep 11 14:21:10.139934 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:10.139892    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sbbh\" (UniqueName: \"kubernetes.io/projected/7aaeccef-8a46-415b-8592-c14d66acc290-kube-api-access-4sbbh\") pod \"cert-manager-55b4d954d5-vm264\" (UID: \"7aaeccef-8a46-415b-8592-c14d66acc290\") " pod="cert-manager/cert-manager-55b4d954d5-vm264"
Sep 11 14:21:10.140115 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:10.139943    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7aaeccef-8a46-415b-8592-c14d66acc290-bound-sa-token\") pod \"cert-manager-55b4d954d5-vm264\" (UID: \"7aaeccef-8a46-415b-8592-c14d66acc290\") " pod="cert-manager/cert-manager-55b4d954d5-vm264"
Sep 11 14:21:10.151275 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:10.151242    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7aaeccef-8a46-415b-8592-c14d66acc290-bound-sa-token\") pod \"cert-manager-55b4d954d5-vm264\" (UID: \"7aaeccef-8a46-415b-8592-c14d66acc290\") " pod="cert-manager/cert-manager-55b4d954d5-vm264"
Sep 11 14:21:10.151465 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:10.151424    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sbbh\" (UniqueName: \"kubernetes.io/projected/7aaeccef-8a46-415b-8592-c14d66acc290-kube-api-access-4sbbh\") pod \"cert-manager-55b4d954d5-vm264\" (UID: \"7aaeccef-8a46-415b-8592-c14d66acc290\") " pod="cert-manager/cert-manager-55b4d954d5-vm264"
Sep 11 14:21:10.239395 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:10.239307    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-vm264"
Sep 11 14:21:10.514279 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:10.514245    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-vm264"]
Sep 11 14:21:10.517207 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:21:10.517171    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7aaeccef_8a46_415b_8592_c14d66acc290.slice/crio-2910a0d7ddcc59cdc367873fee8a0663c99b86ade62502403cf05896fd4b0e9c WatchSource:0}: Error finding container 2910a0d7ddcc59cdc367873fee8a0663c99b86ade62502403cf05896fd4b0e9c: Status 404 returned error can't find the container with id 2910a0d7ddcc59cdc367873fee8a0663c99b86ade62502403cf05896fd4b0e9c
Sep 11 14:21:10.858292 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:10.858210    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-vm264" event={"ID":"7aaeccef-8a46-415b-8592-c14d66acc290","Type":"ContainerStarted","Data":"2910a0d7ddcc59cdc367873fee8a0663c99b86ade62502403cf05896fd4b0e9c"}
Sep 11 14:21:12.407561 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:12.407524    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-585fl"]
Sep 11 14:21:12.411037 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:12.411004    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-585fl"
Sep 11 14:21:12.413990 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:12.413920    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 11 14:21:12.415111 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:12.415087    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-kr77x\""
Sep 11 14:21:12.415242 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:12.415087    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 11 14:21:12.423131 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:12.423105    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-585fl"]
Sep 11 14:21:12.561921 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:12.561886    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2332a63b-b2db-48ec-9185-d8bcd25cf754-tmp\") pod \"openshift-lws-operator-fd8ccff4c-585fl\" (UID: \"2332a63b-b2db-48ec-9185-d8bcd25cf754\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-585fl"
Sep 11 14:21:12.561921 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:12.561922    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxt6p\" (UniqueName: \"kubernetes.io/projected/2332a63b-b2db-48ec-9185-d8bcd25cf754-kube-api-access-nxt6p\") pod \"openshift-lws-operator-fd8ccff4c-585fl\" (UID: \"2332a63b-b2db-48ec-9185-d8bcd25cf754\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-585fl"
Sep 11 14:21:12.663135 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:12.663044    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2332a63b-b2db-48ec-9185-d8bcd25cf754-tmp\") pod \"openshift-lws-operator-fd8ccff4c-585fl\" (UID: \"2332a63b-b2db-48ec-9185-d8bcd25cf754\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-585fl"
Sep 11 14:21:12.663135 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:12.663087    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxt6p\" (UniqueName: \"kubernetes.io/projected/2332a63b-b2db-48ec-9185-d8bcd25cf754-kube-api-access-nxt6p\") pod \"openshift-lws-operator-fd8ccff4c-585fl\" (UID: \"2332a63b-b2db-48ec-9185-d8bcd25cf754\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-585fl"
Sep 11 14:21:12.663509 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:12.663484    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2332a63b-b2db-48ec-9185-d8bcd25cf754-tmp\") pod \"openshift-lws-operator-fd8ccff4c-585fl\" (UID: \"2332a63b-b2db-48ec-9185-d8bcd25cf754\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-585fl"
Sep 11 14:21:12.675913 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:12.675879    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxt6p\" (UniqueName: \"kubernetes.io/projected/2332a63b-b2db-48ec-9185-d8bcd25cf754-kube-api-access-nxt6p\") pod \"openshift-lws-operator-fd8ccff4c-585fl\" (UID: \"2332a63b-b2db-48ec-9185-d8bcd25cf754\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-585fl"
Sep 11 14:21:12.722945 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:12.722900    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-585fl"
Sep 11 14:21:13.445620 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:13.445535    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-585fl"]
Sep 11 14:21:13.448638 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:21:13.448609    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2332a63b_b2db_48ec_9185_d8bcd25cf754.slice/crio-4883958a54a31a677dc0322b133b28f121a919b03aa55e9a48c5bf1191818249 WatchSource:0}: Error finding container 4883958a54a31a677dc0322b133b28f121a919b03aa55e9a48c5bf1191818249: Status 404 returned error can't find the container with id 4883958a54a31a677dc0322b133b28f121a919b03aa55e9a48c5bf1191818249
Sep 11 14:21:13.876355 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:13.876308    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-585fl" event={"ID":"2332a63b-b2db-48ec-9185-d8bcd25cf754","Type":"ContainerStarted","Data":"4883958a54a31a677dc0322b133b28f121a919b03aa55e9a48c5bf1191818249"}
Sep 11 14:21:13.877751 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:13.877724    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-vm264" event={"ID":"7aaeccef-8a46-415b-8592-c14d66acc290","Type":"ContainerStarted","Data":"180b6da112ed758507e2c084afd6e18096fd81c6ac7bb5a53108f4146ec4c1c9"}
Sep 11 14:21:13.913852 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:13.913796    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-vm264" podStartSLOduration=2.198275974 podStartE2EDuration="4.913779425s" totalImagesPullingTime="2.715503451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:21:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:21:13.913398824 +0000 UTC m=+377.063948884" watchObservedRunningTime="2026-09-11 14:21:13.913779425 +0000 UTC m=+377.064329488"
Sep 11 14:21:16.892964 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:16.892930    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-585fl" event={"ID":"2332a63b-b2db-48ec-9185-d8bcd25cf754","Type":"ContainerStarted","Data":"1c77cefcb5be60e07739aedcc67e7debd0cfe3f5d6a1ad37f36ff781feb77347"}
Sep 11 14:21:16.916452 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:16.916387    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-585fl" podStartSLOduration=2.383100025 podStartE2EDuration="4.916368353s" totalImagesPullingTime="2.533268328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:21:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:21:16.914667594 +0000 UTC m=+380.065217662" watchObservedRunningTime="2026-09-11 14:21:16.916368353 +0000 UTC m=+380.066918421"
Sep 11 14:21:34.205636 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:34.205602    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-4lss9"]
Sep 11 14:21:34.265616 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:34.265584    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-4lss9"
Sep 11 14:21:34.273694 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:34.273667    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 11 14:21:34.273694 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:34.273689    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-srflq\""
Sep 11 14:21:34.273901 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:34.273720    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 11 14:21:34.273901 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:34.273730    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 11 14:21:34.273901 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:34.273780    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 11 14:21:34.284128 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:34.284101    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-4lss9"]
Sep 11 14:21:34.340299 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:34.340267    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7897ae2e-9365-4a43-95a5-6206a1f8d0e0-tmp\") pod \"opendatahub-operator-controller-manager-7d8f96d84b-4lss9\" (UID: \"7897ae2e-9365-4a43-95a5-6206a1f8d0e0\") " pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-4lss9"
Sep 11 14:21:34.340505 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:34.340318    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q5vv\" (UniqueName: \"kubernetes.io/projected/7897ae2e-9365-4a43-95a5-6206a1f8d0e0-kube-api-access-8q5vv\") pod \"opendatahub-operator-controller-manager-7d8f96d84b-4lss9\" (UID: \"7897ae2e-9365-4a43-95a5-6206a1f8d0e0\") " pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-4lss9"
Sep 11 14:21:34.340505 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:34.340408    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7897ae2e-9365-4a43-95a5-6206a1f8d0e0-manifests\") pod \"opendatahub-operator-controller-manager-7d8f96d84b-4lss9\" (UID: \"7897ae2e-9365-4a43-95a5-6206a1f8d0e0\") " pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-4lss9"
Sep 11 14:21:34.340505 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:34.340479    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7897ae2e-9365-4a43-95a5-6206a1f8d0e0-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d8f96d84b-4lss9\" (UID: \"7897ae2e-9365-4a43-95a5-6206a1f8d0e0\") " pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-4lss9"
Sep 11 14:21:34.340642 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:34.340512    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7897ae2e-9365-4a43-95a5-6206a1f8d0e0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d8f96d84b-4lss9\" (UID: \"7897ae2e-9365-4a43-95a5-6206a1f8d0e0\") " pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-4lss9"
Sep 11 14:21:34.441786 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:34.441744    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7897ae2e-9365-4a43-95a5-6206a1f8d0e0-manifests\") pod \"opendatahub-operator-controller-manager-7d8f96d84b-4lss9\" (UID: \"7897ae2e-9365-4a43-95a5-6206a1f8d0e0\") " pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-4lss9"
Sep 11 14:21:34.442016 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:34.441801    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7897ae2e-9365-4a43-95a5-6206a1f8d0e0-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d8f96d84b-4lss9\" (UID: \"7897ae2e-9365-4a43-95a5-6206a1f8d0e0\") " pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-4lss9"
Sep 11 14:21:34.442016 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:34.441831    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7897ae2e-9365-4a43-95a5-6206a1f8d0e0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d8f96d84b-4lss9\" (UID: \"7897ae2e-9365-4a43-95a5-6206a1f8d0e0\") " pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-4lss9"
Sep 11 14:21:34.442016 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:34.441873    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7897ae2e-9365-4a43-95a5-6206a1f8d0e0-tmp\") pod \"opendatahub-operator-controller-manager-7d8f96d84b-4lss9\" (UID: \"7897ae2e-9365-4a43-95a5-6206a1f8d0e0\") " pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-4lss9"
Sep 11 14:21:34.442016 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:34.441913    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q5vv\" (UniqueName: \"kubernetes.io/projected/7897ae2e-9365-4a43-95a5-6206a1f8d0e0-kube-api-access-8q5vv\") pod \"opendatahub-operator-controller-manager-7d8f96d84b-4lss9\" (UID: \"7897ae2e-9365-4a43-95a5-6206a1f8d0e0\") " pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-4lss9"
Sep 11 14:21:34.442283 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:34.442257    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7897ae2e-9365-4a43-95a5-6206a1f8d0e0-manifests\") pod \"opendatahub-operator-controller-manager-7d8f96d84b-4lss9\" (UID: \"7897ae2e-9365-4a43-95a5-6206a1f8d0e0\") " pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-4lss9"
Sep 11 14:21:34.444663 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:34.444639    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7897ae2e-9365-4a43-95a5-6206a1f8d0e0-tmp\") pod \"opendatahub-operator-controller-manager-7d8f96d84b-4lss9\" (UID: \"7897ae2e-9365-4a43-95a5-6206a1f8d0e0\") " pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-4lss9"
Sep 11 14:21:34.444790 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:34.444770    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7897ae2e-9365-4a43-95a5-6206a1f8d0e0-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d8f96d84b-4lss9\" (UID: \"7897ae2e-9365-4a43-95a5-6206a1f8d0e0\") " pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-4lss9"
Sep 11 14:21:34.444934 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:34.444914    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7897ae2e-9365-4a43-95a5-6206a1f8d0e0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d8f96d84b-4lss9\" (UID: \"7897ae2e-9365-4a43-95a5-6206a1f8d0e0\") " pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-4lss9"
Sep 11 14:21:34.488933 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:34.488864    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q5vv\" (UniqueName: \"kubernetes.io/projected/7897ae2e-9365-4a43-95a5-6206a1f8d0e0-kube-api-access-8q5vv\") pod \"opendatahub-operator-controller-manager-7d8f96d84b-4lss9\" (UID: \"7897ae2e-9365-4a43-95a5-6206a1f8d0e0\") " pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-4lss9"
Sep 11 14:21:34.575246 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:34.575206    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-4lss9"
Sep 11 14:21:34.884415 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:34.884382    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-4lss9"]
Sep 11 14:21:34.889117 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:21:34.889093    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7897ae2e_9365_4a43_95a5_6206a1f8d0e0.slice/crio-e2845e0765d9955102c1132bb1cc1a08f4e515786c931874a1798a8365f1d430 WatchSource:0}: Error finding container e2845e0765d9955102c1132bb1cc1a08f4e515786c931874a1798a8365f1d430: Status 404 returned error can't find the container with id e2845e0765d9955102c1132bb1cc1a08f4e515786c931874a1798a8365f1d430
Sep 11 14:21:34.963362 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:34.963329    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-4lss9" event={"ID":"7897ae2e-9365-4a43-95a5-6206a1f8d0e0","Type":"ContainerStarted","Data":"e2845e0765d9955102c1132bb1cc1a08f4e515786c931874a1798a8365f1d430"}
Sep 11 14:21:37.978184 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:37.978150    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-4lss9" event={"ID":"7897ae2e-9365-4a43-95a5-6206a1f8d0e0","Type":"ContainerStarted","Data":"0c6b8d040214264eec20ba93dc49256b260f41eebc96e2c19c7422ff85146c4c"}
Sep 11 14:21:39.987947 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:39.987913    2581 generic.go:356] "Generic (PLEG): container finished" podID="7897ae2e-9365-4a43-95a5-6206a1f8d0e0" containerID="0c6b8d040214264eec20ba93dc49256b260f41eebc96e2c19c7422ff85146c4c" exitCode=0
Sep 11 14:21:39.988463 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:39.987997    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-4lss9" event={"ID":"7897ae2e-9365-4a43-95a5-6206a1f8d0e0","Type":"ContainerDied","Data":"0c6b8d040214264eec20ba93dc49256b260f41eebc96e2c19c7422ff85146c4c"}
Sep 11 14:21:40.994032 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:40.993994    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-4lss9" event={"ID":"7897ae2e-9365-4a43-95a5-6206a1f8d0e0","Type":"ContainerStarted","Data":"cd2933c4efb4a5515abdfd7032a60647b5b2d6cfd646add12a3c31f423429d50"}
Sep 11 14:21:40.994521 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:40.994235    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-4lss9"
Sep 11 14:21:41.052744 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:41.052686    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-4lss9" podStartSLOduration=2.712134399 podStartE2EDuration="7.052669361s" totalImagesPullingTime="3.31855177s" totalInitContainerRuntime="1.021983192s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:21:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:21:41.051287707 +0000 UTC m=+404.201837761" watchObservedRunningTime="2026-09-11 14:21:41.052669361 +0000 UTC m=+404.203219424"
Sep 11 14:21:51.999917 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:51.999880    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7d8f96d84b-4lss9"
Sep 11 14:21:54.426921 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:54.426881    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-57f955fb8f-j97rf"]
Sep 11 14:21:54.430161 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:54.430142    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-57f955fb8f-j97rf"
Sep 11 14:21:54.433317 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:21:54.433274    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kube-auth-proxy-57f955fb8f-j97rf\" is forbidden: User \"system:node:ip-10-0-135-150.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-135-150.ec2.internal' and this object" podUID="a9d0d8dc-ecfc-4fc7-89a0-7ae86ca0ff57" pod="openshift-ingress/kube-auth-proxy-57f955fb8f-j97rf"
Sep 11 14:21:54.433317 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:21:54.433293    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-135-150.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-135-150.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Sep 11 14:21:54.433544 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:21:54.433420    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"kube-auth-proxy-tls\" is forbidden: User \"system:node:ip-10-0-135-150.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-135-150.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" type="*v1.Secret"
Sep 11 14:21:54.433605 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:21:54.433565    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-135-150.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-135-150.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Sep 11 14:21:54.434104 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:21:54.434082    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"kube-auth-proxy-creds\" is forbidden: User \"system:node:ip-10-0-135-150.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-135-150.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" type="*v1.Secret"
Sep 11 14:21:54.460180 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:54.460129    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-57f955fb8f-j97rf"]
Sep 11 14:21:54.511905 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:54.511876    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9d0d8dc-ecfc-4fc7-89a0-7ae86ca0ff57-tls-certs\") pod \"kube-auth-proxy-57f955fb8f-j97rf\" (UID: \"a9d0d8dc-ecfc-4fc7-89a0-7ae86ca0ff57\") " pod="openshift-ingress/kube-auth-proxy-57f955fb8f-j97rf"
Sep 11 14:21:54.512084 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:54.511936    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6m89\" (UniqueName: \"kubernetes.io/projected/a9d0d8dc-ecfc-4fc7-89a0-7ae86ca0ff57-kube-api-access-s6m89\") pod \"kube-auth-proxy-57f955fb8f-j97rf\" (UID: \"a9d0d8dc-ecfc-4fc7-89a0-7ae86ca0ff57\") " pod="openshift-ingress/kube-auth-proxy-57f955fb8f-j97rf"
Sep 11 14:21:54.512084 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:54.511956    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a9d0d8dc-ecfc-4fc7-89a0-7ae86ca0ff57-tmp\") pod \"kube-auth-proxy-57f955fb8f-j97rf\" (UID: \"a9d0d8dc-ecfc-4fc7-89a0-7ae86ca0ff57\") " pod="openshift-ingress/kube-auth-proxy-57f955fb8f-j97rf"
Sep 11 14:21:54.613258 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:54.613221    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6m89\" (UniqueName: \"kubernetes.io/projected/a9d0d8dc-ecfc-4fc7-89a0-7ae86ca0ff57-kube-api-access-s6m89\") pod \"kube-auth-proxy-57f955fb8f-j97rf\" (UID: \"a9d0d8dc-ecfc-4fc7-89a0-7ae86ca0ff57\") " pod="openshift-ingress/kube-auth-proxy-57f955fb8f-j97rf"
Sep 11 14:21:54.613447 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:54.613270    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a9d0d8dc-ecfc-4fc7-89a0-7ae86ca0ff57-tmp\") pod \"kube-auth-proxy-57f955fb8f-j97rf\" (UID: \"a9d0d8dc-ecfc-4fc7-89a0-7ae86ca0ff57\") " pod="openshift-ingress/kube-auth-proxy-57f955fb8f-j97rf"
Sep 11 14:21:54.613497 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:54.613455    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9d0d8dc-ecfc-4fc7-89a0-7ae86ca0ff57-tls-certs\") pod \"kube-auth-proxy-57f955fb8f-j97rf\" (UID: \"a9d0d8dc-ecfc-4fc7-89a0-7ae86ca0ff57\") " pod="openshift-ingress/kube-auth-proxy-57f955fb8f-j97rf"
Sep 11 14:21:54.615564 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:54.615543    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a9d0d8dc-ecfc-4fc7-89a0-7ae86ca0ff57-tmp\") pod \"kube-auth-proxy-57f955fb8f-j97rf\" (UID: \"a9d0d8dc-ecfc-4fc7-89a0-7ae86ca0ff57\") " pod="openshift-ingress/kube-auth-proxy-57f955fb8f-j97rf"
Sep 11 14:21:55.463633 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:55.463596    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 11 14:21:55.466080 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:55.466054    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9d0d8dc-ecfc-4fc7-89a0-7ae86ca0ff57-tls-certs\") pod \"kube-auth-proxy-57f955fb8f-j97rf\" (UID: \"a9d0d8dc-ecfc-4fc7-89a0-7ae86ca0ff57\") " pod="openshift-ingress/kube-auth-proxy-57f955fb8f-j97rf"
Sep 11 14:21:55.489481 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:55.489449    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 11 14:21:55.637527 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:21:55.637487    2581 projected.go:291] Couldn't get configMap openshift-ingress/kube-root-ca.crt: failed to sync configmap cache: timed out waiting for the condition
Sep 11 14:21:55.637704 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:21:55.637539    2581 projected.go:196] Error preparing data for projected volume kube-api-access-s6m89 for pod openshift-ingress/kube-auth-proxy-57f955fb8f-j97rf: failed to sync configmap cache: timed out waiting for the condition
Sep 11 14:21:55.637704 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:21:55.637603    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a9d0d8dc-ecfc-4fc7-89a0-7ae86ca0ff57-kube-api-access-s6m89 podName:a9d0d8dc-ecfc-4fc7-89a0-7ae86ca0ff57 nodeName:}" failed. No retries permitted until 2026-09-11 14:21:56.137585611 +0000 UTC m=+419.288135659 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-s6m89" (UniqueName: "kubernetes.io/projected/a9d0d8dc-ecfc-4fc7-89a0-7ae86ca0ff57-kube-api-access-s6m89") pod "kube-auth-proxy-57f955fb8f-j97rf" (UID: "a9d0d8dc-ecfc-4fc7-89a0-7ae86ca0ff57") : failed to sync configmap cache: timed out waiting for the condition
Sep 11 14:21:55.886087 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:55.886056    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 11 14:21:56.078470 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:56.078438    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 11 14:21:56.228219 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:56.228118    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6m89\" (UniqueName: \"kubernetes.io/projected/a9d0d8dc-ecfc-4fc7-89a0-7ae86ca0ff57-kube-api-access-s6m89\") pod \"kube-auth-proxy-57f955fb8f-j97rf\" (UID: \"a9d0d8dc-ecfc-4fc7-89a0-7ae86ca0ff57\") " pod="openshift-ingress/kube-auth-proxy-57f955fb8f-j97rf"
Sep 11 14:21:56.230596 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:56.230564    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6m89\" (UniqueName: \"kubernetes.io/projected/a9d0d8dc-ecfc-4fc7-89a0-7ae86ca0ff57-kube-api-access-s6m89\") pod \"kube-auth-proxy-57f955fb8f-j97rf\" (UID: \"a9d0d8dc-ecfc-4fc7-89a0-7ae86ca0ff57\") " pod="openshift-ingress/kube-auth-proxy-57f955fb8f-j97rf"
Sep 11 14:21:56.239377 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:56.239352    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-57f955fb8f-j97rf"
Sep 11 14:21:56.521904 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:56.521882    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-57f955fb8f-j97rf"]
Sep 11 14:21:56.523979 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:21:56.523940    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9d0d8dc_ecfc_4fc7_89a0_7ae86ca0ff57.slice/crio-d429fb7d673f4bd916fed4f358999ff6ddfa5c24fdc7051b232607809a0d781e WatchSource:0}: Error finding container d429fb7d673f4bd916fed4f358999ff6ddfa5c24fdc7051b232607809a0d781e: Status 404 returned error can't find the container with id d429fb7d673f4bd916fed4f358999ff6ddfa5c24fdc7051b232607809a0d781e
Sep 11 14:21:57.062720 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:57.062640    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-57f955fb8f-j97rf" event={"ID":"a9d0d8dc-ecfc-4fc7-89a0-7ae86ca0ff57","Type":"ContainerStarted","Data":"d429fb7d673f4bd916fed4f358999ff6ddfa5c24fdc7051b232607809a0d781e"}
Sep 11 14:21:59.017701 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:21:59.017673    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 11 14:22:00.076732 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:00.076697    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-57f955fb8f-j97rf" event={"ID":"a9d0d8dc-ecfc-4fc7-89a0-7ae86ca0ff57","Type":"ContainerStarted","Data":"4ee9ad62a2084c40b538b6d0a864eef32ecc01dc08661bc0402022d3d5741416"}
Sep 11 14:22:00.102562 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:00.102464    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-57f955fb8f-j97rf" podStartSLOduration=3.623539738 podStartE2EDuration="6.102445387s" totalImagesPullingTime="2.478905649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:21:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:22:00.101801822 +0000 UTC m=+423.252351886" watchObservedRunningTime="2026-09-11 14:22:00.102445387 +0000 UTC m=+423.252995452"
Sep 11 14:22:02.324868 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:02.324836    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct"]
Sep 11 14:22:02.328161 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:02.328142    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct"
Sep 11 14:22:02.331057 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:02.331032    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 11 14:22:02.331177 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:02.331133    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-6nvmq\""
Sep 11 14:22:02.347516 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:02.347485    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct"]
Sep 11 14:22:02.482917 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:02.482873    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fa7ea94-b71e-4e6f-98c8-68bd8eb103e7-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct\" (UID: \"1fa7ea94-b71e-4e6f-98c8-68bd8eb103e7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct"
Sep 11 14:22:02.482917 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:02.482917    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa7ea94-b71e-4e6f-98c8-68bd8eb103e7-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct\" (UID: \"1fa7ea94-b71e-4e6f-98c8-68bd8eb103e7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct"
Sep 11 14:22:02.483173 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:02.482940    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fa7ea94-b71e-4e6f-98c8-68bd8eb103e7-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct\" (UID: \"1fa7ea94-b71e-4e6f-98c8-68bd8eb103e7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct"
Sep 11 14:22:02.483173 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:02.483055    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fa7ea94-b71e-4e6f-98c8-68bd8eb103e7-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct\" (UID: \"1fa7ea94-b71e-4e6f-98c8-68bd8eb103e7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct"
Sep 11 14:22:02.483173 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:02.483158    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa7ea94-b71e-4e6f-98c8-68bd8eb103e7-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct\" (UID: \"1fa7ea94-b71e-4e6f-98c8-68bd8eb103e7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct"
Sep 11 14:22:02.483276 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:02.483186    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fa7ea94-b71e-4e6f-98c8-68bd8eb103e7-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct\" (UID: \"1fa7ea94-b71e-4e6f-98c8-68bd8eb103e7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct"
Sep 11 14:22:02.483276 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:02.483215    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fa7ea94-b71e-4e6f-98c8-68bd8eb103e7-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct\" (UID: \"1fa7ea94-b71e-4e6f-98c8-68bd8eb103e7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct"
Sep 11 14:22:02.483361 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:02.483271    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fa7ea94-b71e-4e6f-98c8-68bd8eb103e7-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct\" (UID: \"1fa7ea94-b71e-4e6f-98c8-68bd8eb103e7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct"
Sep 11 14:22:02.483361 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:02.483316    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q8s7\" (UniqueName: \"kubernetes.io/projected/1fa7ea94-b71e-4e6f-98c8-68bd8eb103e7-kube-api-access-4q8s7\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct\" (UID: \"1fa7ea94-b71e-4e6f-98c8-68bd8eb103e7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct"
Sep 11 14:22:02.584560 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:02.584460    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q8s7\" (UniqueName: \"kubernetes.io/projected/1fa7ea94-b71e-4e6f-98c8-68bd8eb103e7-kube-api-access-4q8s7\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct\" (UID: \"1fa7ea94-b71e-4e6f-98c8-68bd8eb103e7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct"
Sep 11 14:22:02.584560 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:02.584506    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fa7ea94-b71e-4e6f-98c8-68bd8eb103e7-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct\" (UID: \"1fa7ea94-b71e-4e6f-98c8-68bd8eb103e7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct"
Sep 11 14:22:02.584560 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:02.584532    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa7ea94-b71e-4e6f-98c8-68bd8eb103e7-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct\" (UID: \"1fa7ea94-b71e-4e6f-98c8-68bd8eb103e7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct"
Sep 11 14:22:02.584560 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:02.584553    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fa7ea94-b71e-4e6f-98c8-68bd8eb103e7-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct\" (UID: \"1fa7ea94-b71e-4e6f-98c8-68bd8eb103e7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct"
Sep 11 14:22:02.584888 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:02.584572    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fa7ea94-b71e-4e6f-98c8-68bd8eb103e7-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct\" (UID: \"1fa7ea94-b71e-4e6f-98c8-68bd8eb103e7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct"
Sep 11 14:22:02.584888 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:02.584622    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa7ea94-b71e-4e6f-98c8-68bd8eb103e7-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct\" (UID: \"1fa7ea94-b71e-4e6f-98c8-68bd8eb103e7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct"
Sep 11 14:22:02.584888 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:02.584649    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fa7ea94-b71e-4e6f-98c8-68bd8eb103e7-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct\" (UID: \"1fa7ea94-b71e-4e6f-98c8-68bd8eb103e7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct"
Sep 11 14:22:02.584888 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:02.584694    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fa7ea94-b71e-4e6f-98c8-68bd8eb103e7-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct\" (UID: \"1fa7ea94-b71e-4e6f-98c8-68bd8eb103e7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct"
Sep 11 14:22:02.584888 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:02.584721    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fa7ea94-b71e-4e6f-98c8-68bd8eb103e7-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct\" (UID: \"1fa7ea94-b71e-4e6f-98c8-68bd8eb103e7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct"
Sep 11 14:22:02.585185 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:02.585029    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fa7ea94-b71e-4e6f-98c8-68bd8eb103e7-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct\" (UID: \"1fa7ea94-b71e-4e6f-98c8-68bd8eb103e7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct"
Sep 11 14:22:02.585185 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:02.585094    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa7ea94-b71e-4e6f-98c8-68bd8eb103e7-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct\" (UID: \"1fa7ea94-b71e-4e6f-98c8-68bd8eb103e7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct"
Sep 11 14:22:02.585391 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:02.585358    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fa7ea94-b71e-4e6f-98c8-68bd8eb103e7-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct\" (UID: \"1fa7ea94-b71e-4e6f-98c8-68bd8eb103e7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct"
Sep 11 14:22:02.585468 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:02.585400    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fa7ea94-b71e-4e6f-98c8-68bd8eb103e7-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct\" (UID: \"1fa7ea94-b71e-4e6f-98c8-68bd8eb103e7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct"
Sep 11 14:22:02.585524 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:02.585497    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa7ea94-b71e-4e6f-98c8-68bd8eb103e7-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct\" (UID: \"1fa7ea94-b71e-4e6f-98c8-68bd8eb103e7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct"
Sep 11 14:22:02.587132 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:02.587113    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fa7ea94-b71e-4e6f-98c8-68bd8eb103e7-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct\" (UID: \"1fa7ea94-b71e-4e6f-98c8-68bd8eb103e7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct"
Sep 11 14:22:02.587363 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:02.587345    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fa7ea94-b71e-4e6f-98c8-68bd8eb103e7-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct\" (UID: \"1fa7ea94-b71e-4e6f-98c8-68bd8eb103e7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct"
Sep 11 14:22:02.600616 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:02.600586    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q8s7\" (UniqueName: \"kubernetes.io/projected/1fa7ea94-b71e-4e6f-98c8-68bd8eb103e7-kube-api-access-4q8s7\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct\" (UID: \"1fa7ea94-b71e-4e6f-98c8-68bd8eb103e7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct"
Sep 11 14:22:02.602160 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:02.602141    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fa7ea94-b71e-4e6f-98c8-68bd8eb103e7-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct\" (UID: \"1fa7ea94-b71e-4e6f-98c8-68bd8eb103e7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct"
Sep 11 14:22:02.639278 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:02.639232    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct"
Sep 11 14:22:02.944393 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:02.944362    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct"]
Sep 11 14:22:03.089658 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:03.089624    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct" event={"ID":"1fa7ea94-b71e-4e6f-98c8-68bd8eb103e7","Type":"ContainerStarted","Data":"84b60f2471b802777e2e55e781770a4c0a732ec02e16ebe3b0aeea27dcc64dd2"}
Sep 11 14:22:05.374672 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:05.374638    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-v2qsw"]
Sep 11 14:22:05.380740 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:05.380717    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-v2qsw"
Sep 11 14:22:05.384090 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:05.384068    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-wj45j\""
Sep 11 14:22:05.384202 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:05.384108    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 11 14:22:05.393089 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:05.393062    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-v2qsw"]
Sep 11 14:22:05.510165 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:05.510125    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f6ded6ea-11f1-49f8-bd47-cc2fe4016904-cert\") pod \"odh-model-controller-7998c67dc4-v2qsw\" (UID: \"f6ded6ea-11f1-49f8-bd47-cc2fe4016904\") " pod="opendatahub/odh-model-controller-7998c67dc4-v2qsw"
Sep 11 14:22:05.510345 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:05.510274    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5dss\" (UniqueName: \"kubernetes.io/projected/f6ded6ea-11f1-49f8-bd47-cc2fe4016904-kube-api-access-g5dss\") pod \"odh-model-controller-7998c67dc4-v2qsw\" (UID: \"f6ded6ea-11f1-49f8-bd47-cc2fe4016904\") " pod="opendatahub/odh-model-controller-7998c67dc4-v2qsw"
Sep 11 14:22:05.611649 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:05.611610    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f6ded6ea-11f1-49f8-bd47-cc2fe4016904-cert\") pod \"odh-model-controller-7998c67dc4-v2qsw\" (UID: \"f6ded6ea-11f1-49f8-bd47-cc2fe4016904\") " pod="opendatahub/odh-model-controller-7998c67dc4-v2qsw"
Sep 11 14:22:05.611844 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:05.611687    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5dss\" (UniqueName: \"kubernetes.io/projected/f6ded6ea-11f1-49f8-bd47-cc2fe4016904-kube-api-access-g5dss\") pod \"odh-model-controller-7998c67dc4-v2qsw\" (UID: \"f6ded6ea-11f1-49f8-bd47-cc2fe4016904\") " pod="opendatahub/odh-model-controller-7998c67dc4-v2qsw"
Sep 11 14:22:05.611844 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:22:05.611769    2581 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 11 14:22:05.611950 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:22:05.611868    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6ded6ea-11f1-49f8-bd47-cc2fe4016904-cert podName:f6ded6ea-11f1-49f8-bd47-cc2fe4016904 nodeName:}" failed. No retries permitted until 2026-09-11 14:22:06.111848061 +0000 UTC m=+429.262398118 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f6ded6ea-11f1-49f8-bd47-cc2fe4016904-cert") pod "odh-model-controller-7998c67dc4-v2qsw" (UID: "f6ded6ea-11f1-49f8-bd47-cc2fe4016904") : secret "odh-model-controller-webhook-cert" not found
Sep 11 14:22:05.636935 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:05.636859    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5dss\" (UniqueName: \"kubernetes.io/projected/f6ded6ea-11f1-49f8-bd47-cc2fe4016904-kube-api-access-g5dss\") pod \"odh-model-controller-7998c67dc4-v2qsw\" (UID: \"f6ded6ea-11f1-49f8-bd47-cc2fe4016904\") " pod="opendatahub/odh-model-controller-7998c67dc4-v2qsw"
Sep 11 14:22:05.686732 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:05.686692    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:22:05.686796 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:05.686767    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:22:05.686831 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:05.686797    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:22:06.105946 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:06.105901    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct" event={"ID":"1fa7ea94-b71e-4e6f-98c8-68bd8eb103e7","Type":"ContainerStarted","Data":"29226213bc5aac6fe82688cee49e4ba07f150b0e45cd1c0691fe7da2580f355c"}
Sep 11 14:22:06.117224 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:06.117189    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f6ded6ea-11f1-49f8-bd47-cc2fe4016904-cert\") pod \"odh-model-controller-7998c67dc4-v2qsw\" (UID: \"f6ded6ea-11f1-49f8-bd47-cc2fe4016904\") " pod="opendatahub/odh-model-controller-7998c67dc4-v2qsw"
Sep 11 14:22:06.119934 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:06.119904    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f6ded6ea-11f1-49f8-bd47-cc2fe4016904-cert\") pod \"odh-model-controller-7998c67dc4-v2qsw\" (UID: \"f6ded6ea-11f1-49f8-bd47-cc2fe4016904\") " pod="opendatahub/odh-model-controller-7998c67dc4-v2qsw"
Sep 11 14:22:06.141459 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:06.141408    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct" podStartSLOduration=1.40296833 podStartE2EDuration="4.141391192s" totalImagesPullingTime="2.738422862s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:22:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:22:06.139965081 +0000 UTC m=+429.290515143" watchObservedRunningTime="2026-09-11 14:22:06.141391192 +0000 UTC m=+429.291941256"
Sep 11 14:22:06.293759 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:06.293725    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-v2qsw"
Sep 11 14:22:06.580222 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:06.580195    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-v2qsw"]
Sep 11 14:22:06.582714 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:22:06.582686    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6ded6ea_11f1_49f8_bd47_cc2fe4016904.slice/crio-26613483fcaaf1f379cb92248dc5c30d2b0550e6b3fe3cfcf6da6ef3efcd0b32 WatchSource:0}: Error finding container 26613483fcaaf1f379cb92248dc5c30d2b0550e6b3fe3cfcf6da6ef3efcd0b32: Status 404 returned error can't find the container with id 26613483fcaaf1f379cb92248dc5c30d2b0550e6b3fe3cfcf6da6ef3efcd0b32
Sep 11 14:22:06.639571 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:06.639541    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct"
Sep 11 14:22:06.644229 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:06.644200    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct"
Sep 11 14:22:07.111010 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:07.110954    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-v2qsw" event={"ID":"f6ded6ea-11f1-49f8-bd47-cc2fe4016904","Type":"ContainerStarted","Data":"26613483fcaaf1f379cb92248dc5c30d2b0550e6b3fe3cfcf6da6ef3efcd0b32"}
Sep 11 14:22:07.111271 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:07.111223    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct"
Sep 11 14:22:07.112156 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:07.112136    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct"
Sep 11 14:22:07.429147 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:07.428997    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-76d48cb76f-rckgx"]
Sep 11 14:22:07.432730 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:07.432693    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76d48cb76f-rckgx"
Sep 11 14:22:07.455018 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:07.454987    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76d48cb76f-rckgx"]
Sep 11 14:22:07.531631 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:07.531598    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/91e6b88f-64e0-4141-bd4d-eac6ad6e13cb-console-oauth-config\") pod \"console-76d48cb76f-rckgx\" (UID: \"91e6b88f-64e0-4141-bd4d-eac6ad6e13cb\") " pod="openshift-console/console-76d48cb76f-rckgx"
Sep 11 14:22:07.531854 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:07.531646    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rhq5\" (UniqueName: \"kubernetes.io/projected/91e6b88f-64e0-4141-bd4d-eac6ad6e13cb-kube-api-access-6rhq5\") pod \"console-76d48cb76f-rckgx\" (UID: \"91e6b88f-64e0-4141-bd4d-eac6ad6e13cb\") " pod="openshift-console/console-76d48cb76f-rckgx"
Sep 11 14:22:07.531854 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:07.531719    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91e6b88f-64e0-4141-bd4d-eac6ad6e13cb-trusted-ca-bundle\") pod \"console-76d48cb76f-rckgx\" (UID: \"91e6b88f-64e0-4141-bd4d-eac6ad6e13cb\") " pod="openshift-console/console-76d48cb76f-rckgx"
Sep 11 14:22:07.531854 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:07.531778    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/91e6b88f-64e0-4141-bd4d-eac6ad6e13cb-console-config\") pod \"console-76d48cb76f-rckgx\" (UID: \"91e6b88f-64e0-4141-bd4d-eac6ad6e13cb\") " pod="openshift-console/console-76d48cb76f-rckgx"
Sep 11 14:22:07.531854 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:07.531807    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/91e6b88f-64e0-4141-bd4d-eac6ad6e13cb-oauth-serving-cert\") pod \"console-76d48cb76f-rckgx\" (UID: \"91e6b88f-64e0-4141-bd4d-eac6ad6e13cb\") " pod="openshift-console/console-76d48cb76f-rckgx"
Sep 11 14:22:07.532058 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:07.531910    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/91e6b88f-64e0-4141-bd4d-eac6ad6e13cb-console-serving-cert\") pod \"console-76d48cb76f-rckgx\" (UID: \"91e6b88f-64e0-4141-bd4d-eac6ad6e13cb\") " pod="openshift-console/console-76d48cb76f-rckgx"
Sep 11 14:22:07.532058 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:07.531939    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/91e6b88f-64e0-4141-bd4d-eac6ad6e13cb-service-ca\") pod \"console-76d48cb76f-rckgx\" (UID: \"91e6b88f-64e0-4141-bd4d-eac6ad6e13cb\") " pod="openshift-console/console-76d48cb76f-rckgx"
Sep 11 14:22:07.632623 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:07.632588    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/91e6b88f-64e0-4141-bd4d-eac6ad6e13cb-console-serving-cert\") pod \"console-76d48cb76f-rckgx\" (UID: \"91e6b88f-64e0-4141-bd4d-eac6ad6e13cb\") " pod="openshift-console/console-76d48cb76f-rckgx"
Sep 11 14:22:07.632623 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:07.632626    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/91e6b88f-64e0-4141-bd4d-eac6ad6e13cb-service-ca\") pod \"console-76d48cb76f-rckgx\" (UID: \"91e6b88f-64e0-4141-bd4d-eac6ad6e13cb\") " pod="openshift-console/console-76d48cb76f-rckgx"
Sep 11 14:22:07.633184 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:07.632679    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/91e6b88f-64e0-4141-bd4d-eac6ad6e13cb-console-oauth-config\") pod \"console-76d48cb76f-rckgx\" (UID: \"91e6b88f-64e0-4141-bd4d-eac6ad6e13cb\") " pod="openshift-console/console-76d48cb76f-rckgx"
Sep 11 14:22:07.633184 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:07.632706    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rhq5\" (UniqueName: \"kubernetes.io/projected/91e6b88f-64e0-4141-bd4d-eac6ad6e13cb-kube-api-access-6rhq5\") pod \"console-76d48cb76f-rckgx\" (UID: \"91e6b88f-64e0-4141-bd4d-eac6ad6e13cb\") " pod="openshift-console/console-76d48cb76f-rckgx"
Sep 11 14:22:07.633184 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:07.632738    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91e6b88f-64e0-4141-bd4d-eac6ad6e13cb-trusted-ca-bundle\") pod \"console-76d48cb76f-rckgx\" (UID: \"91e6b88f-64e0-4141-bd4d-eac6ad6e13cb\") " pod="openshift-console/console-76d48cb76f-rckgx"
Sep 11 14:22:07.633184 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:07.632774    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/91e6b88f-64e0-4141-bd4d-eac6ad6e13cb-console-config\") pod \"console-76d48cb76f-rckgx\" (UID: \"91e6b88f-64e0-4141-bd4d-eac6ad6e13cb\") " pod="openshift-console/console-76d48cb76f-rckgx"
Sep 11 14:22:07.633184 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:07.632796    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/91e6b88f-64e0-4141-bd4d-eac6ad6e13cb-oauth-serving-cert\") pod \"console-76d48cb76f-rckgx\" (UID: \"91e6b88f-64e0-4141-bd4d-eac6ad6e13cb\") " pod="openshift-console/console-76d48cb76f-rckgx"
Sep 11 14:22:07.633645 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:07.633583    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/91e6b88f-64e0-4141-bd4d-eac6ad6e13cb-oauth-serving-cert\") pod \"console-76d48cb76f-rckgx\" (UID: \"91e6b88f-64e0-4141-bd4d-eac6ad6e13cb\") " pod="openshift-console/console-76d48cb76f-rckgx"
Sep 11 14:22:07.633645 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:07.633624    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/91e6b88f-64e0-4141-bd4d-eac6ad6e13cb-service-ca\") pod \"console-76d48cb76f-rckgx\" (UID: \"91e6b88f-64e0-4141-bd4d-eac6ad6e13cb\") " pod="openshift-console/console-76d48cb76f-rckgx"
Sep 11 14:22:07.633892 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:07.633841    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/91e6b88f-64e0-4141-bd4d-eac6ad6e13cb-console-config\") pod \"console-76d48cb76f-rckgx\" (UID: \"91e6b88f-64e0-4141-bd4d-eac6ad6e13cb\") " pod="openshift-console/console-76d48cb76f-rckgx"
Sep 11 14:22:07.634210 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:07.634089    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91e6b88f-64e0-4141-bd4d-eac6ad6e13cb-trusted-ca-bundle\") pod \"console-76d48cb76f-rckgx\" (UID: \"91e6b88f-64e0-4141-bd4d-eac6ad6e13cb\") " pod="openshift-console/console-76d48cb76f-rckgx"
Sep 11 14:22:07.635819 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:07.635769    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/91e6b88f-64e0-4141-bd4d-eac6ad6e13cb-console-oauth-config\") pod \"console-76d48cb76f-rckgx\" (UID: \"91e6b88f-64e0-4141-bd4d-eac6ad6e13cb\") " pod="openshift-console/console-76d48cb76f-rckgx"
Sep 11 14:22:07.636099 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:07.636078    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/91e6b88f-64e0-4141-bd4d-eac6ad6e13cb-console-serving-cert\") pod \"console-76d48cb76f-rckgx\" (UID: \"91e6b88f-64e0-4141-bd4d-eac6ad6e13cb\") " pod="openshift-console/console-76d48cb76f-rckgx"
Sep 11 14:22:07.648566 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:07.648509    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rhq5\" (UniqueName: \"kubernetes.io/projected/91e6b88f-64e0-4141-bd4d-eac6ad6e13cb-kube-api-access-6rhq5\") pod \"console-76d48cb76f-rckgx\" (UID: \"91e6b88f-64e0-4141-bd4d-eac6ad6e13cb\") " pod="openshift-console/console-76d48cb76f-rckgx"
Sep 11 14:22:07.744843 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:07.744784    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76d48cb76f-rckgx"
Sep 11 14:22:08.043006 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:08.042915    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76d48cb76f-rckgx"]
Sep 11 14:22:09.154988 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:22:09.154942    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e6b88f_64e0_4141_bd4d_eac6ad6e13cb.slice/crio-e89dd2923310caa3e3bc629b1a64cc782c1750ce9417abbf340f343ad54ac29c WatchSource:0}: Error finding container e89dd2923310caa3e3bc629b1a64cc782c1750ce9417abbf340f343ad54ac29c: Status 404 returned error can't find the container with id e89dd2923310caa3e3bc629b1a64cc782c1750ce9417abbf340f343ad54ac29c
Sep 11 14:22:10.127636 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:10.127589    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76d48cb76f-rckgx" event={"ID":"91e6b88f-64e0-4141-bd4d-eac6ad6e13cb","Type":"ContainerStarted","Data":"dc127566097a2fedcd9190c4022bdbd07a50266ae05b003217103b70633f5302"}
Sep 11 14:22:10.127838 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:10.127651    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76d48cb76f-rckgx" event={"ID":"91e6b88f-64e0-4141-bd4d-eac6ad6e13cb","Type":"ContainerStarted","Data":"e89dd2923310caa3e3bc629b1a64cc782c1750ce9417abbf340f343ad54ac29c"}
Sep 11 14:22:10.129160 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:10.129138    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-v2qsw" event={"ID":"f6ded6ea-11f1-49f8-bd47-cc2fe4016904","Type":"ContainerStarted","Data":"effa745febab88dd9b3c71570fa38ae50e1bec62914e9b70e54f6a940aba01b0"}
Sep 11 14:22:10.129324 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:10.129308    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-v2qsw"
Sep 11 14:22:10.159002 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:10.158936    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-76d48cb76f-rckgx" podStartSLOduration=3.158918146 podStartE2EDuration="3.158918146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:22:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:22:10.156405099 +0000 UTC m=+433.306955163" watchObservedRunningTime="2026-09-11 14:22:10.158918146 +0000 UTC m=+433.309468210"
Sep 11 14:22:12.102531 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:12.102457    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-v2qsw" podStartSLOduration=4.479558022 podStartE2EDuration="7.102438521s" totalImagesPullingTime="2.622880499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:22:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:22:10.183192162 +0000 UTC m=+433.333742227" watchObservedRunningTime="2026-09-11 14:22:12.102438521 +0000 UTC m=+435.252988585"
Sep 11 14:22:12.103039 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:12.103018    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-744bf4ff48-8rn2q"]
Sep 11 14:22:12.106446 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:12.106431    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-744bf4ff48-8rn2q"
Sep 11 14:22:12.119751 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:12.119727    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 11 14:22:12.119879 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:12.119727    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 11 14:22:12.119879 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:12.119727    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-jr844\""
Sep 11 14:22:12.119879 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:12.119728    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 11 14:22:12.130624 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:12.130594    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-744bf4ff48-8rn2q"]
Sep 11 14:22:12.173248 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:12.173216    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f0948f34-1747-491e-b499-fe02725bc43c-metrics-cert\") pod \"lws-controller-manager-744bf4ff48-8rn2q\" (UID: \"f0948f34-1747-491e-b499-fe02725bc43c\") " pod="openshift-lws-operator/lws-controller-manager-744bf4ff48-8rn2q"
Sep 11 14:22:12.173248 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:12.173246    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cmjf\" (UniqueName: \"kubernetes.io/projected/f0948f34-1747-491e-b499-fe02725bc43c-kube-api-access-8cmjf\") pod \"lws-controller-manager-744bf4ff48-8rn2q\" (UID: \"f0948f34-1747-491e-b499-fe02725bc43c\") " pod="openshift-lws-operator/lws-controller-manager-744bf4ff48-8rn2q"
Sep 11 14:22:12.173448 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:12.173381    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f0948f34-1747-491e-b499-fe02725bc43c-cert\") pod \"lws-controller-manager-744bf4ff48-8rn2q\" (UID: \"f0948f34-1747-491e-b499-fe02725bc43c\") " pod="openshift-lws-operator/lws-controller-manager-744bf4ff48-8rn2q"
Sep 11 14:22:12.173448 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:12.173416    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f0948f34-1747-491e-b499-fe02725bc43c-manager-config\") pod \"lws-controller-manager-744bf4ff48-8rn2q\" (UID: \"f0948f34-1747-491e-b499-fe02725bc43c\") " pod="openshift-lws-operator/lws-controller-manager-744bf4ff48-8rn2q"
Sep 11 14:22:12.274500 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:12.274463    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f0948f34-1747-491e-b499-fe02725bc43c-cert\") pod \"lws-controller-manager-744bf4ff48-8rn2q\" (UID: \"f0948f34-1747-491e-b499-fe02725bc43c\") " pod="openshift-lws-operator/lws-controller-manager-744bf4ff48-8rn2q"
Sep 11 14:22:12.274500 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:12.274502    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f0948f34-1747-491e-b499-fe02725bc43c-manager-config\") pod \"lws-controller-manager-744bf4ff48-8rn2q\" (UID: \"f0948f34-1747-491e-b499-fe02725bc43c\") " pod="openshift-lws-operator/lws-controller-manager-744bf4ff48-8rn2q"
Sep 11 14:22:12.274701 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:12.274532    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f0948f34-1747-491e-b499-fe02725bc43c-metrics-cert\") pod \"lws-controller-manager-744bf4ff48-8rn2q\" (UID: \"f0948f34-1747-491e-b499-fe02725bc43c\") " pod="openshift-lws-operator/lws-controller-manager-744bf4ff48-8rn2q"
Sep 11 14:22:12.274701 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:12.274656    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cmjf\" (UniqueName: \"kubernetes.io/projected/f0948f34-1747-491e-b499-fe02725bc43c-kube-api-access-8cmjf\") pod \"lws-controller-manager-744bf4ff48-8rn2q\" (UID: \"f0948f34-1747-491e-b499-fe02725bc43c\") " pod="openshift-lws-operator/lws-controller-manager-744bf4ff48-8rn2q"
Sep 11 14:22:12.275204 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:12.275176    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f0948f34-1747-491e-b499-fe02725bc43c-manager-config\") pod \"lws-controller-manager-744bf4ff48-8rn2q\" (UID: \"f0948f34-1747-491e-b499-fe02725bc43c\") " pod="openshift-lws-operator/lws-controller-manager-744bf4ff48-8rn2q"
Sep 11 14:22:12.277150 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:12.277128    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f0948f34-1747-491e-b499-fe02725bc43c-cert\") pod \"lws-controller-manager-744bf4ff48-8rn2q\" (UID: \"f0948f34-1747-491e-b499-fe02725bc43c\") " pod="openshift-lws-operator/lws-controller-manager-744bf4ff48-8rn2q"
Sep 11 14:22:12.277224 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:12.277163    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f0948f34-1747-491e-b499-fe02725bc43c-metrics-cert\") pod \"lws-controller-manager-744bf4ff48-8rn2q\" (UID: \"f0948f34-1747-491e-b499-fe02725bc43c\") " pod="openshift-lws-operator/lws-controller-manager-744bf4ff48-8rn2q"
Sep 11 14:22:12.297549 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:12.297518    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cmjf\" (UniqueName: \"kubernetes.io/projected/f0948f34-1747-491e-b499-fe02725bc43c-kube-api-access-8cmjf\") pod \"lws-controller-manager-744bf4ff48-8rn2q\" (UID: \"f0948f34-1747-491e-b499-fe02725bc43c\") " pod="openshift-lws-operator/lws-controller-manager-744bf4ff48-8rn2q"
Sep 11 14:22:12.415352 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:12.415264    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-744bf4ff48-8rn2q"
Sep 11 14:22:12.724094 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:12.724065    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-744bf4ff48-8rn2q"]
Sep 11 14:22:12.725039 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:22:12.725009    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0948f34_1747_491e_b499_fe02725bc43c.slice/crio-3abf59a0294dd3827b82109632d68df809185ff8dc375f625312fcfda942c3e8 WatchSource:0}: Error finding container 3abf59a0294dd3827b82109632d68df809185ff8dc375f625312fcfda942c3e8: Status 404 returned error can't find the container with id 3abf59a0294dd3827b82109632d68df809185ff8dc375f625312fcfda942c3e8
Sep 11 14:22:13.144554 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:13.144521    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-744bf4ff48-8rn2q" event={"ID":"f0948f34-1747-491e-b499-fe02725bc43c","Type":"ContainerStarted","Data":"3abf59a0294dd3827b82109632d68df809185ff8dc375f625312fcfda942c3e8"}
Sep 11 14:22:15.155493 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:15.155457    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-744bf4ff48-8rn2q" event={"ID":"f0948f34-1747-491e-b499-fe02725bc43c","Type":"ContainerStarted","Data":"509cd64b4e1b3becd3143a442776ed4f95ecc3c49908ef34c38099f5e3c3cf39"}
Sep 11 14:22:15.155878 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:15.155522    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-744bf4ff48-8rn2q"
Sep 11 14:22:15.191935 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:15.191870    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-744bf4ff48-8rn2q" podStartSLOduration=1.491320858 podStartE2EDuration="3.191845873s" totalImagesPullingTime="1.700525015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:22:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:22:15.191058071 +0000 UTC m=+438.341608136" watchObservedRunningTime="2026-09-11 14:22:15.191845873 +0000 UTC m=+438.342395938"
Sep 11 14:22:17.745985 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:17.745940    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-76d48cb76f-rckgx"
Sep 11 14:22:17.746377 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:17.746031    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-76d48cb76f-rckgx"
Sep 11 14:22:17.750671 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:17.750650    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-76d48cb76f-rckgx"
Sep 11 14:22:18.173888 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:18.173806    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-76d48cb76f-rckgx"
Sep 11 14:22:18.311433 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:18.311399    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65c7b69c6c-nlhdj"]
Sep 11 14:22:21.135553 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:21.135472    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-v2qsw"
Sep 11 14:22:26.162450 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:26.162413    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-744bf4ff48-8rn2q"
Sep 11 14:22:43.330038 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:43.329996    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-65c7b69c6c-nlhdj" podUID="af59576c-6739-4472-89e1-796775e0aa3a" containerName="console" containerID="cri-o://f6aae31e758caef7472f056f3594fc62b10991a83e8c8a8dd51ca5ad05055b92" gracePeriod=15
Sep 11 14:22:43.574893 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:43.574859    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65c7b69c6c-nlhdj"
Sep 11 14:22:43.638534 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:43.638455    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/af59576c-6739-4472-89e1-796775e0aa3a-oauth-serving-cert\") pod \"af59576c-6739-4472-89e1-796775e0aa3a\" (UID: \"af59576c-6739-4472-89e1-796775e0aa3a\") "
Sep 11 14:22:43.638534 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:43.638516    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/af59576c-6739-4472-89e1-796775e0aa3a-console-config\") pod \"af59576c-6739-4472-89e1-796775e0aa3a\" (UID: \"af59576c-6739-4472-89e1-796775e0aa3a\") "
Sep 11 14:22:43.638793 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:43.638560    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af59576c-6739-4472-89e1-796775e0aa3a-trusted-ca-bundle\") pod \"af59576c-6739-4472-89e1-796775e0aa3a\" (UID: \"af59576c-6739-4472-89e1-796775e0aa3a\") "
Sep 11 14:22:43.638793 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:43.638588    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/af59576c-6739-4472-89e1-796775e0aa3a-console-serving-cert\") pod \"af59576c-6739-4472-89e1-796775e0aa3a\" (UID: \"af59576c-6739-4472-89e1-796775e0aa3a\") "
Sep 11 14:22:43.638793 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:43.638623    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/af59576c-6739-4472-89e1-796775e0aa3a-console-oauth-config\") pod \"af59576c-6739-4472-89e1-796775e0aa3a\" (UID: \"af59576c-6739-4472-89e1-796775e0aa3a\") "
Sep 11 14:22:43.638793 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:43.638651    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/af59576c-6739-4472-89e1-796775e0aa3a-service-ca\") pod \"af59576c-6739-4472-89e1-796775e0aa3a\" (UID: \"af59576c-6739-4472-89e1-796775e0aa3a\") "
Sep 11 14:22:43.638793 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:43.638695    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-98fbt\" (UniqueName: \"kubernetes.io/projected/af59576c-6739-4472-89e1-796775e0aa3a-kube-api-access-98fbt\") pod \"af59576c-6739-4472-89e1-796775e0aa3a\" (UID: \"af59576c-6739-4472-89e1-796775e0aa3a\") "
Sep 11 14:22:43.639079 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:43.638923    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af59576c-6739-4472-89e1-796775e0aa3a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "af59576c-6739-4472-89e1-796775e0aa3a" (UID: "af59576c-6739-4472-89e1-796775e0aa3a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:22:43.639137 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:43.639100    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af59576c-6739-4472-89e1-796775e0aa3a-console-config" (OuterVolumeSpecName: "console-config") pod "af59576c-6739-4472-89e1-796775e0aa3a" (UID: "af59576c-6739-4472-89e1-796775e0aa3a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:22:43.639296 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:43.639266    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af59576c-6739-4472-89e1-796775e0aa3a-service-ca" (OuterVolumeSpecName: "service-ca") pod "af59576c-6739-4472-89e1-796775e0aa3a" (UID: "af59576c-6739-4472-89e1-796775e0aa3a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:22:43.639408 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:43.639377    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af59576c-6739-4472-89e1-796775e0aa3a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "af59576c-6739-4472-89e1-796775e0aa3a" (UID: "af59576c-6739-4472-89e1-796775e0aa3a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:22:43.641686 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:43.641656    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af59576c-6739-4472-89e1-796775e0aa3a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "af59576c-6739-4472-89e1-796775e0aa3a" (UID: "af59576c-6739-4472-89e1-796775e0aa3a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:22:43.641862 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:43.641838    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af59576c-6739-4472-89e1-796775e0aa3a-kube-api-access-98fbt" (OuterVolumeSpecName: "kube-api-access-98fbt") pod "af59576c-6739-4472-89e1-796775e0aa3a" (UID: "af59576c-6739-4472-89e1-796775e0aa3a"). InnerVolumeSpecName "kube-api-access-98fbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:22:43.642373 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:43.642340    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af59576c-6739-4472-89e1-796775e0aa3a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "af59576c-6739-4472-89e1-796775e0aa3a" (UID: "af59576c-6739-4472-89e1-796775e0aa3a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:22:43.740129 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:43.740097    2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/af59576c-6739-4472-89e1-796775e0aa3a-console-config\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:22:43.740129 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:43.740124    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af59576c-6739-4472-89e1-796775e0aa3a-trusted-ca-bundle\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:22:43.740129 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:43.740135    2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/af59576c-6739-4472-89e1-796775e0aa3a-console-serving-cert\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:22:43.740349 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:43.740145    2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/af59576c-6739-4472-89e1-796775e0aa3a-console-oauth-config\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:22:43.740349 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:43.740154    2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/af59576c-6739-4472-89e1-796775e0aa3a-service-ca\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:22:43.740349 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:43.740162    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98fbt\" (UniqueName: \"kubernetes.io/projected/af59576c-6739-4472-89e1-796775e0aa3a-kube-api-access-98fbt\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:22:43.740349 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:43.740170    2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/af59576c-6739-4472-89e1-796775e0aa3a-oauth-serving-cert\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:22:44.284467 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:44.284429    2581 generic.go:356] "Generic (PLEG): container finished" podID="af59576c-6739-4472-89e1-796775e0aa3a" containerID="f6aae31e758caef7472f056f3594fc62b10991a83e8c8a8dd51ca5ad05055b92" exitCode=0
Sep 11 14:22:44.284644 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:44.284477    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65c7b69c6c-nlhdj"
Sep 11 14:22:44.284644 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:44.284520    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65c7b69c6c-nlhdj" event={"ID":"af59576c-6739-4472-89e1-796775e0aa3a","Type":"ContainerDied","Data":"f6aae31e758caef7472f056f3594fc62b10991a83e8c8a8dd51ca5ad05055b92"}
Sep 11 14:22:44.284644 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:44.284574    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65c7b69c6c-nlhdj" event={"ID":"af59576c-6739-4472-89e1-796775e0aa3a","Type":"ContainerDied","Data":"dc7b1b61a692f93896274f6d1795e0bc3410dc6897d6a8021e3ea003ad0fb37e"}
Sep 11 14:22:44.284644 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:44.284595    2581 scope.go:117] "RemoveContainer" containerID="f6aae31e758caef7472f056f3594fc62b10991a83e8c8a8dd51ca5ad05055b92"
Sep 11 14:22:44.293109 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:44.293089    2581 scope.go:117] "RemoveContainer" containerID="f6aae31e758caef7472f056f3594fc62b10991a83e8c8a8dd51ca5ad05055b92"
Sep 11 14:22:44.293377 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:22:44.293356    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6aae31e758caef7472f056f3594fc62b10991a83e8c8a8dd51ca5ad05055b92\": container with ID starting with f6aae31e758caef7472f056f3594fc62b10991a83e8c8a8dd51ca5ad05055b92 not found: ID does not exist" containerID="f6aae31e758caef7472f056f3594fc62b10991a83e8c8a8dd51ca5ad05055b92"
Sep 11 14:22:44.293419 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:44.293385    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6aae31e758caef7472f056f3594fc62b10991a83e8c8a8dd51ca5ad05055b92"} err="failed to get container status \"f6aae31e758caef7472f056f3594fc62b10991a83e8c8a8dd51ca5ad05055b92\": rpc error: code = NotFound desc = could not find container \"f6aae31e758caef7472f056f3594fc62b10991a83e8c8a8dd51ca5ad05055b92\": container with ID starting with f6aae31e758caef7472f056f3594fc62b10991a83e8c8a8dd51ca5ad05055b92 not found: ID does not exist"
Sep 11 14:22:44.317013 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:44.316989    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65c7b69c6c-nlhdj"]
Sep 11 14:22:44.323834 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:44.323811    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-65c7b69c6c-nlhdj"]
Sep 11 14:22:44.552087 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:44.551985    2581 patch_prober.go:28] interesting pod/console-65c7b69c6c-nlhdj container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.132.0.17:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 11 14:22:44.552087 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:44.552069    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/console-65c7b69c6c-nlhdj" podUID="af59576c-6739-4472-89e1-796775e0aa3a" containerName="console" probeResult="failure" output="Get \"https://10.132.0.17:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 11 14:22:45.499126 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:22:45.499093    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af59576c-6739-4472-89e1-796775e0aa3a" path="/var/lib/kubelet/pods/af59576c-6739-4472-89e1-796775e0aa3a/volumes"
Sep 11 14:24:15.668863 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:24:15.668780    2581 generic.go:356] "Generic (PLEG): container finished" podID="f0948f34-1747-491e-b499-fe02725bc43c" containerID="509cd64b4e1b3becd3143a442776ed4f95ecc3c49908ef34c38099f5e3c3cf39" exitCode=1
Sep 11 14:24:15.668863 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:24:15.668852    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-744bf4ff48-8rn2q" event={"ID":"f0948f34-1747-491e-b499-fe02725bc43c","Type":"ContainerDied","Data":"509cd64b4e1b3becd3143a442776ed4f95ecc3c49908ef34c38099f5e3c3cf39"}
Sep 11 14:24:15.669325 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:24:15.669190    2581 scope.go:117] "RemoveContainer" containerID="509cd64b4e1b3becd3143a442776ed4f95ecc3c49908ef34c38099f5e3c3cf39"
Sep 11 14:24:16.160715 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:24:16.160684    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-744bf4ff48-8rn2q"
Sep 11 14:24:16.675254 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:24:16.675216    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-744bf4ff48-8rn2q" event={"ID":"f0948f34-1747-491e-b499-fe02725bc43c","Type":"ContainerStarted","Data":"e7897c2401b0f30e84fad05376ddade2694aeb91fb2d3dbc5220034d7148d642"}
Sep 11 14:24:16.675695 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:24:16.675297    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-744bf4ff48-8rn2q"
Sep 11 14:24:16.676667 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:24:16.676645    2581 generic.go:356] "Generic (PLEG): container finished" podID="f6ded6ea-11f1-49f8-bd47-cc2fe4016904" containerID="effa745febab88dd9b3c71570fa38ae50e1bec62914e9b70e54f6a940aba01b0" exitCode=1
Sep 11 14:24:16.676764 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:24:16.676692    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-v2qsw" event={"ID":"f6ded6ea-11f1-49f8-bd47-cc2fe4016904","Type":"ContainerDied","Data":"effa745febab88dd9b3c71570fa38ae50e1bec62914e9b70e54f6a940aba01b0"}
Sep 11 14:24:16.676945 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:24:16.676934    2581 scope.go:117] "RemoveContainer" containerID="effa745febab88dd9b3c71570fa38ae50e1bec62914e9b70e54f6a940aba01b0"
Sep 11 14:24:17.682873 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:24:17.682833    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-v2qsw" event={"ID":"f6ded6ea-11f1-49f8-bd47-cc2fe4016904","Type":"ContainerStarted","Data":"cb7a3c2a7afaf12a9b03dac1deb7f7872e897fab1b30e959a4eaf27c0a25ee86"}
Sep 11 14:24:17.683293 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:24:17.683126    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-v2qsw"
Sep 11 14:24:18.924211 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:24:18.924157    2581 controller.go:195] "Failed to update lease" err="Put \"https://a75f141b2f27e4f3f980fa6204891153-e1e569fb23f34ba2.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-135-150.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 11 14:24:20.296154 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:24:20.296126    2581 controller.go:195] "Failed to update lease" err="Operation cannot be fulfilled on leases.coordination.k8s.io \"ip-10-0-135-150.ec2.internal\": the object has been modified; please apply your changes to the latest version and try again"
Sep 11 14:24:27.685449 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:24:27.685413    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-744bf4ff48-8rn2q"
Sep 11 14:24:28.689242 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:24:28.689207    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-v2qsw"
Sep 11 14:25:08.338862 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:08.338827    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hxbs8"]
Sep 11 14:25:08.339375 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:08.339198    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af59576c-6739-4472-89e1-796775e0aa3a" containerName="console"
Sep 11 14:25:08.339375 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:08.339212    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="af59576c-6739-4472-89e1-796775e0aa3a" containerName="console"
Sep 11 14:25:08.339375 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:08.339266    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="af59576c-6739-4472-89e1-796775e0aa3a" containerName="console"
Sep 11 14:25:08.342189 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:08.342171    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hxbs8"
Sep 11 14:25:08.347395 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:08.347369    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-kf8rb\""
Sep 11 14:25:08.373498 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:08.373468    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hxbs8"]
Sep 11 14:25:08.485443 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:08.485402    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cc78453-02ad-45d4-8efe-b247d33d55d4-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hxbs8\" (UID: \"2cc78453-02ad-45d4-8efe-b247d33d55d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hxbs8"
Sep 11 14:25:08.485443 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:08.485442    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csxqp\" (UniqueName: \"kubernetes.io/projected/2cc78453-02ad-45d4-8efe-b247d33d55d4-kube-api-access-csxqp\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hxbs8\" (UID: \"2cc78453-02ad-45d4-8efe-b247d33d55d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hxbs8"
Sep 11 14:25:08.485666 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:08.485462    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cc78453-02ad-45d4-8efe-b247d33d55d4-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hxbs8\" (UID: \"2cc78453-02ad-45d4-8efe-b247d33d55d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hxbs8"
Sep 11 14:25:08.485666 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:08.485546    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cc78453-02ad-45d4-8efe-b247d33d55d4-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hxbs8\" (UID: \"2cc78453-02ad-45d4-8efe-b247d33d55d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hxbs8"
Sep 11 14:25:08.485666 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:08.485579    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cc78453-02ad-45d4-8efe-b247d33d55d4-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hxbs8\" (UID: \"2cc78453-02ad-45d4-8efe-b247d33d55d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hxbs8"
Sep 11 14:25:08.485666 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:08.485601    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cc78453-02ad-45d4-8efe-b247d33d55d4-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hxbs8\" (UID: \"2cc78453-02ad-45d4-8efe-b247d33d55d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hxbs8"
Sep 11 14:25:08.485811 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:08.485683    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cc78453-02ad-45d4-8efe-b247d33d55d4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hxbs8\" (UID: \"2cc78453-02ad-45d4-8efe-b247d33d55d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hxbs8"
Sep 11 14:25:08.485811 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:08.485726    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cc78453-02ad-45d4-8efe-b247d33d55d4-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hxbs8\" (UID: \"2cc78453-02ad-45d4-8efe-b247d33d55d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hxbs8"
Sep 11 14:25:08.485811 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:08.485756    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cc78453-02ad-45d4-8efe-b247d33d55d4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hxbs8\" (UID: \"2cc78453-02ad-45d4-8efe-b247d33d55d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hxbs8"
Sep 11 14:25:08.586787 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:08.586756    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cc78453-02ad-45d4-8efe-b247d33d55d4-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hxbs8\" (UID: \"2cc78453-02ad-45d4-8efe-b247d33d55d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hxbs8"
Sep 11 14:25:08.586986 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:08.586795    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cc78453-02ad-45d4-8efe-b247d33d55d4-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hxbs8\" (UID: \"2cc78453-02ad-45d4-8efe-b247d33d55d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hxbs8"
Sep 11 14:25:08.586986 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:08.586825    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cc78453-02ad-45d4-8efe-b247d33d55d4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hxbs8\" (UID: \"2cc78453-02ad-45d4-8efe-b247d33d55d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hxbs8"
Sep 11 14:25:08.586986 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:08.586855    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cc78453-02ad-45d4-8efe-b247d33d55d4-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hxbs8\" (UID: \"2cc78453-02ad-45d4-8efe-b247d33d55d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hxbs8"
Sep 11 14:25:08.586986 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:08.586881    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cc78453-02ad-45d4-8efe-b247d33d55d4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hxbs8\" (UID: \"2cc78453-02ad-45d4-8efe-b247d33d55d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hxbs8"
Sep 11 14:25:08.586986 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:08.586925    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cc78453-02ad-45d4-8efe-b247d33d55d4-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hxbs8\" (UID: \"2cc78453-02ad-45d4-8efe-b247d33d55d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hxbs8"
Sep 11 14:25:08.586986 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:08.586949    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csxqp\" (UniqueName: \"kubernetes.io/projected/2cc78453-02ad-45d4-8efe-b247d33d55d4-kube-api-access-csxqp\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hxbs8\" (UID: \"2cc78453-02ad-45d4-8efe-b247d33d55d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hxbs8"
Sep 11 14:25:08.587311 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:08.586988    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cc78453-02ad-45d4-8efe-b247d33d55d4-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hxbs8\" (UID: \"2cc78453-02ad-45d4-8efe-b247d33d55d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hxbs8"
Sep 11 14:25:08.587311 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:08.587038    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cc78453-02ad-45d4-8efe-b247d33d55d4-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hxbs8\" (UID: \"2cc78453-02ad-45d4-8efe-b247d33d55d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hxbs8"
Sep 11 14:25:08.587311 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:08.587246    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cc78453-02ad-45d4-8efe-b247d33d55d4-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hxbs8\" (UID: \"2cc78453-02ad-45d4-8efe-b247d33d55d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hxbs8"
Sep 11 14:25:08.587469 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:08.587451    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cc78453-02ad-45d4-8efe-b247d33d55d4-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hxbs8\" (UID: \"2cc78453-02ad-45d4-8efe-b247d33d55d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hxbs8"
Sep 11 14:25:08.587643 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:08.587626    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cc78453-02ad-45d4-8efe-b247d33d55d4-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hxbs8\" (UID: \"2cc78453-02ad-45d4-8efe-b247d33d55d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hxbs8"
Sep 11 14:25:08.587725 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:08.587699    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cc78453-02ad-45d4-8efe-b247d33d55d4-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hxbs8\" (UID: \"2cc78453-02ad-45d4-8efe-b247d33d55d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hxbs8"
Sep 11 14:25:08.587947 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:08.587916    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cc78453-02ad-45d4-8efe-b247d33d55d4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hxbs8\" (UID: \"2cc78453-02ad-45d4-8efe-b247d33d55d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hxbs8"
Sep 11 14:25:08.589587 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:08.589535    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cc78453-02ad-45d4-8efe-b247d33d55d4-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hxbs8\" (UID: \"2cc78453-02ad-45d4-8efe-b247d33d55d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hxbs8"
Sep 11 14:25:08.589657 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:08.589641    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cc78453-02ad-45d4-8efe-b247d33d55d4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hxbs8\" (UID: \"2cc78453-02ad-45d4-8efe-b247d33d55d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hxbs8"
Sep 11 14:25:08.598180 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:08.598159    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csxqp\" (UniqueName: \"kubernetes.io/projected/2cc78453-02ad-45d4-8efe-b247d33d55d4-kube-api-access-csxqp\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hxbs8\" (UID: \"2cc78453-02ad-45d4-8efe-b247d33d55d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hxbs8"
Sep 11 14:25:08.599811 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:08.599789    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cc78453-02ad-45d4-8efe-b247d33d55d4-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-hxbs8\" (UID: \"2cc78453-02ad-45d4-8efe-b247d33d55d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hxbs8"
Sep 11 14:25:08.652537 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:08.652504    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hxbs8"
Sep 11 14:25:08.948742 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:08.948659    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hxbs8"]
Sep 11 14:25:08.951634 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:25:08.951605    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cc78453_02ad_45d4_8efe_b247d33d55d4.slice/crio-11891d78fadff73e446456ce7342abdcc7ed552882d20eaf36275985142b0eda WatchSource:0}: Error finding container 11891d78fadff73e446456ce7342abdcc7ed552882d20eaf36275985142b0eda: Status 404 returned error can't find the container with id 11891d78fadff73e446456ce7342abdcc7ed552882d20eaf36275985142b0eda
Sep 11 14:25:08.953722 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:08.953687    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:25:08.953811 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:08.953774    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:25:08.953871 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:08.953820    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:25:09.908323 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:09.908286    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hxbs8" event={"ID":"2cc78453-02ad-45d4-8efe-b247d33d55d4","Type":"ContainerStarted","Data":"75ae120fe9fda369d4eb3a3298039634ad936a5829c1b113071d60faafb8ab4b"}
Sep 11 14:25:09.908733 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:09.908341    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hxbs8" event={"ID":"2cc78453-02ad-45d4-8efe-b247d33d55d4","Type":"ContainerStarted","Data":"11891d78fadff73e446456ce7342abdcc7ed552882d20eaf36275985142b0eda"}
Sep 11 14:25:09.945050 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:09.944991    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hxbs8" podStartSLOduration=1.944956784 podStartE2EDuration="1.944956784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:25:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:25:09.94353909 +0000 UTC m=+613.094089154" watchObservedRunningTime="2026-09-11 14:25:09.944956784 +0000 UTC m=+613.095506848"
Sep 11 14:25:10.653162 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:10.653127    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hxbs8"
Sep 11 14:25:10.657705 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:10.657679    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hxbs8"
Sep 11 14:25:10.912392 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:10.912304    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hxbs8"
Sep 11 14:25:10.913303 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:10.913285    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hxbs8"
Sep 11 14:25:24.385957 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:24.385864    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-qmnpw"]
Sep 11 14:25:24.389426 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:24.389397    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-qmnpw"
Sep 11 14:25:24.392849 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:24.392826    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 11 14:25:24.394147 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:24.394126    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 11 14:25:24.394147 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:24.394144    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-7mdtb\""
Sep 11 14:25:24.400102 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:24.400082    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-qmnpw"]
Sep 11 14:25:24.426072 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:24.426043    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6zz5\" (UniqueName: \"kubernetes.io/projected/63de0bbc-1b09-4624-843d-bf595cec98a7-kube-api-access-z6zz5\") pod \"authorino-7bcfd8f9cd-qmnpw\" (UID: \"63de0bbc-1b09-4624-843d-bf595cec98a7\") " pod="kuadrant-system/authorino-7bcfd8f9cd-qmnpw"
Sep 11 14:25:24.526715 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:24.526676    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6zz5\" (UniqueName: \"kubernetes.io/projected/63de0bbc-1b09-4624-843d-bf595cec98a7-kube-api-access-z6zz5\") pod \"authorino-7bcfd8f9cd-qmnpw\" (UID: \"63de0bbc-1b09-4624-843d-bf595cec98a7\") " pod="kuadrant-system/authorino-7bcfd8f9cd-qmnpw"
Sep 11 14:25:24.544874 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:24.544839    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6zz5\" (UniqueName: \"kubernetes.io/projected/63de0bbc-1b09-4624-843d-bf595cec98a7-kube-api-access-z6zz5\") pod \"authorino-7bcfd8f9cd-qmnpw\" (UID: \"63de0bbc-1b09-4624-843d-bf595cec98a7\") " pod="kuadrant-system/authorino-7bcfd8f9cd-qmnpw"
Sep 11 14:25:24.699346 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:24.699257    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-qmnpw"
Sep 11 14:25:24.994407 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:24.994378    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-qmnpw"]
Sep 11 14:25:24.997736 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:25:24.997712    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod63de0bbc_1b09_4624_843d_bf595cec98a7.slice/crio-d41820511f3ff367a018591d14a9a6bed231d9b2eb8a446a8c35da4d813905a2 WatchSource:0}: Error finding container d41820511f3ff367a018591d14a9a6bed231d9b2eb8a446a8c35da4d813905a2: Status 404 returned error can't find the container with id d41820511f3ff367a018591d14a9a6bed231d9b2eb8a446a8c35da4d813905a2
Sep 11 14:25:25.976421 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:25.976383    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-qmnpw" event={"ID":"63de0bbc-1b09-4624-843d-bf595cec98a7","Type":"ContainerStarted","Data":"d41820511f3ff367a018591d14a9a6bed231d9b2eb8a446a8c35da4d813905a2"}
Sep 11 14:25:27.536879 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:27.536839    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-qmnpw"]
Sep 11 14:25:29.996749 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:29.996715    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-qmnpw" event={"ID":"63de0bbc-1b09-4624-843d-bf595cec98a7","Type":"ContainerStarted","Data":"08197c49d6db981f19e2ee98b9489c6f1950a0d8273b9d963ccd0d5752371ba9"}
Sep 11 14:25:29.997193 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:29.996804    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-qmnpw" podUID="63de0bbc-1b09-4624-843d-bf595cec98a7" containerName="authorino" containerID="cri-o://08197c49d6db981f19e2ee98b9489c6f1950a0d8273b9d963ccd0d5752371ba9" gracePeriod=30
Sep 11 14:25:30.020292 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:30.020240    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-qmnpw" podStartSLOduration=1.5289258380000001 podStartE2EDuration="6.020223853s" totalImagesPullingTime="4.491298015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:25:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:25:30.019177479 +0000 UTC m=+633.169727545" watchObservedRunningTime="2026-09-11 14:25:30.020223853 +0000 UTC m=+633.170773916"
Sep 11 14:25:30.241164 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:30.241139    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-qmnpw"
Sep 11 14:25:30.277436 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:30.277365    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6zz5\" (UniqueName: \"kubernetes.io/projected/63de0bbc-1b09-4624-843d-bf595cec98a7-kube-api-access-z6zz5\") pod \"63de0bbc-1b09-4624-843d-bf595cec98a7\" (UID: \"63de0bbc-1b09-4624-843d-bf595cec98a7\") "
Sep 11 14:25:30.279445 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:30.279417    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63de0bbc-1b09-4624-843d-bf595cec98a7-kube-api-access-z6zz5" (OuterVolumeSpecName: "kube-api-access-z6zz5") pod "63de0bbc-1b09-4624-843d-bf595cec98a7" (UID: "63de0bbc-1b09-4624-843d-bf595cec98a7"). InnerVolumeSpecName "kube-api-access-z6zz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:25:30.378728 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:30.378679    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6zz5\" (UniqueName: \"kubernetes.io/projected/63de0bbc-1b09-4624-843d-bf595cec98a7-kube-api-access-z6zz5\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:25:31.002625 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:31.002594    2581 generic.go:356] "Generic (PLEG): container finished" podID="63de0bbc-1b09-4624-843d-bf595cec98a7" containerID="08197c49d6db981f19e2ee98b9489c6f1950a0d8273b9d963ccd0d5752371ba9" exitCode=0
Sep 11 14:25:31.003073 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:31.002636    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-qmnpw" event={"ID":"63de0bbc-1b09-4624-843d-bf595cec98a7","Type":"ContainerDied","Data":"08197c49d6db981f19e2ee98b9489c6f1950a0d8273b9d963ccd0d5752371ba9"}
Sep 11 14:25:31.003073 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:31.002663    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-qmnpw" event={"ID":"63de0bbc-1b09-4624-843d-bf595cec98a7","Type":"ContainerDied","Data":"d41820511f3ff367a018591d14a9a6bed231d9b2eb8a446a8c35da4d813905a2"}
Sep 11 14:25:31.003073 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:31.002680    2581 scope.go:117] "RemoveContainer" containerID="08197c49d6db981f19e2ee98b9489c6f1950a0d8273b9d963ccd0d5752371ba9"
Sep 11 14:25:31.003073 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:31.002636    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-qmnpw"
Sep 11 14:25:31.012071 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:31.012050    2581 scope.go:117] "RemoveContainer" containerID="08197c49d6db981f19e2ee98b9489c6f1950a0d8273b9d963ccd0d5752371ba9"
Sep 11 14:25:31.012349 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:25:31.012333    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08197c49d6db981f19e2ee98b9489c6f1950a0d8273b9d963ccd0d5752371ba9\": container with ID starting with 08197c49d6db981f19e2ee98b9489c6f1950a0d8273b9d963ccd0d5752371ba9 not found: ID does not exist" containerID="08197c49d6db981f19e2ee98b9489c6f1950a0d8273b9d963ccd0d5752371ba9"
Sep 11 14:25:31.012394 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:31.012359    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08197c49d6db981f19e2ee98b9489c6f1950a0d8273b9d963ccd0d5752371ba9"} err="failed to get container status \"08197c49d6db981f19e2ee98b9489c6f1950a0d8273b9d963ccd0d5752371ba9\": rpc error: code = NotFound desc = could not find container \"08197c49d6db981f19e2ee98b9489c6f1950a0d8273b9d963ccd0d5752371ba9\": container with ID starting with 08197c49d6db981f19e2ee98b9489c6f1950a0d8273b9d963ccd0d5752371ba9 not found: ID does not exist"
Sep 11 14:25:31.033929 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:31.033897    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-qmnpw"]
Sep 11 14:25:31.041153 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:31.041129    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-qmnpw"]
Sep 11 14:25:31.498958 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:25:31.498913    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63de0bbc-1b09-4624-843d-bf595cec98a7" path="/var/lib/kubelet/pods/63de0bbc-1b09-4624-843d-bf595cec98a7/volumes"
Sep 11 14:26:53.363181 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:26:53.363142    2581 generic.go:356] "Generic (PLEG): container finished" podID="2cc78453-02ad-45d4-8efe-b247d33d55d4" containerID="75ae120fe9fda369d4eb3a3298039634ad936a5829c1b113071d60faafb8ab4b" exitCode=0
Sep 11 14:26:53.363612 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:26:53.363214    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hxbs8" event={"ID":"2cc78453-02ad-45d4-8efe-b247d33d55d4","Type":"ContainerDied","Data":"75ae120fe9fda369d4eb3a3298039634ad936a5829c1b113071d60faafb8ab4b"}
Sep 11 14:26:53.363612 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:26:53.363582    2581 scope.go:117] "RemoveContainer" containerID="75ae120fe9fda369d4eb3a3298039634ad936a5829c1b113071d60faafb8ab4b"
Sep 11 14:26:53.364558 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:26:53.364527    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:26:53.364652 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:26:53.364617    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:26:53.364712 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:26:53.364664    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:26:54.369687 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:26:54.369655    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hxbs8" event={"ID":"2cc78453-02ad-45d4-8efe-b247d33d55d4","Type":"ContainerStarted","Data":"67fe6edce1c2c4569f7c02ac97f6da3d66dfb4d01154a52c7108542843240c48"}
Sep 11 14:26:54.653089 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:26:54.652988    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hxbs8"
Sep 11 14:26:54.654077 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:26:54.654049    2581 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-hxbs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body=
Sep 11 14:26:54.654207 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:26:54.654095    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hxbs8" podUID="2cc78453-02ad-45d4-8efe-b247d33d55d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused"
Sep 11 14:26:55.653392 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:26:55.653356    2581 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-hxbs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body=
Sep 11 14:26:55.653779 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:26:55.653410    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hxbs8" podUID="2cc78453-02ad-45d4-8efe-b247d33d55d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused"
Sep 11 14:26:55.913343 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:26:55.913243    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hxbs8"
Sep 11 14:26:56.653754 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:26:56.653714    2581 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-hxbs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body=
Sep 11 14:26:56.654187 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:26:56.653772    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hxbs8" podUID="2cc78453-02ad-45d4-8efe-b247d33d55d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused"
Sep 11 14:26:57.653357 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:26:57.653320    2581 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-hxbs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body=
Sep 11 14:26:57.653536 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:26:57.653391    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hxbs8" podUID="2cc78453-02ad-45d4-8efe-b247d33d55d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused"
Sep 11 14:26:58.653607 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:26:58.653571    2581 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-hxbs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body=
Sep 11 14:26:58.654037 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:26:58.653630    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hxbs8" podUID="2cc78453-02ad-45d4-8efe-b247d33d55d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused"
Sep 11 14:26:59.657172 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:26:59.657144    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hxbs8"
Sep 11 14:26:59.658032 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:26:59.658011    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-hxbs8"
Sep 11 14:27:29.118145 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:29.118110    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-pjmfg"]
Sep 11 14:27:29.118633 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:29.118422    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63de0bbc-1b09-4624-843d-bf595cec98a7" containerName="authorino"
Sep 11 14:27:29.118633 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:29.118434    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="63de0bbc-1b09-4624-843d-bf595cec98a7" containerName="authorino"
Sep 11 14:27:29.118633 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:29.118487    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="63de0bbc-1b09-4624-843d-bf595cec98a7" containerName="authorino"
Sep 11 14:27:29.121287 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:29.121269    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pjmfg"
Sep 11 14:27:29.125793 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:29.125769    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 11 14:27:29.127080 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:29.127062    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-m8wqn\""
Sep 11 14:27:29.127178 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:29.127062    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 11 14:27:29.127178 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:29.127127    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 11 14:27:29.142184 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:29.142151    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-pjmfg"]
Sep 11 14:27:29.203115 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:29.203082    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f769f831-73e9-4992-a406-b5aebd8a37e1-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pjmfg\" (UID: \"f769f831-73e9-4992-a406-b5aebd8a37e1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pjmfg"
Sep 11 14:27:29.203115 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:29.203118    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f769f831-73e9-4992-a406-b5aebd8a37e1-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pjmfg\" (UID: \"f769f831-73e9-4992-a406-b5aebd8a37e1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pjmfg"
Sep 11 14:27:29.203364 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:29.203151    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f769f831-73e9-4992-a406-b5aebd8a37e1-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pjmfg\" (UID: \"f769f831-73e9-4992-a406-b5aebd8a37e1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pjmfg"
Sep 11 14:27:29.203364 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:29.203245    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k7km\" (UniqueName: \"kubernetes.io/projected/f769f831-73e9-4992-a406-b5aebd8a37e1-kube-api-access-8k7km\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pjmfg\" (UID: \"f769f831-73e9-4992-a406-b5aebd8a37e1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pjmfg"
Sep 11 14:27:29.203364 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:29.203306    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f769f831-73e9-4992-a406-b5aebd8a37e1-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pjmfg\" (UID: \"f769f831-73e9-4992-a406-b5aebd8a37e1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pjmfg"
Sep 11 14:27:29.203364 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:29.203349    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f769f831-73e9-4992-a406-b5aebd8a37e1-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pjmfg\" (UID: \"f769f831-73e9-4992-a406-b5aebd8a37e1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pjmfg"
Sep 11 14:27:29.304095 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:29.304054    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f769f831-73e9-4992-a406-b5aebd8a37e1-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pjmfg\" (UID: \"f769f831-73e9-4992-a406-b5aebd8a37e1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pjmfg"
Sep 11 14:27:29.304302 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:29.304111    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k7km\" (UniqueName: \"kubernetes.io/projected/f769f831-73e9-4992-a406-b5aebd8a37e1-kube-api-access-8k7km\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pjmfg\" (UID: \"f769f831-73e9-4992-a406-b5aebd8a37e1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pjmfg"
Sep 11 14:27:29.304302 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:29.304142    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f769f831-73e9-4992-a406-b5aebd8a37e1-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pjmfg\" (UID: \"f769f831-73e9-4992-a406-b5aebd8a37e1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pjmfg"
Sep 11 14:27:29.304302 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:29.304165    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f769f831-73e9-4992-a406-b5aebd8a37e1-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pjmfg\" (UID: \"f769f831-73e9-4992-a406-b5aebd8a37e1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pjmfg"
Sep 11 14:27:29.304302 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:29.304197    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f769f831-73e9-4992-a406-b5aebd8a37e1-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pjmfg\" (UID: \"f769f831-73e9-4992-a406-b5aebd8a37e1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pjmfg"
Sep 11 14:27:29.304302 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:29.304214    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f769f831-73e9-4992-a406-b5aebd8a37e1-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pjmfg\" (UID: \"f769f831-73e9-4992-a406-b5aebd8a37e1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pjmfg"
Sep 11 14:27:29.304620 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:29.304597    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f769f831-73e9-4992-a406-b5aebd8a37e1-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pjmfg\" (UID: \"f769f831-73e9-4992-a406-b5aebd8a37e1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pjmfg"
Sep 11 14:27:29.304684 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:29.304626    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f769f831-73e9-4992-a406-b5aebd8a37e1-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pjmfg\" (UID: \"f769f831-73e9-4992-a406-b5aebd8a37e1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pjmfg"
Sep 11 14:27:29.304684 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:29.304658    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f769f831-73e9-4992-a406-b5aebd8a37e1-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pjmfg\" (UID: \"f769f831-73e9-4992-a406-b5aebd8a37e1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pjmfg"
Sep 11 14:27:29.306420 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:29.306402    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f769f831-73e9-4992-a406-b5aebd8a37e1-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pjmfg\" (UID: \"f769f831-73e9-4992-a406-b5aebd8a37e1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pjmfg"
Sep 11 14:27:29.306692 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:29.306673    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f769f831-73e9-4992-a406-b5aebd8a37e1-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pjmfg\" (UID: \"f769f831-73e9-4992-a406-b5aebd8a37e1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pjmfg"
Sep 11 14:27:29.315425 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:29.315403    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k7km\" (UniqueName: \"kubernetes.io/projected/f769f831-73e9-4992-a406-b5aebd8a37e1-kube-api-access-8k7km\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pjmfg\" (UID: \"f769f831-73e9-4992-a406-b5aebd8a37e1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pjmfg"
Sep 11 14:27:29.431185 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:29.431090    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pjmfg"
Sep 11 14:27:29.710416 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:29.710335    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-pjmfg"]
Sep 11 14:27:29.713378 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:27:29.713349    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf769f831_73e9_4992_a406_b5aebd8a37e1.slice/crio-abe3efb0d702b0f98c48f7d19673331b8320c34f1b8cb03f87215db1451cd3be WatchSource:0}: Error finding container abe3efb0d702b0f98c48f7d19673331b8320c34f1b8cb03f87215db1451cd3be: Status 404 returned error can't find the container with id abe3efb0d702b0f98c48f7d19673331b8320c34f1b8cb03f87215db1451cd3be
Sep 11 14:27:29.714645 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:29.714625    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 14:27:30.530876 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:30.530839    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pjmfg" event={"ID":"f769f831-73e9-4992-a406-b5aebd8a37e1","Type":"ContainerStarted","Data":"abe3efb0d702b0f98c48f7d19673331b8320c34f1b8cb03f87215db1451cd3be"}
Sep 11 14:27:31.537562 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:31.537525    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pjmfg" event={"ID":"f769f831-73e9-4992-a406-b5aebd8a37e1","Type":"ContainerStarted","Data":"1a1ec0222432174305706a86bda877ac3e6ac8275d4c3470577ee1285f2e249d"}
Sep 11 14:27:31.569012 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:31.568948    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pjmfg" podStartSLOduration=1.584926013 podStartE2EDuration="2.568930054s" totalImagesPullingTime="984.004041ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:27:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:27:31.568047712 +0000 UTC m=+754.718597775" watchObservedRunningTime="2026-09-11 14:27:31.568930054 +0000 UTC m=+754.719480117"
Sep 11 14:27:39.431277 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:39.431238    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pjmfg"
Sep 11 14:27:39.431680 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:39.431363    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pjmfg"
Sep 11 14:27:39.443079 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:39.443053    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pjmfg"
Sep 11 14:27:39.584238 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:39.584213    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pjmfg"
Sep 11 14:27:43.610351 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:43.610314    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-9ffx4"]
Sep 11 14:27:43.613698 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:43.613680    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-9ffx4"
Sep 11 14:27:43.616916 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:43.616889    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 11 14:27:43.628512 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:43.628486    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-9ffx4"]
Sep 11 14:27:43.735643 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:43.735613    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c677d32f-fae0-46b8-a9b0-de4f57a388ec-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-9ffx4\" (UID: \"c677d32f-fae0-46b8-a9b0-de4f57a388ec\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-9ffx4"
Sep 11 14:27:43.735643 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:43.735647    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c677d32f-fae0-46b8-a9b0-de4f57a388ec-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-9ffx4\" (UID: \"c677d32f-fae0-46b8-a9b0-de4f57a388ec\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-9ffx4"
Sep 11 14:27:43.735855 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:43.735739    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvsxh\" (UniqueName: \"kubernetes.io/projected/c677d32f-fae0-46b8-a9b0-de4f57a388ec-kube-api-access-dvsxh\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-9ffx4\" (UID: \"c677d32f-fae0-46b8-a9b0-de4f57a388ec\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-9ffx4"
Sep 11 14:27:43.735855 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:43.735791    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c677d32f-fae0-46b8-a9b0-de4f57a388ec-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-9ffx4\" (UID: \"c677d32f-fae0-46b8-a9b0-de4f57a388ec\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-9ffx4"
Sep 11 14:27:43.735855 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:43.735834    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c677d32f-fae0-46b8-a9b0-de4f57a388ec-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-9ffx4\" (UID: \"c677d32f-fae0-46b8-a9b0-de4f57a388ec\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-9ffx4"
Sep 11 14:27:43.735855 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:43.735855    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c677d32f-fae0-46b8-a9b0-de4f57a388ec-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-9ffx4\" (UID: \"c677d32f-fae0-46b8-a9b0-de4f57a388ec\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-9ffx4"
Sep 11 14:27:43.836660 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:43.836620    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvsxh\" (UniqueName: \"kubernetes.io/projected/c677d32f-fae0-46b8-a9b0-de4f57a388ec-kube-api-access-dvsxh\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-9ffx4\" (UID: \"c677d32f-fae0-46b8-a9b0-de4f57a388ec\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-9ffx4"
Sep 11 14:27:43.836851 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:43.836680    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c677d32f-fae0-46b8-a9b0-de4f57a388ec-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-9ffx4\" (UID: \"c677d32f-fae0-46b8-a9b0-de4f57a388ec\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-9ffx4"
Sep 11 14:27:43.836851 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:43.836703    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c677d32f-fae0-46b8-a9b0-de4f57a388ec-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-9ffx4\" (UID: \"c677d32f-fae0-46b8-a9b0-de4f57a388ec\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-9ffx4"
Sep 11 14:27:43.836851 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:43.836720    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c677d32f-fae0-46b8-a9b0-de4f57a388ec-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-9ffx4\" (UID: \"c677d32f-fae0-46b8-a9b0-de4f57a388ec\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-9ffx4"
Sep 11 14:27:43.836851 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:43.836742    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c677d32f-fae0-46b8-a9b0-de4f57a388ec-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-9ffx4\" (UID: \"c677d32f-fae0-46b8-a9b0-de4f57a388ec\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-9ffx4"
Sep 11 14:27:43.836851 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:43.836759    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c677d32f-fae0-46b8-a9b0-de4f57a388ec-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-9ffx4\" (UID: \"c677d32f-fae0-46b8-a9b0-de4f57a388ec\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-9ffx4"
Sep 11 14:27:43.837290 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:43.837255    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c677d32f-fae0-46b8-a9b0-de4f57a388ec-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-9ffx4\" (UID: \"c677d32f-fae0-46b8-a9b0-de4f57a388ec\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-9ffx4"
Sep 11 14:27:43.837435 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:43.837262    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c677d32f-fae0-46b8-a9b0-de4f57a388ec-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-9ffx4\" (UID: \"c677d32f-fae0-46b8-a9b0-de4f57a388ec\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-9ffx4"
Sep 11 14:27:43.837435 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:43.837303    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c677d32f-fae0-46b8-a9b0-de4f57a388ec-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-9ffx4\" (UID: \"c677d32f-fae0-46b8-a9b0-de4f57a388ec\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-9ffx4"
Sep 11 14:27:43.839091 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:43.839069    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c677d32f-fae0-46b8-a9b0-de4f57a388ec-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-9ffx4\" (UID: \"c677d32f-fae0-46b8-a9b0-de4f57a388ec\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-9ffx4"
Sep 11 14:27:43.839355 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:43.839336    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c677d32f-fae0-46b8-a9b0-de4f57a388ec-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-9ffx4\" (UID: \"c677d32f-fae0-46b8-a9b0-de4f57a388ec\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-9ffx4"
Sep 11 14:27:43.856156 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:43.856123    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvsxh\" (UniqueName: \"kubernetes.io/projected/c677d32f-fae0-46b8-a9b0-de4f57a388ec-kube-api-access-dvsxh\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-9ffx4\" (UID: \"c677d32f-fae0-46b8-a9b0-de4f57a388ec\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-9ffx4"
Sep 11 14:27:43.923865 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:43.923777    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-9ffx4"
Sep 11 14:27:44.202766 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:44.202738    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-9ffx4"]
Sep 11 14:27:44.204477 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:27:44.204449    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc677d32f_fae0_46b8_a9b0_de4f57a388ec.slice/crio-52c9e94d1e97aab6f5e6651de69e59416e6b2c9a74ac380b7b8849f8f0242190 WatchSource:0}: Error finding container 52c9e94d1e97aab6f5e6651de69e59416e6b2c9a74ac380b7b8849f8f0242190: Status 404 returned error can't find the container with id 52c9e94d1e97aab6f5e6651de69e59416e6b2c9a74ac380b7b8849f8f0242190
Sep 11 14:27:44.598594 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:44.598562    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-9ffx4" event={"ID":"c677d32f-fae0-46b8-a9b0-de4f57a388ec","Type":"ContainerStarted","Data":"fa362bb59853877ad29108e3f0d0ebddb4131a5cdbe346fd9c64a48ec34a7dea"}
Sep 11 14:27:44.598767 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:44.598617    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-9ffx4" event={"ID":"c677d32f-fae0-46b8-a9b0-de4f57a388ec","Type":"ContainerStarted","Data":"52c9e94d1e97aab6f5e6651de69e59416e6b2c9a74ac380b7b8849f8f0242190"}
Sep 11 14:27:44.621540 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:44.621480    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-9ffx4" podStartSLOduration=1.4075312389999999 podStartE2EDuration="1.621437866s" totalImagesPullingTime="213.906627ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:27:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:27:44.620470246 +0000 UTC m=+767.771020337" watchObservedRunningTime="2026-09-11 14:27:44.621437866 +0000 UTC m=+767.771988248"
Sep 11 14:27:53.924581 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:53.924528    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-9ffx4"
Sep 11 14:27:53.924581 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:53.924603    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-9ffx4"
Sep 11 14:27:53.937519 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:53.937490    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-9ffx4"
Sep 11 14:27:54.655435 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:27:54.655401    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-9ffx4"
Sep 11 14:28:27.148584 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:28:27.148496    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-fz7tk"]
Sep 11 14:28:27.151667 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:28:27.151650    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fz7tk"
Sep 11 14:28:27.155010 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:28:27.154990    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 11 14:28:27.182083 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:28:27.182054    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-fz7tk"]
Sep 11 14:28:27.223705 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:28:27.223671    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1d188ff-0917-4693-b832-cd5b85c76c72-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fz7tk\" (UID: \"c1d188ff-0917-4693-b832-cd5b85c76c72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fz7tk"
Sep 11 14:28:27.223705 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:28:27.223717    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1d188ff-0917-4693-b832-cd5b85c76c72-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fz7tk\" (UID: \"c1d188ff-0917-4693-b832-cd5b85c76c72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fz7tk"
Sep 11 14:28:27.223891 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:28:27.223792    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1d188ff-0917-4693-b832-cd5b85c76c72-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fz7tk\" (UID: \"c1d188ff-0917-4693-b832-cd5b85c76c72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fz7tk"
Sep 11 14:28:27.223891 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:28:27.223831    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk4x9\" (UniqueName: \"kubernetes.io/projected/c1d188ff-0917-4693-b832-cd5b85c76c72-kube-api-access-bk4x9\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fz7tk\" (UID: \"c1d188ff-0917-4693-b832-cd5b85c76c72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fz7tk"
Sep 11 14:28:27.223891 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:28:27.223882    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1d188ff-0917-4693-b832-cd5b85c76c72-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fz7tk\" (UID: \"c1d188ff-0917-4693-b832-cd5b85c76c72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fz7tk"
Sep 11 14:28:27.224241 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:28:27.223916    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1d188ff-0917-4693-b832-cd5b85c76c72-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fz7tk\" (UID: \"c1d188ff-0917-4693-b832-cd5b85c76c72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fz7tk"
Sep 11 14:28:27.324896 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:28:27.324856    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1d188ff-0917-4693-b832-cd5b85c76c72-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fz7tk\" (UID: \"c1d188ff-0917-4693-b832-cd5b85c76c72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fz7tk"
Sep 11 14:28:27.324896 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:28:27.324902    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1d188ff-0917-4693-b832-cd5b85c76c72-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fz7tk\" (UID: \"c1d188ff-0917-4693-b832-cd5b85c76c72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fz7tk"
Sep 11 14:28:27.325150 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:28:27.324928    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1d188ff-0917-4693-b832-cd5b85c76c72-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fz7tk\" (UID: \"c1d188ff-0917-4693-b832-cd5b85c76c72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fz7tk"
Sep 11 14:28:27.325150 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:28:27.324953    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1d188ff-0917-4693-b832-cd5b85c76c72-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fz7tk\" (UID: \"c1d188ff-0917-4693-b832-cd5b85c76c72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fz7tk"
Sep 11 14:28:27.325150 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:28:27.325021    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1d188ff-0917-4693-b832-cd5b85c76c72-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fz7tk\" (UID: \"c1d188ff-0917-4693-b832-cd5b85c76c72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fz7tk"
Sep 11 14:28:27.325150 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:28:27.325046    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk4x9\" (UniqueName: \"kubernetes.io/projected/c1d188ff-0917-4693-b832-cd5b85c76c72-kube-api-access-bk4x9\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fz7tk\" (UID: \"c1d188ff-0917-4693-b832-cd5b85c76c72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fz7tk"
Sep 11 14:28:27.325354 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:28:27.325299    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1d188ff-0917-4693-b832-cd5b85c76c72-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fz7tk\" (UID: \"c1d188ff-0917-4693-b832-cd5b85c76c72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fz7tk"
Sep 11 14:28:27.325354 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:28:27.325338    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1d188ff-0917-4693-b832-cd5b85c76c72-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fz7tk\" (UID: \"c1d188ff-0917-4693-b832-cd5b85c76c72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fz7tk"
Sep 11 14:28:27.325455 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:28:27.325411    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1d188ff-0917-4693-b832-cd5b85c76c72-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fz7tk\" (UID: \"c1d188ff-0917-4693-b832-cd5b85c76c72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fz7tk"
Sep 11 14:28:27.327307 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:28:27.327283    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1d188ff-0917-4693-b832-cd5b85c76c72-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fz7tk\" (UID: \"c1d188ff-0917-4693-b832-cd5b85c76c72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fz7tk"
Sep 11 14:28:27.327472 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:28:27.327447    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1d188ff-0917-4693-b832-cd5b85c76c72-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fz7tk\" (UID: \"c1d188ff-0917-4693-b832-cd5b85c76c72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fz7tk"
Sep 11 14:28:27.335746 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:28:27.335716    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk4x9\" (UniqueName: \"kubernetes.io/projected/c1d188ff-0917-4693-b832-cd5b85c76c72-kube-api-access-bk4x9\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-fz7tk\" (UID: \"c1d188ff-0917-4693-b832-cd5b85c76c72\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fz7tk"
Sep 11 14:28:27.461016 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:28:27.460905    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fz7tk"
Sep 11 14:28:27.752770 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:28:27.752731    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc1d188ff_0917_4693_b832_cd5b85c76c72.slice/crio-247e5ae8a08422c2605a45730235c6e8aa252ba55dff8d4d66f448bf9d1fb7f9 WatchSource:0}: Error finding container 247e5ae8a08422c2605a45730235c6e8aa252ba55dff8d4d66f448bf9d1fb7f9: Status 404 returned error can't find the container with id 247e5ae8a08422c2605a45730235c6e8aa252ba55dff8d4d66f448bf9d1fb7f9
Sep 11 14:28:27.753519 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:28:27.753497    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-fz7tk"]
Sep 11 14:28:27.792427 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:28:27.792395    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fz7tk" event={"ID":"c1d188ff-0917-4693-b832-cd5b85c76c72","Type":"ContainerStarted","Data":"247e5ae8a08422c2605a45730235c6e8aa252ba55dff8d4d66f448bf9d1fb7f9"}
Sep 11 14:28:28.798379 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:28:28.798341    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fz7tk" event={"ID":"c1d188ff-0917-4693-b832-cd5b85c76c72","Type":"ContainerStarted","Data":"c009dfd484070ad6950057e5edae718aea7b517c23160b00ebbafc6a7190c80b"}
Sep 11 14:28:28.821982 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:28:28.821923    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fz7tk" podStartSLOduration=1.573799536 podStartE2EDuration="1.821905809s" totalImagesPullingTime="248.106273ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:28:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:28:28.82078575 +0000 UTC m=+811.971335814" watchObservedRunningTime="2026-09-11 14:28:28.821905809 +0000 UTC m=+811.972455874"
Sep 11 14:28:37.461884 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:28:37.461842    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fz7tk"
Sep 11 14:28:37.462346 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:28:37.461900    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fz7tk"
Sep 11 14:28:37.474391 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:28:37.474365    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fz7tk"
Sep 11 14:28:37.850060 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:28:37.850027    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-fz7tk"
Sep 11 14:30:56.256182 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:30:56.256148    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg"]
Sep 11 14:30:56.258192 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:30:56.258174    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg"
Sep 11 14:30:56.261251 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:30:56.261224    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-e85961-openshift-default-dockercfg-kp9bj\""
Sep 11 14:30:56.279812 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:30:56.279782    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg"]
Sep 11 14:30:56.419046 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:30:56.419009    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0984e71-aec3-4372-b101-ecf16c20488a-istio-envoy\") pod \"e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg\" (UID: \"b0984e71-aec3-4372-b101-ecf16c20488a\") " pod="openshift-ingress/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg"
Sep 11 14:30:56.419243 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:30:56.419120    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0984e71-aec3-4372-b101-ecf16c20488a-istio-data\") pod \"e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg\" (UID: \"b0984e71-aec3-4372-b101-ecf16c20488a\") " pod="openshift-ingress/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg"
Sep 11 14:30:56.419243 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:30:56.419174    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0984e71-aec3-4372-b101-ecf16c20488a-istiod-ca-cert\") pod \"e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg\" (UID: \"b0984e71-aec3-4372-b101-ecf16c20488a\") " pod="openshift-ingress/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg"
Sep 11 14:30:56.419352 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:30:56.419244    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0984e71-aec3-4372-b101-ecf16c20488a-credential-socket\") pod \"e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg\" (UID: \"b0984e71-aec3-4372-b101-ecf16c20488a\") " pod="openshift-ingress/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg"
Sep 11 14:30:56.419352 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:30:56.419303    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmkpn\" (UniqueName: \"kubernetes.io/projected/b0984e71-aec3-4372-b101-ecf16c20488a-kube-api-access-wmkpn\") pod \"e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg\" (UID: \"b0984e71-aec3-4372-b101-ecf16c20488a\") " pod="openshift-ingress/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg"
Sep 11 14:30:56.419456 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:30:56.419351    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0984e71-aec3-4372-b101-ecf16c20488a-workload-certs\") pod \"e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg\" (UID: \"b0984e71-aec3-4372-b101-ecf16c20488a\") " pod="openshift-ingress/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg"
Sep 11 14:30:56.419456 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:30:56.419382    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0984e71-aec3-4372-b101-ecf16c20488a-istio-token\") pod \"e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg\" (UID: \"b0984e71-aec3-4372-b101-ecf16c20488a\") " pod="openshift-ingress/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg"
Sep 11 14:30:56.419456 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:30:56.419417    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0984e71-aec3-4372-b101-ecf16c20488a-workload-socket\") pod \"e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg\" (UID: \"b0984e71-aec3-4372-b101-ecf16c20488a\") " pod="openshift-ingress/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg"
Sep 11 14:30:56.419604 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:30:56.419458    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0984e71-aec3-4372-b101-ecf16c20488a-istio-podinfo\") pod \"e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg\" (UID: \"b0984e71-aec3-4372-b101-ecf16c20488a\") " pod="openshift-ingress/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg"
Sep 11 14:30:56.520862 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:30:56.520775    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0984e71-aec3-4372-b101-ecf16c20488a-istio-data\") pod \"e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg\" (UID: \"b0984e71-aec3-4372-b101-ecf16c20488a\") " pod="openshift-ingress/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg"
Sep 11 14:30:56.520862 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:30:56.520820    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0984e71-aec3-4372-b101-ecf16c20488a-istiod-ca-cert\") pod \"e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg\" (UID: \"b0984e71-aec3-4372-b101-ecf16c20488a\") " pod="openshift-ingress/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg"
Sep 11 14:30:56.520862 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:30:56.520838    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0984e71-aec3-4372-b101-ecf16c20488a-credential-socket\") pod \"e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg\" (UID: \"b0984e71-aec3-4372-b101-ecf16c20488a\") " pod="openshift-ingress/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg"
Sep 11 14:30:56.521174 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:30:56.520955    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmkpn\" (UniqueName: \"kubernetes.io/projected/b0984e71-aec3-4372-b101-ecf16c20488a-kube-api-access-wmkpn\") pod \"e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg\" (UID: \"b0984e71-aec3-4372-b101-ecf16c20488a\") " pod="openshift-ingress/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg"
Sep 11 14:30:56.521174 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:30:56.521058    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0984e71-aec3-4372-b101-ecf16c20488a-workload-certs\") pod \"e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg\" (UID: \"b0984e71-aec3-4372-b101-ecf16c20488a\") " pod="openshift-ingress/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg"
Sep 11 14:30:56.521174 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:30:56.521097    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0984e71-aec3-4372-b101-ecf16c20488a-istio-token\") pod \"e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg\" (UID: \"b0984e71-aec3-4372-b101-ecf16c20488a\") " pod="openshift-ingress/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg"
Sep 11 14:30:56.521174 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:30:56.521125    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0984e71-aec3-4372-b101-ecf16c20488a-workload-socket\") pod \"e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg\" (UID: \"b0984e71-aec3-4372-b101-ecf16c20488a\") " pod="openshift-ingress/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg"
Sep 11 14:30:56.521376 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:30:56.521178    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0984e71-aec3-4372-b101-ecf16c20488a-istio-podinfo\") pod \"e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg\" (UID: \"b0984e71-aec3-4372-b101-ecf16c20488a\") " pod="openshift-ingress/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg"
Sep 11 14:30:56.521376 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:30:56.521226    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0984e71-aec3-4372-b101-ecf16c20488a-istio-envoy\") pod \"e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg\" (UID: \"b0984e71-aec3-4372-b101-ecf16c20488a\") " pod="openshift-ingress/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg"
Sep 11 14:30:56.521376 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:30:56.521225    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0984e71-aec3-4372-b101-ecf16c20488a-credential-socket\") pod \"e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg\" (UID: \"b0984e71-aec3-4372-b101-ecf16c20488a\") " pod="openshift-ingress/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg"
Sep 11 14:30:56.521376 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:30:56.521319    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0984e71-aec3-4372-b101-ecf16c20488a-istio-data\") pod \"e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg\" (UID: \"b0984e71-aec3-4372-b101-ecf16c20488a\") " pod="openshift-ingress/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg"
Sep 11 14:30:56.521536 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:30:56.521376    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0984e71-aec3-4372-b101-ecf16c20488a-workload-certs\") pod \"e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg\" (UID: \"b0984e71-aec3-4372-b101-ecf16c20488a\") " pod="openshift-ingress/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg"
Sep 11 14:30:56.521536 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:30:56.521487    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0984e71-aec3-4372-b101-ecf16c20488a-workload-socket\") pod \"e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg\" (UID: \"b0984e71-aec3-4372-b101-ecf16c20488a\") " pod="openshift-ingress/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg"
Sep 11 14:30:56.521610 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:30:56.521576    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0984e71-aec3-4372-b101-ecf16c20488a-istiod-ca-cert\") pod \"e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg\" (UID: \"b0984e71-aec3-4372-b101-ecf16c20488a\") " pod="openshift-ingress/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg"
Sep 11 14:30:56.523404 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:30:56.523382    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0984e71-aec3-4372-b101-ecf16c20488a-istio-envoy\") pod \"e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg\" (UID: \"b0984e71-aec3-4372-b101-ecf16c20488a\") " pod="openshift-ingress/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg"
Sep 11 14:30:56.523587 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:30:56.523571    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0984e71-aec3-4372-b101-ecf16c20488a-istio-podinfo\") pod \"e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg\" (UID: \"b0984e71-aec3-4372-b101-ecf16c20488a\") " pod="openshift-ingress/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg"
Sep 11 14:30:56.531414 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:30:56.531391    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0984e71-aec3-4372-b101-ecf16c20488a-istio-token\") pod \"e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg\" (UID: \"b0984e71-aec3-4372-b101-ecf16c20488a\") " pod="openshift-ingress/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg"
Sep 11 14:30:56.532277 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:30:56.532253    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmkpn\" (UniqueName: \"kubernetes.io/projected/b0984e71-aec3-4372-b101-ecf16c20488a-kube-api-access-wmkpn\") pod \"e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg\" (UID: \"b0984e71-aec3-4372-b101-ecf16c20488a\") " pod="openshift-ingress/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg"
Sep 11 14:30:56.568981 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:30:56.568936    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg"
Sep 11 14:30:56.856502 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:30:56.856436    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg"]
Sep 11 14:30:56.859268 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:30:56.859239    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0984e71_aec3_4372_b101_ecf16c20488a.slice/crio-ed1916cfbd0f370d630ea29997019a1722f63c509b86ad6a15ff749219e52be2 WatchSource:0}: Error finding container ed1916cfbd0f370d630ea29997019a1722f63c509b86ad6a15ff749219e52be2: Status 404 returned error can't find the container with id ed1916cfbd0f370d630ea29997019a1722f63c509b86ad6a15ff749219e52be2
Sep 11 14:30:56.864015 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:30:56.863965    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:30:56.864082 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:30:56.864056    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:30:56.864127 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:30:56.864085    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:30:57.477702 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:30:57.477662    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg" event={"ID":"b0984e71-aec3-4372-b101-ecf16c20488a","Type":"ContainerStarted","Data":"b21e822f719dcfcd695ac024cd3876796f8f18297affd36803174cd42777ee4f"}
Sep 11 14:30:57.478112 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:30:57.477725    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg" event={"ID":"b0984e71-aec3-4372-b101-ecf16c20488a","Type":"ContainerStarted","Data":"ed1916cfbd0f370d630ea29997019a1722f63c509b86ad6a15ff749219e52be2"}
Sep 11 14:30:57.506540 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:30:57.506493    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg" podStartSLOduration=1.506476779 podStartE2EDuration="1.506476779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:30:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:30:57.505117243 +0000 UTC m=+960.655667308" watchObservedRunningTime="2026-09-11 14:30:57.506476779 +0000 UTC m=+960.657026845"
Sep 11 14:30:57.569288 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:30:57.569251    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg"
Sep 11 14:30:57.574626 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:30:57.574599    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg"
Sep 11 14:30:58.483050 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:30:58.483019    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg"
Sep 11 14:30:58.484162 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:30:58.484146    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg"
Sep 11 14:31:04.349363 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:04.349324    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-ca8d752a-kserve-5f75475f98-5bmm9"]
Sep 11 14:31:04.352232 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:04.352202    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-ca8d752a-kserve-5f75475f98-5bmm9"
Sep 11 14:31:04.355165 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:04.355142    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-ca8d752a-kserve-self-signed-certs\""
Sep 11 14:31:04.372598 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:04.372562    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-ca8d752a-kserve-5f75475f98-5bmm9"]
Sep 11 14:31:04.498589 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:04.498546    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb0cb030-aed3-4445-b7b0-6ad3326effb6-dshm\") pod \"e2e-conflict-a-ca8d752a-kserve-5f75475f98-5bmm9\" (UID: \"eb0cb030-aed3-4445-b7b0-6ad3326effb6\") " pod="llm/e2e-conflict-a-ca8d752a-kserve-5f75475f98-5bmm9"
Sep 11 14:31:04.498589 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:04.498591    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jztg\" (UniqueName: \"kubernetes.io/projected/eb0cb030-aed3-4445-b7b0-6ad3326effb6-kube-api-access-2jztg\") pod \"e2e-conflict-a-ca8d752a-kserve-5f75475f98-5bmm9\" (UID: \"eb0cb030-aed3-4445-b7b0-6ad3326effb6\") " pod="llm/e2e-conflict-a-ca8d752a-kserve-5f75475f98-5bmm9"
Sep 11 14:31:04.498844 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:04.498622    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb0cb030-aed3-4445-b7b0-6ad3326effb6-tmp-dir\") pod \"e2e-conflict-a-ca8d752a-kserve-5f75475f98-5bmm9\" (UID: \"eb0cb030-aed3-4445-b7b0-6ad3326effb6\") " pod="llm/e2e-conflict-a-ca8d752a-kserve-5f75475f98-5bmm9"
Sep 11 14:31:04.498844 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:04.498642    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb0cb030-aed3-4445-b7b0-6ad3326effb6-model-cache\") pod \"e2e-conflict-a-ca8d752a-kserve-5f75475f98-5bmm9\" (UID: \"eb0cb030-aed3-4445-b7b0-6ad3326effb6\") " pod="llm/e2e-conflict-a-ca8d752a-kserve-5f75475f98-5bmm9"
Sep 11 14:31:04.498844 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:04.498657    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb0cb030-aed3-4445-b7b0-6ad3326effb6-tls-certs\") pod \"e2e-conflict-a-ca8d752a-kserve-5f75475f98-5bmm9\" (UID: \"eb0cb030-aed3-4445-b7b0-6ad3326effb6\") " pod="llm/e2e-conflict-a-ca8d752a-kserve-5f75475f98-5bmm9"
Sep 11 14:31:04.498844 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:04.498746    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb0cb030-aed3-4445-b7b0-6ad3326effb6-home\") pod \"e2e-conflict-a-ca8d752a-kserve-5f75475f98-5bmm9\" (UID: \"eb0cb030-aed3-4445-b7b0-6ad3326effb6\") " pod="llm/e2e-conflict-a-ca8d752a-kserve-5f75475f98-5bmm9"
Sep 11 14:31:04.600249 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:04.600151    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb0cb030-aed3-4445-b7b0-6ad3326effb6-model-cache\") pod \"e2e-conflict-a-ca8d752a-kserve-5f75475f98-5bmm9\" (UID: \"eb0cb030-aed3-4445-b7b0-6ad3326effb6\") " pod="llm/e2e-conflict-a-ca8d752a-kserve-5f75475f98-5bmm9"
Sep 11 14:31:04.600249 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:04.600188    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb0cb030-aed3-4445-b7b0-6ad3326effb6-tls-certs\") pod \"e2e-conflict-a-ca8d752a-kserve-5f75475f98-5bmm9\" (UID: \"eb0cb030-aed3-4445-b7b0-6ad3326effb6\") " pod="llm/e2e-conflict-a-ca8d752a-kserve-5f75475f98-5bmm9"
Sep 11 14:31:04.600249 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:04.600234    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb0cb030-aed3-4445-b7b0-6ad3326effb6-home\") pod \"e2e-conflict-a-ca8d752a-kserve-5f75475f98-5bmm9\" (UID: \"eb0cb030-aed3-4445-b7b0-6ad3326effb6\") " pod="llm/e2e-conflict-a-ca8d752a-kserve-5f75475f98-5bmm9"
Sep 11 14:31:04.600508 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:04.600316    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb0cb030-aed3-4445-b7b0-6ad3326effb6-dshm\") pod \"e2e-conflict-a-ca8d752a-kserve-5f75475f98-5bmm9\" (UID: \"eb0cb030-aed3-4445-b7b0-6ad3326effb6\") " pod="llm/e2e-conflict-a-ca8d752a-kserve-5f75475f98-5bmm9"
Sep 11 14:31:04.600508 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:04.600335    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jztg\" (UniqueName: \"kubernetes.io/projected/eb0cb030-aed3-4445-b7b0-6ad3326effb6-kube-api-access-2jztg\") pod \"e2e-conflict-a-ca8d752a-kserve-5f75475f98-5bmm9\" (UID: \"eb0cb030-aed3-4445-b7b0-6ad3326effb6\") " pod="llm/e2e-conflict-a-ca8d752a-kserve-5f75475f98-5bmm9"
Sep 11 14:31:04.600508 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:04.600375    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb0cb030-aed3-4445-b7b0-6ad3326effb6-tmp-dir\") pod \"e2e-conflict-a-ca8d752a-kserve-5f75475f98-5bmm9\" (UID: \"eb0cb030-aed3-4445-b7b0-6ad3326effb6\") " pod="llm/e2e-conflict-a-ca8d752a-kserve-5f75475f98-5bmm9"
Sep 11 14:31:04.600654 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:04.600581    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb0cb030-aed3-4445-b7b0-6ad3326effb6-model-cache\") pod \"e2e-conflict-a-ca8d752a-kserve-5f75475f98-5bmm9\" (UID: \"eb0cb030-aed3-4445-b7b0-6ad3326effb6\") " pod="llm/e2e-conflict-a-ca8d752a-kserve-5f75475f98-5bmm9"
Sep 11 14:31:04.600710 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:04.600669    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb0cb030-aed3-4445-b7b0-6ad3326effb6-home\") pod \"e2e-conflict-a-ca8d752a-kserve-5f75475f98-5bmm9\" (UID: \"eb0cb030-aed3-4445-b7b0-6ad3326effb6\") " pod="llm/e2e-conflict-a-ca8d752a-kserve-5f75475f98-5bmm9"
Sep 11 14:31:04.600790 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:04.600760    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb0cb030-aed3-4445-b7b0-6ad3326effb6-tmp-dir\") pod \"e2e-conflict-a-ca8d752a-kserve-5f75475f98-5bmm9\" (UID: \"eb0cb030-aed3-4445-b7b0-6ad3326effb6\") " pod="llm/e2e-conflict-a-ca8d752a-kserve-5f75475f98-5bmm9"
Sep 11 14:31:04.602574 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:04.602547    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb0cb030-aed3-4445-b7b0-6ad3326effb6-dshm\") pod \"e2e-conflict-a-ca8d752a-kserve-5f75475f98-5bmm9\" (UID: \"eb0cb030-aed3-4445-b7b0-6ad3326effb6\") " pod="llm/e2e-conflict-a-ca8d752a-kserve-5f75475f98-5bmm9"
Sep 11 14:31:04.602870 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:04.602854    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb0cb030-aed3-4445-b7b0-6ad3326effb6-tls-certs\") pod \"e2e-conflict-a-ca8d752a-kserve-5f75475f98-5bmm9\" (UID: \"eb0cb030-aed3-4445-b7b0-6ad3326effb6\") " pod="llm/e2e-conflict-a-ca8d752a-kserve-5f75475f98-5bmm9"
Sep 11 14:31:04.617002 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:04.616953    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jztg\" (UniqueName: \"kubernetes.io/projected/eb0cb030-aed3-4445-b7b0-6ad3326effb6-kube-api-access-2jztg\") pod \"e2e-conflict-a-ca8d752a-kserve-5f75475f98-5bmm9\" (UID: \"eb0cb030-aed3-4445-b7b0-6ad3326effb6\") " pod="llm/e2e-conflict-a-ca8d752a-kserve-5f75475f98-5bmm9"
Sep 11 14:31:04.663721 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:04.663685    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-ca8d752a-kserve-5f75475f98-5bmm9"
Sep 11 14:31:04.953554 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:04.953529    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-ca8d752a-kserve-5f75475f98-5bmm9"]
Sep 11 14:31:04.955418 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:31:04.955395    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb0cb030_aed3_4445_b7b0_6ad3326effb6.slice/crio-7662c605159fa174b560fa46ecf61ab506bed19c1e97744650a34fc035e04e73 WatchSource:0}: Error finding container 7662c605159fa174b560fa46ecf61ab506bed19c1e97744650a34fc035e04e73: Status 404 returned error can't find the container with id 7662c605159fa174b560fa46ecf61ab506bed19c1e97744650a34fc035e04e73
Sep 11 14:31:05.516474 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:05.516438    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-ca8d752a-kserve-5f75475f98-5bmm9" event={"ID":"eb0cb030-aed3-4445-b7b0-6ad3326effb6","Type":"ContainerStarted","Data":"9b1afb9049690b44c60bb68576a335fa2d48d4320d2ec067adb66288813b3549"}
Sep 11 14:31:05.516886 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:05.516482    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-ca8d752a-kserve-5f75475f98-5bmm9" event={"ID":"eb0cb030-aed3-4445-b7b0-6ad3326effb6","Type":"ContainerStarted","Data":"7662c605159fa174b560fa46ecf61ab506bed19c1e97744650a34fc035e04e73"}
Sep 11 14:31:05.545597 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:05.545540    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-ca8d752a-kserve-5f75475f98-5bmm9" podStartSLOduration=1.545520955 podStartE2EDuration="1.545520955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:31:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:31:05.543191526 +0000 UTC m=+968.693741590" watchObservedRunningTime="2026-09-11 14:31:05.545520955 +0000 UTC m=+968.696071028"
Sep 11 14:31:06.227815 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:06.227777    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57"]
Sep 11 14:31:06.231265 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:06.231245    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57"
Sep 11 14:31:06.234522 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:06.234504    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-1b6b934c-openshift-default-dockercfg-xf5w5\""
Sep 11 14:31:06.258722 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:06.258690    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57"]
Sep 11 14:31:06.313695 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:06.313659    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4017c78-f096-4332-9d98-389296470ab5-istio-token\") pod \"e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57\" (UID: \"b4017c78-f096-4332-9d98-389296470ab5\") " pod="openshift-ingress/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57"
Sep 11 14:31:06.313908 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:06.313721    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4017c78-f096-4332-9d98-389296470ab5-workload-certs\") pod \"e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57\" (UID: \"b4017c78-f096-4332-9d98-389296470ab5\") " pod="openshift-ingress/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57"
Sep 11 14:31:06.313908 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:06.313765    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4017c78-f096-4332-9d98-389296470ab5-credential-socket\") pod \"e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57\" (UID: \"b4017c78-f096-4332-9d98-389296470ab5\") " pod="openshift-ingress/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57"
Sep 11 14:31:06.313908 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:06.313853    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4017c78-f096-4332-9d98-389296470ab5-workload-socket\") pod \"e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57\" (UID: \"b4017c78-f096-4332-9d98-389296470ab5\") " pod="openshift-ingress/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57"
Sep 11 14:31:06.313908 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:06.313894    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4017c78-f096-4332-9d98-389296470ab5-istio-podinfo\") pod \"e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57\" (UID: \"b4017c78-f096-4332-9d98-389296470ab5\") " pod="openshift-ingress/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57"
Sep 11 14:31:06.314114 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:06.313921    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4017c78-f096-4332-9d98-389296470ab5-istiod-ca-cert\") pod \"e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57\" (UID: \"b4017c78-f096-4332-9d98-389296470ab5\") " pod="openshift-ingress/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57"
Sep 11 14:31:06.314114 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:06.313948    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4017c78-f096-4332-9d98-389296470ab5-istio-envoy\") pod \"e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57\" (UID: \"b4017c78-f096-4332-9d98-389296470ab5\") " pod="openshift-ingress/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57"
Sep 11 14:31:06.314114 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:06.313964    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4017c78-f096-4332-9d98-389296470ab5-istio-data\") pod \"e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57\" (UID: \"b4017c78-f096-4332-9d98-389296470ab5\") " pod="openshift-ingress/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57"
Sep 11 14:31:06.314114 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:06.313999    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmdr8\" (UniqueName: \"kubernetes.io/projected/b4017c78-f096-4332-9d98-389296470ab5-kube-api-access-xmdr8\") pod \"e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57\" (UID: \"b4017c78-f096-4332-9d98-389296470ab5\") " pod="openshift-ingress/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57"
Sep 11 14:31:06.414567 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:06.414498    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4017c78-f096-4332-9d98-389296470ab5-workload-socket\") pod \"e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57\" (UID: \"b4017c78-f096-4332-9d98-389296470ab5\") " pod="openshift-ingress/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57"
Sep 11 14:31:06.414567 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:06.414554    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4017c78-f096-4332-9d98-389296470ab5-istio-podinfo\") pod \"e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57\" (UID: \"b4017c78-f096-4332-9d98-389296470ab5\") " pod="openshift-ingress/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57"
Sep 11 14:31:06.415054 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:06.414598    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4017c78-f096-4332-9d98-389296470ab5-istiod-ca-cert\") pod \"e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57\" (UID: \"b4017c78-f096-4332-9d98-389296470ab5\") " pod="openshift-ingress/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57"
Sep 11 14:31:06.415054 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:06.414637    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4017c78-f096-4332-9d98-389296470ab5-istio-envoy\") pod \"e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57\" (UID: \"b4017c78-f096-4332-9d98-389296470ab5\") " pod="openshift-ingress/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57"
Sep 11 14:31:06.415054 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:06.414662    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4017c78-f096-4332-9d98-389296470ab5-istio-data\") pod \"e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57\" (UID: \"b4017c78-f096-4332-9d98-389296470ab5\") " pod="openshift-ingress/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57"
Sep 11 14:31:06.415054 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:06.414700    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmdr8\" (UniqueName: \"kubernetes.io/projected/b4017c78-f096-4332-9d98-389296470ab5-kube-api-access-xmdr8\") pod \"e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57\" (UID: \"b4017c78-f096-4332-9d98-389296470ab5\") " pod="openshift-ingress/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57"
Sep 11 14:31:06.415054 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:06.414775    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4017c78-f096-4332-9d98-389296470ab5-istio-token\") pod \"e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57\" (UID: \"b4017c78-f096-4332-9d98-389296470ab5\") " pod="openshift-ingress/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57"
Sep 11 14:31:06.415054 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:06.414853    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4017c78-f096-4332-9d98-389296470ab5-workload-certs\") pod \"e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57\" (UID: \"b4017c78-f096-4332-9d98-389296470ab5\") " pod="openshift-ingress/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57"
Sep 11 14:31:06.415054 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:06.414907    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4017c78-f096-4332-9d98-389296470ab5-credential-socket\") pod \"e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57\" (UID: \"b4017c78-f096-4332-9d98-389296470ab5\") " pod="openshift-ingress/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57"
Sep 11 14:31:06.415469 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:06.415444    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4017c78-f096-4332-9d98-389296470ab5-credential-socket\") pod \"e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57\" (UID: \"b4017c78-f096-4332-9d98-389296470ab5\") " pod="openshift-ingress/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57"
Sep 11 14:31:06.415587 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:06.415561    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4017c78-f096-4332-9d98-389296470ab5-istio-data\") pod \"e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57\" (UID: \"b4017c78-f096-4332-9d98-389296470ab5\") " pod="openshift-ingress/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57"
Sep 11 14:31:06.415830 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:06.415792    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4017c78-f096-4332-9d98-389296470ab5-workload-socket\") pod \"e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57\" (UID: \"b4017c78-f096-4332-9d98-389296470ab5\") " pod="openshift-ingress/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57"
Sep 11 14:31:06.415920 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:06.415898    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4017c78-f096-4332-9d98-389296470ab5-workload-certs\") pod \"e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57\" (UID: \"b4017c78-f096-4332-9d98-389296470ab5\") " pod="openshift-ingress/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57"
Sep 11 14:31:06.416414 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:06.416389    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4017c78-f096-4332-9d98-389296470ab5-istiod-ca-cert\") pod \"e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57\" (UID: \"b4017c78-f096-4332-9d98-389296470ab5\") " pod="openshift-ingress/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57"
Sep 11 14:31:06.417899 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:06.417881    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4017c78-f096-4332-9d98-389296470ab5-istio-envoy\") pod \"e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57\" (UID: \"b4017c78-f096-4332-9d98-389296470ab5\") " pod="openshift-ingress/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57"
Sep 11 14:31:06.418418 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:06.418398    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4017c78-f096-4332-9d98-389296470ab5-istio-podinfo\") pod \"e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57\" (UID: \"b4017c78-f096-4332-9d98-389296470ab5\") " pod="openshift-ingress/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57"
Sep 11 14:31:06.433433 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:06.433405    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4017c78-f096-4332-9d98-389296470ab5-istio-token\") pod \"e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57\" (UID: \"b4017c78-f096-4332-9d98-389296470ab5\") " pod="openshift-ingress/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57"
Sep 11 14:31:06.434676 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:06.434652    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmdr8\" (UniqueName: \"kubernetes.io/projected/b4017c78-f096-4332-9d98-389296470ab5-kube-api-access-xmdr8\") pod \"e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57\" (UID: \"b4017c78-f096-4332-9d98-389296470ab5\") " pod="openshift-ingress/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57"
Sep 11 14:31:06.541739 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:06.541704    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57"
Sep 11 14:31:06.846299 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:06.846273    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57"]
Sep 11 14:31:06.849311 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:31:06.849271    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4017c78_f096_4332_9d98_389296470ab5.slice/crio-2f0e2086acd3ec84342920e3b3b6448d35db0f1c721cf91d4fb8f89a4cfbc8d5 WatchSource:0}: Error finding container 2f0e2086acd3ec84342920e3b3b6448d35db0f1c721cf91d4fb8f89a4cfbc8d5: Status 404 returned error can't find the container with id 2f0e2086acd3ec84342920e3b3b6448d35db0f1c721cf91d4fb8f89a4cfbc8d5
Sep 11 14:31:06.851624 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:06.851591    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:31:06.851694 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:06.851660    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:31:06.851694 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:06.851689    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:31:07.526958 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:07.526916    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57" event={"ID":"b4017c78-f096-4332-9d98-389296470ab5","Type":"ContainerStarted","Data":"7ecb575282a22bc0225a22f78e9ee93822f1dde27719377f66816cfb32f5feeb"}
Sep 11 14:31:07.527519 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:07.526995    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57" event={"ID":"b4017c78-f096-4332-9d98-389296470ab5","Type":"ContainerStarted","Data":"2f0e2086acd3ec84342920e3b3b6448d35db0f1c721cf91d4fb8f89a4cfbc8d5"}
Sep 11 14:31:07.542298 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:07.542270    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57"
Sep 11 14:31:07.547242 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:07.547218    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57"
Sep 11 14:31:07.564884 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:07.564825    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57" podStartSLOduration=1.564810419 podStartE2EDuration="1.564810419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:31:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:31:07.56255911 +0000 UTC m=+970.713109175" watchObservedRunningTime="2026-09-11 14:31:07.564810419 +0000 UTC m=+970.715360481"
Sep 11 14:31:08.532260 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:08.532220    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57"
Sep 11 14:31:08.533245 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:08.533220    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57"
Sep 11 14:31:14.525179 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:14.525139    2581 patch_prober.go:28] interesting pod/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 11 14:31:14.525580 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:14.525209    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg" podUID="b0984e71-aec3-4372-b101-ecf16c20488a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 11 14:31:14.664581 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:14.664548    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-ca8d752a-kserve-5f75475f98-5bmm9"
Sep 11 14:31:14.664773 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:14.664600    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-ca8d752a-kserve-5f75475f98-5bmm9"
Sep 11 14:31:14.680306 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:14.680263    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-ca8d752a-kserve-5f75475f98-5bmm9"
Sep 11 14:31:15.577784 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:15.577749    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-ca8d752a-kserve-5f75475f98-5bmm9"
Sep 11 14:31:16.572090 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:16.572055    2581 generic.go:356] "Generic (PLEG): container finished" podID="b0984e71-aec3-4372-b101-ecf16c20488a" containerID="b21e822f719dcfcd695ac024cd3876796f8f18297affd36803174cd42777ee4f" exitCode=0
Sep 11 14:31:16.572279 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:16.572127    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg" event={"ID":"b0984e71-aec3-4372-b101-ecf16c20488a","Type":"ContainerDied","Data":"b21e822f719dcfcd695ac024cd3876796f8f18297affd36803174cd42777ee4f"}
Sep 11 14:31:16.572673 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:16.572655    2581 scope.go:117] "RemoveContainer" containerID="b21e822f719dcfcd695ac024cd3876796f8f18297affd36803174cd42777ee4f"
Sep 11 14:31:16.573664 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:16.573635    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:31:16.573723 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:16.573706    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:31:16.573763 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:16.573736    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:31:17.582219 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:17.582185    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg" event={"ID":"b0984e71-aec3-4372-b101-ecf16c20488a","Type":"ContainerStarted","Data":"9c6c6dc399f3c3953584bd1a943ab1c011c5dcf7bc50dba4c8b29ed7884c1f14"}
Sep 11 14:31:18.569410 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:18.569373    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg"
Sep 11 14:31:18.569607 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:18.569586    2581 patch_prober.go:28] interesting pod/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Sep 11 14:31:18.569697 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:18.569669    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg" podUID="b0984e71-aec3-4372-b101-ecf16c20488a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Sep 11 14:31:19.569470 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:19.569387    2581 patch_prober.go:28] interesting pod/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Sep 11 14:31:19.569833 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:19.569458    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg" podUID="b0984e71-aec3-4372-b101-ecf16c20488a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Sep 11 14:31:20.569473 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:20.569437    2581 patch_prober.go:28] interesting pod/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Sep 11 14:31:20.569840 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:20.569496    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg" podUID="b0984e71-aec3-4372-b101-ecf16c20488a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Sep 11 14:31:21.569986 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:21.569945    2581 patch_prober.go:28] interesting pod/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Sep 11 14:31:21.570365 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:21.570013    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg" podUID="b0984e71-aec3-4372-b101-ecf16c20488a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Sep 11 14:31:22.569716 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:22.569675    2581 patch_prober.go:28] interesting pod/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Sep 11 14:31:22.569912 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:22.569734    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg" podUID="b0984e71-aec3-4372-b101-ecf16c20488a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Sep 11 14:31:23.573192 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:23.573162    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg"
Sep 11 14:31:23.573588 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:23.573463    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg"
Sep 11 14:31:23.574199 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:23.574177    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg"
Sep 11 14:31:24.501592 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:24.501555    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b6b934c-bfntg"]
Sep 11 14:31:24.504928 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:24.504908    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b6b934c-bfntg"
Sep 11 14:31:24.508849 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:24.508829    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 14:31:24.508953 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:24.508877    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hcm4g\""
Sep 11 14:31:24.520754 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:24.520729    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b6b934c-bfntg"]
Sep 11 14:31:24.599981 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:24.599942    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c35a368c-d847-4413-87a8-9174993a27bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1b6b934c-bfntg\" (UID: \"c35a368c-d847-4413-87a8-9174993a27bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b6b934c-bfntg"
Sep 11 14:31:24.700924 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:24.700884    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c35a368c-d847-4413-87a8-9174993a27bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1b6b934c-bfntg\" (UID: \"c35a368c-d847-4413-87a8-9174993a27bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b6b934c-bfntg"
Sep 11 14:31:24.701740 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:24.701715    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c35a368c-d847-4413-87a8-9174993a27bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1b6b934c-bfntg\" (UID: \"c35a368c-d847-4413-87a8-9174993a27bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b6b934c-bfntg"
Sep 11 14:31:24.815341 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:24.815260    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b6b934c-bfntg"
Sep 11 14:31:25.101279 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:25.101246    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b6b934c-bfntg"]
Sep 11 14:31:25.102898 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:31:25.102869    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc35a368c_d847_4413_87a8_9174993a27bf.slice/crio-67d5e893fa7e93aa9a40fb0a340dd0316b3052c2ffc649fbda03838ac1f482c2 WatchSource:0}: Error finding container 67d5e893fa7e93aa9a40fb0a340dd0316b3052c2ffc649fbda03838ac1f482c2: Status 404 returned error can't find the container with id 67d5e893fa7e93aa9a40fb0a340dd0316b3052c2ffc649fbda03838ac1f482c2
Sep 11 14:31:25.457311 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:25.457225    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt"]
Sep 11 14:31:25.462384 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:25.462353    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt"
Sep 11 14:31:25.465330 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:25.465308    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-5a7035-openshift-default-dockercfg-hb6gj\""
Sep 11 14:31:25.483261 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:25.483228    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt"]
Sep 11 14:31:25.510553 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:25.510516    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bde0890-f752-4b4b-976f-804512c66e7c-credential-socket\") pod \"e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt\" (UID: \"8bde0890-f752-4b4b-976f-804512c66e7c\") " pod="openshift-ingress/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt"
Sep 11 14:31:25.510553 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:25.510561    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bde0890-f752-4b4b-976f-804512c66e7c-workload-certs\") pod \"e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt\" (UID: \"8bde0890-f752-4b4b-976f-804512c66e7c\") " pod="openshift-ingress/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt"
Sep 11 14:31:25.510787 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:25.510629    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sjhh\" (UniqueName: \"kubernetes.io/projected/8bde0890-f752-4b4b-976f-804512c66e7c-kube-api-access-8sjhh\") pod \"e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt\" (UID: \"8bde0890-f752-4b4b-976f-804512c66e7c\") " pod="openshift-ingress/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt"
Sep 11 14:31:25.510787 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:25.510680    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bde0890-f752-4b4b-976f-804512c66e7c-workload-socket\") pod \"e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt\" (UID: \"8bde0890-f752-4b4b-976f-804512c66e7c\") " pod="openshift-ingress/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt"
Sep 11 14:31:25.510787 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:25.510767    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bde0890-f752-4b4b-976f-804512c66e7c-istiod-ca-cert\") pod \"e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt\" (UID: \"8bde0890-f752-4b4b-976f-804512c66e7c\") " pod="openshift-ingress/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt"
Sep 11 14:31:25.510910 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:25.510816    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bde0890-f752-4b4b-976f-804512c66e7c-istio-data\") pod \"e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt\" (UID: \"8bde0890-f752-4b4b-976f-804512c66e7c\") " pod="openshift-ingress/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt"
Sep 11 14:31:25.510945 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:25.510921    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bde0890-f752-4b4b-976f-804512c66e7c-istio-podinfo\") pod \"e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt\" (UID: \"8bde0890-f752-4b4b-976f-804512c66e7c\") " pod="openshift-ingress/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt"
Sep 11 14:31:25.511000 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:25.510955    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bde0890-f752-4b4b-976f-804512c66e7c-istio-envoy\") pod \"e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt\" (UID: \"8bde0890-f752-4b4b-976f-804512c66e7c\") " pod="openshift-ingress/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt"
Sep 11 14:31:25.511034 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:25.511015    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bde0890-f752-4b4b-976f-804512c66e7c-istio-token\") pod \"e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt\" (UID: \"8bde0890-f752-4b4b-976f-804512c66e7c\") " pod="openshift-ingress/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt"
Sep 11 14:31:25.612484 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:25.612435    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bde0890-f752-4b4b-976f-804512c66e7c-istio-data\") pod \"e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt\" (UID: \"8bde0890-f752-4b4b-976f-804512c66e7c\") " pod="openshift-ingress/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt"
Sep 11 14:31:25.613020 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:25.612512    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bde0890-f752-4b4b-976f-804512c66e7c-istio-podinfo\") pod \"e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt\" (UID: \"8bde0890-f752-4b4b-976f-804512c66e7c\") " pod="openshift-ingress/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt"
Sep 11 14:31:25.613020 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:25.612544    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bde0890-f752-4b4b-976f-804512c66e7c-istio-envoy\") pod \"e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt\" (UID: \"8bde0890-f752-4b4b-976f-804512c66e7c\") " pod="openshift-ingress/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt"
Sep 11 14:31:25.613020 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:25.612565    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bde0890-f752-4b4b-976f-804512c66e7c-istio-token\") pod \"e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt\" (UID: \"8bde0890-f752-4b4b-976f-804512c66e7c\") " pod="openshift-ingress/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt"
Sep 11 14:31:25.613020 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:25.612590    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bde0890-f752-4b4b-976f-804512c66e7c-credential-socket\") pod \"e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt\" (UID: \"8bde0890-f752-4b4b-976f-804512c66e7c\") " pod="openshift-ingress/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt"
Sep 11 14:31:25.613020 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:25.612608    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bde0890-f752-4b4b-976f-804512c66e7c-workload-certs\") pod \"e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt\" (UID: \"8bde0890-f752-4b4b-976f-804512c66e7c\") " pod="openshift-ingress/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt"
Sep 11 14:31:25.613020 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:25.612626    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sjhh\" (UniqueName: \"kubernetes.io/projected/8bde0890-f752-4b4b-976f-804512c66e7c-kube-api-access-8sjhh\") pod \"e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt\" (UID: \"8bde0890-f752-4b4b-976f-804512c66e7c\") " pod="openshift-ingress/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt"
Sep 11 14:31:25.613020 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:25.612662    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bde0890-f752-4b4b-976f-804512c66e7c-workload-socket\") pod \"e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt\" (UID: \"8bde0890-f752-4b4b-976f-804512c66e7c\") " pod="openshift-ingress/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt"
Sep 11 14:31:25.613020 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:25.612719    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bde0890-f752-4b4b-976f-804512c66e7c-istiod-ca-cert\") pod \"e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt\" (UID: \"8bde0890-f752-4b4b-976f-804512c66e7c\") " pod="openshift-ingress/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt"
Sep 11 14:31:25.613447 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:25.613362    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bde0890-f752-4b4b-976f-804512c66e7c-istiod-ca-cert\") pod \"e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt\" (UID: \"8bde0890-f752-4b4b-976f-804512c66e7c\") " pod="openshift-ingress/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt"
Sep 11 14:31:25.613950 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:25.613734    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bde0890-f752-4b4b-976f-804512c66e7c-istio-data\") pod \"e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt\" (UID: \"8bde0890-f752-4b4b-976f-804512c66e7c\") " pod="openshift-ingress/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt"
Sep 11 14:31:25.614403 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:25.614343    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bde0890-f752-4b4b-976f-804512c66e7c-workload-certs\") pod \"e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt\" (UID: \"8bde0890-f752-4b4b-976f-804512c66e7c\") " pod="openshift-ingress/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt"
Sep 11 14:31:25.614517 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:25.614420    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bde0890-f752-4b4b-976f-804512c66e7c-workload-socket\") pod \"e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt\" (UID: \"8bde0890-f752-4b4b-976f-804512c66e7c\") " pod="openshift-ingress/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt"
Sep 11 14:31:25.614517 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:25.614463    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bde0890-f752-4b4b-976f-804512c66e7c-credential-socket\") pod \"e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt\" (UID: \"8bde0890-f752-4b4b-976f-804512c66e7c\") " pod="openshift-ingress/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt"
Sep 11 14:31:25.616117 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:25.616099    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bde0890-f752-4b4b-976f-804512c66e7c-istio-envoy\") pod \"e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt\" (UID: \"8bde0890-f752-4b4b-976f-804512c66e7c\") " pod="openshift-ingress/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt"
Sep 11 14:31:25.619963 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:25.619939    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bde0890-f752-4b4b-976f-804512c66e7c-istio-podinfo\") pod \"e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt\" (UID: \"8bde0890-f752-4b4b-976f-804512c66e7c\") " pod="openshift-ingress/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt"
Sep 11 14:31:25.628651 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:25.628628    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bde0890-f752-4b4b-976f-804512c66e7c-istio-token\") pod \"e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt\" (UID: \"8bde0890-f752-4b4b-976f-804512c66e7c\") " pod="openshift-ingress/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt"
Sep 11 14:31:25.629021 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:25.629001    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sjhh\" (UniqueName: \"kubernetes.io/projected/8bde0890-f752-4b4b-976f-804512c66e7c-kube-api-access-8sjhh\") pod \"e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt\" (UID: \"8bde0890-f752-4b4b-976f-804512c66e7c\") " pod="openshift-ingress/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt"
Sep 11 14:31:25.637153 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:25.637127    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b6b934c-bfntg" event={"ID":"c35a368c-d847-4413-87a8-9174993a27bf","Type":"ContainerStarted","Data":"67d5e893fa7e93aa9a40fb0a340dd0316b3052c2ffc649fbda03838ac1f482c2"}
Sep 11 14:31:25.780771 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:25.780736    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt"
Sep 11 14:31:26.087270 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:26.087111    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt"]
Sep 11 14:31:26.089604 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:31:26.089561    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bde0890_f752_4b4b_976f_804512c66e7c.slice/crio-11c4aa3868cb8b334b6e06215c9f78a4f45d8be45370900c9918ed3d29550e99 WatchSource:0}: Error finding container 11c4aa3868cb8b334b6e06215c9f78a4f45d8be45370900c9918ed3d29550e99: Status 404 returned error can't find the container with id 11c4aa3868cb8b334b6e06215c9f78a4f45d8be45370900c9918ed3d29550e99
Sep 11 14:31:26.092068 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:26.092033    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:31:26.092168 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:26.092102    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:31:26.092168 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:26.092132    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:31:26.645261 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:26.645226    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt" event={"ID":"8bde0890-f752-4b4b-976f-804512c66e7c","Type":"ContainerStarted","Data":"05b9327f3bdb2bd741dc4dcd8a48cd091862b275dabf80dfeec4ec0a0c2bed59"}
Sep 11 14:31:26.645738 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:26.645284    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt" event={"ID":"8bde0890-f752-4b4b-976f-804512c66e7c","Type":"ContainerStarted","Data":"11c4aa3868cb8b334b6e06215c9f78a4f45d8be45370900c9918ed3d29550e99"}
Sep 11 14:31:26.677482 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:26.677419    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt" podStartSLOduration=1.677396958 podStartE2EDuration="1.677396958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:31:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:31:26.673910075 +0000 UTC m=+989.824460142" watchObservedRunningTime="2026-09-11 14:31:26.677396958 +0000 UTC m=+989.827947022"
Sep 11 14:31:26.780892 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:26.780853    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt"
Sep 11 14:31:27.786599 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:27.786571    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt"
Sep 11 14:31:28.657337 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:28.657297    2581 generic.go:356] "Generic (PLEG): container finished" podID="c35a368c-d847-4413-87a8-9174993a27bf" containerID="9d06538d8c5d02b135a3bb4da31d70f2e0d44ed2da2e24a98ec5339d909b65b2" exitCode=0
Sep 11 14:31:28.657587 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:28.657381    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b6b934c-bfntg" event={"ID":"c35a368c-d847-4413-87a8-9174993a27bf","Type":"ContainerDied","Data":"9d06538d8c5d02b135a3bb4da31d70f2e0d44ed2da2e24a98ec5339d909b65b2"}
Sep 11 14:31:28.659252 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:28.659228    2581 generic.go:356] "Generic (PLEG): container finished" podID="b4017c78-f096-4332-9d98-389296470ab5" containerID="7ecb575282a22bc0225a22f78e9ee93822f1dde27719377f66816cfb32f5feeb" exitCode=0
Sep 11 14:31:28.659359 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:28.659304    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57" event={"ID":"b4017c78-f096-4332-9d98-389296470ab5","Type":"ContainerDied","Data":"7ecb575282a22bc0225a22f78e9ee93822f1dde27719377f66816cfb32f5feeb"}
Sep 11 14:31:28.659493 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:28.659478    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt"
Sep 11 14:31:28.659775 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:28.659760    2581 scope.go:117] "RemoveContainer" containerID="7ecb575282a22bc0225a22f78e9ee93822f1dde27719377f66816cfb32f5feeb"
Sep 11 14:31:28.660422 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:28.660406    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt"
Sep 11 14:31:28.660805 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:28.660774    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:31:28.660890 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:28.660844    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:31:28.660890 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:28.660876    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:31:29.665950 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:29.665908    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57" event={"ID":"b4017c78-f096-4332-9d98-389296470ab5","Type":"ContainerStarted","Data":"c93f9c13861502bd766f119748163c69aebecc664b77f9af03f65f47d5c44526"}
Sep 11 14:31:29.841331 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:29.841304    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b6b934c-bfntg"
Sep 11 14:31:29.956905 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:29.956817    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c35a368c-d847-4413-87a8-9174993a27bf-maas-api-service-ca\") pod \"c35a368c-d847-4413-87a8-9174993a27bf\" (UID: \"c35a368c-d847-4413-87a8-9174993a27bf\") "
Sep 11 14:31:29.957158 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:29.957136    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c35a368c-d847-4413-87a8-9174993a27bf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c35a368c-d847-4413-87a8-9174993a27bf" (UID: "c35a368c-d847-4413-87a8-9174993a27bf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:31:30.057760 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:30.057709    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c35a368c-d847-4413-87a8-9174993a27bf-maas-api-service-ca\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:31:30.541887 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:30.541850    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57"
Sep 11 14:31:30.542131 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:30.542107    2581 patch_prober.go:28] interesting pod/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Sep 11 14:31:30.542188 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:30.542159    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57" podUID="b4017c78-f096-4332-9d98-389296470ab5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Sep 11 14:31:30.672138 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:30.672106    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b6b934c-bfntg"
Sep 11 14:31:30.672138 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:30.672138    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b6b934c-bfntg" event={"ID":"c35a368c-d847-4413-87a8-9174993a27bf","Type":"ContainerDied","Data":"67d5e893fa7e93aa9a40fb0a340dd0316b3052c2ffc649fbda03838ac1f482c2"}
Sep 11 14:31:30.672684 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:30.672181    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67d5e893fa7e93aa9a40fb0a340dd0316b3052c2ffc649fbda03838ac1f482c2"
Sep 11 14:31:31.542227 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:31.542184    2581 patch_prober.go:28] interesting pod/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Sep 11 14:31:31.542443 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:31.542245    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57" podUID="b4017c78-f096-4332-9d98-389296470ab5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Sep 11 14:31:32.542717 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:32.542677    2581 patch_prober.go:28] interesting pod/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Sep 11 14:31:32.543194 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:32.542746    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57" podUID="b4017c78-f096-4332-9d98-389296470ab5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Sep 11 14:31:33.542342 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:33.542299    2581 patch_prober.go:28] interesting pod/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Sep 11 14:31:33.542514 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:33.542379    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57" podUID="b4017c78-f096-4332-9d98-389296470ab5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Sep 11 14:31:34.542291 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:34.542251    2581 patch_prober.go:28] interesting pod/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Sep 11 14:31:34.542696 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:34.542325    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57" podUID="b4017c78-f096-4332-9d98-389296470ab5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Sep 11 14:31:35.547698 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:35.547665    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57"
Sep 11 14:31:35.548119 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:35.547940    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57"
Sep 11 14:31:35.548601 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:35.548581    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57"
Sep 11 14:31:38.605919 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:38.605880    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b6b934c-bfntg"]
Sep 11 14:31:38.621905 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:38.621823    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b6b934c-bfntg"]
Sep 11 14:31:38.781401 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:38.781358    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57"]
Sep 11 14:31:38.781717 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:38.781686    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57" podUID="b4017c78-f096-4332-9d98-389296470ab5" containerName="istio-proxy" containerID="cri-o://c93f9c13861502bd766f119748163c69aebecc664b77f9af03f65f47d5c44526" gracePeriod=30
Sep 11 14:31:39.499632 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:39.499600    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c35a368c-d847-4413-87a8-9174993a27bf" path="/var/lib/kubelet/pods/c35a368c-d847-4413-87a8-9174993a27bf/volumes"
Sep 11 14:31:40.117769 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:40.117729    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74"]
Sep 11 14:31:40.118277 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:40.118261    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c35a368c-d847-4413-87a8-9174993a27bf" containerName="revoke-keys"
Sep 11 14:31:40.118334 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:40.118280    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c35a368c-d847-4413-87a8-9174993a27bf" containerName="revoke-keys"
Sep 11 14:31:40.118373 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:40.118360    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c35a368c-d847-4413-87a8-9174993a27bf" containerName="revoke-keys"
Sep 11 14:31:40.126243 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:40.126218    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74"
Sep 11 14:31:40.131626 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:40.131600    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-a6adb955-gw-openshift-default-dockercfg-rn5p2\""
Sep 11 14:31:40.147017 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:40.146991    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74"]
Sep 11 14:31:40.262899 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:40.262862    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b08b8a0-c2e7-4095-8eb3-090245fab2b8-workload-socket\") pod \"e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74\" (UID: \"7b08b8a0-c2e7-4095-8eb3-090245fab2b8\") " pod="openshift-ingress/e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74"
Sep 11 14:31:40.262899 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:40.262901    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krcb6\" (UniqueName: \"kubernetes.io/projected/7b08b8a0-c2e7-4095-8eb3-090245fab2b8-kube-api-access-krcb6\") pod \"e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74\" (UID: \"7b08b8a0-c2e7-4095-8eb3-090245fab2b8\") " pod="openshift-ingress/e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74"
Sep 11 14:31:40.263193 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:40.262921    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b08b8a0-c2e7-4095-8eb3-090245fab2b8-credential-socket\") pod \"e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74\" (UID: \"7b08b8a0-c2e7-4095-8eb3-090245fab2b8\") " pod="openshift-ingress/e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74"
Sep 11 14:31:40.263193 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:40.263000    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b08b8a0-c2e7-4095-8eb3-090245fab2b8-istio-data\") pod \"e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74\" (UID: \"7b08b8a0-c2e7-4095-8eb3-090245fab2b8\") " pod="openshift-ingress/e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74"
Sep 11 14:31:40.263193 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:40.263054    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b08b8a0-c2e7-4095-8eb3-090245fab2b8-istio-token\") pod \"e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74\" (UID: \"7b08b8a0-c2e7-4095-8eb3-090245fab2b8\") " pod="openshift-ingress/e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74"
Sep 11 14:31:40.263193 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:40.263088    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b08b8a0-c2e7-4095-8eb3-090245fab2b8-istio-podinfo\") pod \"e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74\" (UID: \"7b08b8a0-c2e7-4095-8eb3-090245fab2b8\") " pod="openshift-ingress/e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74"
Sep 11 14:31:40.263390 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:40.263203    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b08b8a0-c2e7-4095-8eb3-090245fab2b8-istiod-ca-cert\") pod \"e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74\" (UID: \"7b08b8a0-c2e7-4095-8eb3-090245fab2b8\") " pod="openshift-ingress/e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74"
Sep 11 14:31:40.263390 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:40.263248    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b08b8a0-c2e7-4095-8eb3-090245fab2b8-workload-certs\") pod \"e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74\" (UID: \"7b08b8a0-c2e7-4095-8eb3-090245fab2b8\") " pod="openshift-ingress/e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74"
Sep 11 14:31:40.263390 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:40.263280    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b08b8a0-c2e7-4095-8eb3-090245fab2b8-istio-envoy\") pod \"e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74\" (UID: \"7b08b8a0-c2e7-4095-8eb3-090245fab2b8\") " pod="openshift-ingress/e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74"
Sep 11 14:31:40.364149 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:40.364111    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b08b8a0-c2e7-4095-8eb3-090245fab2b8-workload-socket\") pod \"e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74\" (UID: \"7b08b8a0-c2e7-4095-8eb3-090245fab2b8\") " pod="openshift-ingress/e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74"
Sep 11 14:31:40.364364 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:40.364147    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krcb6\" (UniqueName: \"kubernetes.io/projected/7b08b8a0-c2e7-4095-8eb3-090245fab2b8-kube-api-access-krcb6\") pod \"e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74\" (UID: \"7b08b8a0-c2e7-4095-8eb3-090245fab2b8\") " pod="openshift-ingress/e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74"
Sep 11 14:31:40.364364 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:40.364186    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b08b8a0-c2e7-4095-8eb3-090245fab2b8-credential-socket\") pod \"e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74\" (UID: \"7b08b8a0-c2e7-4095-8eb3-090245fab2b8\") " pod="openshift-ingress/e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74"
Sep 11 14:31:40.364364 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:40.364205    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b08b8a0-c2e7-4095-8eb3-090245fab2b8-istio-data\") pod \"e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74\" (UID: \"7b08b8a0-c2e7-4095-8eb3-090245fab2b8\") " pod="openshift-ingress/e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74"
Sep 11 14:31:40.364364 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:40.364237    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b08b8a0-c2e7-4095-8eb3-090245fab2b8-istio-token\") pod \"e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74\" (UID: \"7b08b8a0-c2e7-4095-8eb3-090245fab2b8\") " pod="openshift-ingress/e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74"
Sep 11 14:31:40.364588 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:40.364358    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b08b8a0-c2e7-4095-8eb3-090245fab2b8-istio-podinfo\") pod \"e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74\" (UID: \"7b08b8a0-c2e7-4095-8eb3-090245fab2b8\") " pod="openshift-ingress/e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74"
Sep 11 14:31:40.364588 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:40.364486    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b08b8a0-c2e7-4095-8eb3-090245fab2b8-istiod-ca-cert\") pod \"e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74\" (UID: \"7b08b8a0-c2e7-4095-8eb3-090245fab2b8\") " pod="openshift-ingress/e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74"
Sep 11 14:31:40.364588 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:40.364537    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b08b8a0-c2e7-4095-8eb3-090245fab2b8-workload-certs\") pod \"e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74\" (UID: \"7b08b8a0-c2e7-4095-8eb3-090245fab2b8\") " pod="openshift-ingress/e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74"
Sep 11 14:31:40.364588 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:40.364567    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b08b8a0-c2e7-4095-8eb3-090245fab2b8-istio-envoy\") pod \"e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74\" (UID: \"7b08b8a0-c2e7-4095-8eb3-090245fab2b8\") " pod="openshift-ingress/e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74"
Sep 11 14:31:40.364789 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:40.364623    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b08b8a0-c2e7-4095-8eb3-090245fab2b8-workload-socket\") pod \"e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74\" (UID: \"7b08b8a0-c2e7-4095-8eb3-090245fab2b8\") " pod="openshift-ingress/e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74"
Sep 11 14:31:40.364789 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:40.364721    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b08b8a0-c2e7-4095-8eb3-090245fab2b8-istio-data\") pod \"e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74\" (UID: \"7b08b8a0-c2e7-4095-8eb3-090245fab2b8\") " pod="openshift-ingress/e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74"
Sep 11 14:31:40.364916 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:40.364895    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b08b8a0-c2e7-4095-8eb3-090245fab2b8-workload-certs\") pod \"e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74\" (UID: \"7b08b8a0-c2e7-4095-8eb3-090245fab2b8\") " pod="openshift-ingress/e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74"
Sep 11 14:31:40.365167 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:40.365139    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b08b8a0-c2e7-4095-8eb3-090245fab2b8-credential-socket\") pod \"e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74\" (UID: \"7b08b8a0-c2e7-4095-8eb3-090245fab2b8\") " pod="openshift-ingress/e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74"
Sep 11 14:31:40.365280 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:40.365241    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b08b8a0-c2e7-4095-8eb3-090245fab2b8-istiod-ca-cert\") pod \"e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74\" (UID: \"7b08b8a0-c2e7-4095-8eb3-090245fab2b8\") " pod="openshift-ingress/e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74"
Sep 11 14:31:40.367290 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:40.367265    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b08b8a0-c2e7-4095-8eb3-090245fab2b8-istio-envoy\") pod \"e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74\" (UID: \"7b08b8a0-c2e7-4095-8eb3-090245fab2b8\") " pod="openshift-ingress/e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74"
Sep 11 14:31:40.367490 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:40.367472    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b08b8a0-c2e7-4095-8eb3-090245fab2b8-istio-podinfo\") pod \"e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74\" (UID: \"7b08b8a0-c2e7-4095-8eb3-090245fab2b8\") " pod="openshift-ingress/e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74"
Sep 11 14:31:40.395312 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:40.395238    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krcb6\" (UniqueName: \"kubernetes.io/projected/7b08b8a0-c2e7-4095-8eb3-090245fab2b8-kube-api-access-krcb6\") pod \"e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74\" (UID: \"7b08b8a0-c2e7-4095-8eb3-090245fab2b8\") " pod="openshift-ingress/e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74"
Sep 11 14:31:40.416601 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:40.416578    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b08b8a0-c2e7-4095-8eb3-090245fab2b8-istio-token\") pod \"e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74\" (UID: \"7b08b8a0-c2e7-4095-8eb3-090245fab2b8\") " pod="openshift-ingress/e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74"
Sep 11 14:31:40.436634 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:40.436604    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74"
Sep 11 14:31:40.741879 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:40.741843    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74"]
Sep 11 14:31:40.762933 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:31:40.762905    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b08b8a0_c2e7_4095_8eb3_090245fab2b8.slice/crio-14fc57c21e9208600ca083cd9a75c349e48321c6d5409f9f37e887304494636a WatchSource:0}: Error finding container 14fc57c21e9208600ca083cd9a75c349e48321c6d5409f9f37e887304494636a: Status 404 returned error can't find the container with id 14fc57c21e9208600ca083cd9a75c349e48321c6d5409f9f37e887304494636a
Sep 11 14:31:40.765319 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:40.765285    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:31:40.765405 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:40.765353    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:31:40.765405 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:40.765382    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:31:41.733684 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:41.733644    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74" event={"ID":"7b08b8a0-c2e7-4095-8eb3-090245fab2b8","Type":"ContainerStarted","Data":"32766f3857dac4e62846a8154af1e13de1c63ac540af1b97bf586a398e6d847d"}
Sep 11 14:31:41.734079 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:41.733703    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74" event={"ID":"7b08b8a0-c2e7-4095-8eb3-090245fab2b8","Type":"ContainerStarted","Data":"14fc57c21e9208600ca083cd9a75c349e48321c6d5409f9f37e887304494636a"}
Sep 11 14:31:41.779915 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:41.779862    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74" podStartSLOduration=1.779847519 podStartE2EDuration="1.779847519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:31:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:31:41.778386905 +0000 UTC m=+1004.928936996" watchObservedRunningTime="2026-09-11 14:31:41.779847519 +0000 UTC m=+1004.930397583"
Sep 11 14:31:42.437467 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:42.437432    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74"
Sep 11 14:31:42.442415 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:42.442390    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74"
Sep 11 14:31:42.738836 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:42.738757    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74"
Sep 11 14:31:42.739842 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:42.739825    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74"
Sep 11 14:31:44.040876 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:44.040849    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57"
Sep 11 14:31:44.202109 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:44.202026    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4017c78-f096-4332-9d98-389296470ab5-istio-data\") pod \"b4017c78-f096-4332-9d98-389296470ab5\" (UID: \"b4017c78-f096-4332-9d98-389296470ab5\") "
Sep 11 14:31:44.202109 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:44.202059    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4017c78-f096-4332-9d98-389296470ab5-istio-envoy\") pod \"b4017c78-f096-4332-9d98-389296470ab5\" (UID: \"b4017c78-f096-4332-9d98-389296470ab5\") "
Sep 11 14:31:44.202109 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:44.202077    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4017c78-f096-4332-9d98-389296470ab5-istiod-ca-cert\") pod \"b4017c78-f096-4332-9d98-389296470ab5\" (UID: \"b4017c78-f096-4332-9d98-389296470ab5\") "
Sep 11 14:31:44.202109 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:44.202101    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4017c78-f096-4332-9d98-389296470ab5-istio-token\") pod \"b4017c78-f096-4332-9d98-389296470ab5\" (UID: \"b4017c78-f096-4332-9d98-389296470ab5\") "
Sep 11 14:31:44.202426 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:44.202117    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmdr8\" (UniqueName: \"kubernetes.io/projected/b4017c78-f096-4332-9d98-389296470ab5-kube-api-access-xmdr8\") pod \"b4017c78-f096-4332-9d98-389296470ab5\" (UID: \"b4017c78-f096-4332-9d98-389296470ab5\") "
Sep 11 14:31:44.202426 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:44.202155    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4017c78-f096-4332-9d98-389296470ab5-istio-podinfo\") pod \"b4017c78-f096-4332-9d98-389296470ab5\" (UID: \"b4017c78-f096-4332-9d98-389296470ab5\") "
Sep 11 14:31:44.202426 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:44.202217    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4017c78-f096-4332-9d98-389296470ab5-credential-socket\") pod \"b4017c78-f096-4332-9d98-389296470ab5\" (UID: \"b4017c78-f096-4332-9d98-389296470ab5\") "
Sep 11 14:31:44.202426 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:44.202249    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4017c78-f096-4332-9d98-389296470ab5-workload-socket\") pod \"b4017c78-f096-4332-9d98-389296470ab5\" (UID: \"b4017c78-f096-4332-9d98-389296470ab5\") "
Sep 11 14:31:44.202426 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:44.202328    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4017c78-f096-4332-9d98-389296470ab5-workload-certs\") pod \"b4017c78-f096-4332-9d98-389296470ab5\" (UID: \"b4017c78-f096-4332-9d98-389296470ab5\") "
Sep 11 14:31:44.202426 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:44.202393    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4017c78-f096-4332-9d98-389296470ab5-istio-data" (OuterVolumeSpecName: "istio-data") pod "b4017c78-f096-4332-9d98-389296470ab5" (UID: "b4017c78-f096-4332-9d98-389296470ab5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:31:44.202720 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:44.202484    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4017c78-f096-4332-9d98-389296470ab5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b4017c78-f096-4332-9d98-389296470ab5" (UID: "b4017c78-f096-4332-9d98-389296470ab5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:31:44.202720 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:44.202585    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4017c78-f096-4332-9d98-389296470ab5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b4017c78-f096-4332-9d98-389296470ab5" (UID: "b4017c78-f096-4332-9d98-389296470ab5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:31:44.202811 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:44.202714    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4017c78-f096-4332-9d98-389296470ab5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b4017c78-f096-4332-9d98-389296470ab5" (UID: "b4017c78-f096-4332-9d98-389296470ab5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:31:44.202811 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:44.202746    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4017c78-f096-4332-9d98-389296470ab5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b4017c78-f096-4332-9d98-389296470ab5" (UID: "b4017c78-f096-4332-9d98-389296470ab5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:31:44.202919 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:44.202817    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4017c78-f096-4332-9d98-389296470ab5-workload-socket\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:31:44.202919 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:44.202834    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4017c78-f096-4332-9d98-389296470ab5-workload-certs\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:31:44.202919 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:44.202847    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4017c78-f096-4332-9d98-389296470ab5-istio-data\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:31:44.202919 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:44.202859    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4017c78-f096-4332-9d98-389296470ab5-istiod-ca-cert\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:31:44.204562 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:44.204530    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4017c78-f096-4332-9d98-389296470ab5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b4017c78-f096-4332-9d98-389296470ab5" (UID: "b4017c78-f096-4332-9d98-389296470ab5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 14:31:44.204562 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:44.204545    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4017c78-f096-4332-9d98-389296470ab5-kube-api-access-xmdr8" (OuterVolumeSpecName: "kube-api-access-xmdr8") pod "b4017c78-f096-4332-9d98-389296470ab5" (UID: "b4017c78-f096-4332-9d98-389296470ab5"). InnerVolumeSpecName "kube-api-access-xmdr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:31:44.204722 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:44.204671    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4017c78-f096-4332-9d98-389296470ab5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b4017c78-f096-4332-9d98-389296470ab5" (UID: "b4017c78-f096-4332-9d98-389296470ab5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:31:44.204722 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:44.204689    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4017c78-f096-4332-9d98-389296470ab5-istio-token" (OuterVolumeSpecName: "istio-token") pod "b4017c78-f096-4332-9d98-389296470ab5" (UID: "b4017c78-f096-4332-9d98-389296470ab5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:31:44.303526 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:44.303489    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4017c78-f096-4332-9d98-389296470ab5-istio-token\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:31:44.303526 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:44.303520    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmdr8\" (UniqueName: \"kubernetes.io/projected/b4017c78-f096-4332-9d98-389296470ab5-kube-api-access-xmdr8\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:31:44.303526 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:44.303530    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4017c78-f096-4332-9d98-389296470ab5-istio-podinfo\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:31:44.303747 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:44.303538    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4017c78-f096-4332-9d98-389296470ab5-credential-socket\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:31:44.303747 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:44.303549    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4017c78-f096-4332-9d98-389296470ab5-istio-envoy\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:31:44.751909 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:44.751875    2581 generic.go:356] "Generic (PLEG): container finished" podID="b4017c78-f096-4332-9d98-389296470ab5" containerID="c93f9c13861502bd766f119748163c69aebecc664b77f9af03f65f47d5c44526" exitCode=0
Sep 11 14:31:44.752162 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:44.751917    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57" event={"ID":"b4017c78-f096-4332-9d98-389296470ab5","Type":"ContainerDied","Data":"c93f9c13861502bd766f119748163c69aebecc664b77f9af03f65f47d5c44526"}
Sep 11 14:31:44.752162 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:44.751933    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57"
Sep 11 14:31:44.752162 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:44.752005    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57" event={"ID":"b4017c78-f096-4332-9d98-389296470ab5","Type":"ContainerDied","Data":"2f0e2086acd3ec84342920e3b3b6448d35db0f1c721cf91d4fb8f89a4cfbc8d5"}
Sep 11 14:31:44.752162 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:44.752036    2581 scope.go:117] "RemoveContainer" containerID="c93f9c13861502bd766f119748163c69aebecc664b77f9af03f65f47d5c44526"
Sep 11 14:31:44.761332 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:44.761313    2581 scope.go:117] "RemoveContainer" containerID="7ecb575282a22bc0225a22f78e9ee93822f1dde27719377f66816cfb32f5feeb"
Sep 11 14:31:44.769182 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:44.769151    2581 scope.go:117] "RemoveContainer" containerID="c93f9c13861502bd766f119748163c69aebecc664b77f9af03f65f47d5c44526"
Sep 11 14:31:44.769426 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:31:44.769407    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c93f9c13861502bd766f119748163c69aebecc664b77f9af03f65f47d5c44526\": container with ID starting with c93f9c13861502bd766f119748163c69aebecc664b77f9af03f65f47d5c44526 not found: ID does not exist" containerID="c93f9c13861502bd766f119748163c69aebecc664b77f9af03f65f47d5c44526"
Sep 11 14:31:44.769496 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:44.769439    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c93f9c13861502bd766f119748163c69aebecc664b77f9af03f65f47d5c44526"} err="failed to get container status \"c93f9c13861502bd766f119748163c69aebecc664b77f9af03f65f47d5c44526\": rpc error: code = NotFound desc = could not find container \"c93f9c13861502bd766f119748163c69aebecc664b77f9af03f65f47d5c44526\": container with ID starting with c93f9c13861502bd766f119748163c69aebecc664b77f9af03f65f47d5c44526 not found: ID does not exist"
Sep 11 14:31:44.769496 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:44.769463    2581 scope.go:117] "RemoveContainer" containerID="7ecb575282a22bc0225a22f78e9ee93822f1dde27719377f66816cfb32f5feeb"
Sep 11 14:31:44.769769 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:31:44.769750    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ecb575282a22bc0225a22f78e9ee93822f1dde27719377f66816cfb32f5feeb\": container with ID starting with 7ecb575282a22bc0225a22f78e9ee93822f1dde27719377f66816cfb32f5feeb not found: ID does not exist" containerID="7ecb575282a22bc0225a22f78e9ee93822f1dde27719377f66816cfb32f5feeb"
Sep 11 14:31:44.769812 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:44.769776    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ecb575282a22bc0225a22f78e9ee93822f1dde27719377f66816cfb32f5feeb"} err="failed to get container status \"7ecb575282a22bc0225a22f78e9ee93822f1dde27719377f66816cfb32f5feeb\": rpc error: code = NotFound desc = could not find container \"7ecb575282a22bc0225a22f78e9ee93822f1dde27719377f66816cfb32f5feeb\": container with ID starting with 7ecb575282a22bc0225a22f78e9ee93822f1dde27719377f66816cfb32f5feeb not found: ID does not exist"
Sep 11 14:31:44.785800 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:44.785774    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57"]
Sep 11 14:31:44.805612 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:44.805573    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-1b6b934c-openshift-default-854c4b5d56-7hf57"]
Sep 11 14:31:45.499911 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:45.499875    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4017c78-f096-4332-9d98-389296470ab5" path="/var/lib/kubelet/pods/b4017c78-f096-4332-9d98-389296470ab5/volumes"
Sep 11 14:31:55.809101 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:55.809070    2581 generic.go:356] "Generic (PLEG): container finished" podID="8bde0890-f752-4b4b-976f-804512c66e7c" containerID="05b9327f3bdb2bd741dc4dcd8a48cd091862b275dabf80dfeec4ec0a0c2bed59" exitCode=0
Sep 11 14:31:55.809512 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:55.809148    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt" event={"ID":"8bde0890-f752-4b4b-976f-804512c66e7c","Type":"ContainerDied","Data":"05b9327f3bdb2bd741dc4dcd8a48cd091862b275dabf80dfeec4ec0a0c2bed59"}
Sep 11 14:31:55.809609 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:55.809594    2581 scope.go:117] "RemoveContainer" containerID="05b9327f3bdb2bd741dc4dcd8a48cd091862b275dabf80dfeec4ec0a0c2bed59"
Sep 11 14:31:55.810813 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:55.810781    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:31:55.810866 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:55.810853    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:31:55.810901 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:55.810885    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:31:56.819672 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:56.819629    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt" event={"ID":"8bde0890-f752-4b4b-976f-804512c66e7c","Type":"ContainerStarted","Data":"730dd42bf8c9fe1a78019eab32498867f0d998cf1635563316dc9c74c9e01f0a"}
Sep 11 14:31:57.781849 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:57.781814    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt"
Sep 11 14:31:57.782818 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:57.782792    2581 patch_prober.go:28] interesting pod/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Sep 11 14:31:57.782933 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:57.782858    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt" podUID="8bde0890-f752-4b4b-976f-804512c66e7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Sep 11 14:31:58.284323 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:58.284286    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a6adb955-p4h2f"]
Sep 11 14:31:58.284718 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:58.284629    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4017c78-f096-4332-9d98-389296470ab5" containerName="istio-proxy"
Sep 11 14:31:58.284718 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:58.284641    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4017c78-f096-4332-9d98-389296470ab5" containerName="istio-proxy"
Sep 11 14:31:58.284718 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:58.284649    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4017c78-f096-4332-9d98-389296470ab5" containerName="istio-proxy"
Sep 11 14:31:58.284718 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:58.284655    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4017c78-f096-4332-9d98-389296470ab5" containerName="istio-proxy"
Sep 11 14:31:58.284847 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:58.284725    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4017c78-f096-4332-9d98-389296470ab5" containerName="istio-proxy"
Sep 11 14:31:58.284847 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:58.284734    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4017c78-f096-4332-9d98-389296470ab5" containerName="istio-proxy"
Sep 11 14:31:58.289365 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:58.289331    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a6adb955-p4h2f"
Sep 11 14:31:58.293890 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:58.293854    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hcm4g\""
Sep 11 14:31:58.294188 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:58.294166    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 14:31:58.300489 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:58.300459    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a6adb955-p4h2f"]
Sep 11 14:31:58.347260 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:58.347225    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/346887af-6494-4c1a-b941-3d020d082b73-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a6adb955-p4h2f\" (UID: \"346887af-6494-4c1a-b941-3d020d082b73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a6adb955-p4h2f"
Sep 11 14:31:58.448697 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:58.448647    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/346887af-6494-4c1a-b941-3d020d082b73-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a6adb955-p4h2f\" (UID: \"346887af-6494-4c1a-b941-3d020d082b73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a6adb955-p4h2f"
Sep 11 14:31:58.449343 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:58.449321    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/346887af-6494-4c1a-b941-3d020d082b73-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a6adb955-p4h2f\" (UID: \"346887af-6494-4c1a-b941-3d020d082b73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a6adb955-p4h2f"
Sep 11 14:31:58.609833 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:58.609739    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a6adb955-p4h2f"
Sep 11 14:31:58.660507 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:58.659775    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt"
Sep 11 14:31:58.781833 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:58.781795    2581 patch_prober.go:28] interesting pod/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Sep 11 14:31:58.782047 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:58.781850    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt" podUID="8bde0890-f752-4b4b-976f-804512c66e7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Sep 11 14:31:58.903203 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:58.903103    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a6adb955-p4h2f"]
Sep 11 14:31:59.781957 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:59.781921    2581 patch_prober.go:28] interesting pod/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Sep 11 14:31:59.782389 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:59.782002    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt" podUID="8bde0890-f752-4b4b-976f-804512c66e7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Sep 11 14:31:59.841905 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:59.841856    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a6adb955-p4h2f" event={"ID":"346887af-6494-4c1a-b941-3d020d082b73","Type":"ContainerStarted","Data":"adef2127a688f6e6dfd01ae0818d48a64384ef124836c1691f4273dc4052a22b"}
Sep 11 14:31:59.842224 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:59.841925    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a6adb955-p4h2f" event={"ID":"346887af-6494-4c1a-b941-3d020d082b73","Type":"ContainerStarted","Data":"af0177b8d428ef9fd539ac927751724fd6ee0b2f5a3661aecc86342cb25bd688"}
Sep 11 14:31:59.886828 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:31:59.886775    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a6adb955-p4h2f" podStartSLOduration=1.8867568559999999 podStartE2EDuration="1.886756856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:31:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:31:59.884845869 +0000 UTC m=+1023.035395932" watchObservedRunningTime="2026-09-11 14:31:59.886756856 +0000 UTC m=+1023.037306916"
Sep 11 14:32:00.781325 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:00.781284    2581 patch_prober.go:28] interesting pod/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Sep 11 14:32:00.781528 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:00.781445    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt" podUID="8bde0890-f752-4b4b-976f-804512c66e7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Sep 11 14:32:00.849615 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:00.849580    2581 generic.go:356] "Generic (PLEG): container finished" podID="346887af-6494-4c1a-b941-3d020d082b73" containerID="adef2127a688f6e6dfd01ae0818d48a64384ef124836c1691f4273dc4052a22b" exitCode=0
Sep 11 14:32:00.850001 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:00.849631    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a6adb955-p4h2f" event={"ID":"346887af-6494-4c1a-b941-3d020d082b73","Type":"ContainerDied","Data":"adef2127a688f6e6dfd01ae0818d48a64384ef124836c1691f4273dc4052a22b"}
Sep 11 14:32:01.781395 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:01.781357    2581 patch_prober.go:28] interesting pod/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Sep 11 14:32:01.781563 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:01.781426    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt" podUID="8bde0890-f752-4b4b-976f-804512c66e7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Sep 11 14:32:02.099478 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:02.099146    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a6adb955-p4h2f"
Sep 11 14:32:02.184763 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:02.184727    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/346887af-6494-4c1a-b941-3d020d082b73-maas-api-service-ca\") pod \"346887af-6494-4c1a-b941-3d020d082b73\" (UID: \"346887af-6494-4c1a-b941-3d020d082b73\") "
Sep 11 14:32:02.185222 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:02.185190    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/346887af-6494-4c1a-b941-3d020d082b73-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "346887af-6494-4c1a-b941-3d020d082b73" (UID: "346887af-6494-4c1a-b941-3d020d082b73"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:32:02.286207 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:02.286170    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/346887af-6494-4c1a-b941-3d020d082b73-maas-api-service-ca\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:32:02.785526 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:02.785495    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt"
Sep 11 14:32:02.786366 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:02.786347    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt"
Sep 11 14:32:02.868594 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:02.868558    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a6adb955-p4h2f"
Sep 11 14:32:02.868781 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:02.868603    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a6adb955-p4h2f" event={"ID":"346887af-6494-4c1a-b941-3d020d082b73","Type":"ContainerDied","Data":"af0177b8d428ef9fd539ac927751724fd6ee0b2f5a3661aecc86342cb25bd688"}
Sep 11 14:32:02.868781 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:02.868663    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af0177b8d428ef9fd539ac927751724fd6ee0b2f5a3661aecc86342cb25bd688"
Sep 11 14:32:08.900209 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:08.900175    2581 generic.go:356] "Generic (PLEG): container finished" podID="7b08b8a0-c2e7-4095-8eb3-090245fab2b8" containerID="32766f3857dac4e62846a8154af1e13de1c63ac540af1b97bf586a398e6d847d" exitCode=0
Sep 11 14:32:08.900640 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:08.900252    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74" event={"ID":"7b08b8a0-c2e7-4095-8eb3-090245fab2b8","Type":"ContainerDied","Data":"32766f3857dac4e62846a8154af1e13de1c63ac540af1b97bf586a398e6d847d"}
Sep 11 14:32:08.900761 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:08.900743    2581 scope.go:117] "RemoveContainer" containerID="32766f3857dac4e62846a8154af1e13de1c63ac540af1b97bf586a398e6d847d"
Sep 11 14:32:08.901870 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:08.901844    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:32:08.901935 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:08.901910    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:32:08.901983 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:08.901940    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:32:09.916007 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:09.915725    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74" event={"ID":"7b08b8a0-c2e7-4095-8eb3-090245fab2b8","Type":"ContainerStarted","Data":"c62ec268bb77aaf53307dee95eb8768f10a14cd6b397dabd87cbdb7efa968474"}
Sep 11 14:32:09.947038 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:09.947008    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a6adb955-p4h2f"]
Sep 11 14:32:09.957630 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:09.957602    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a6adb955-p4h2f"]
Sep 11 14:32:10.104468 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:10.104417    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74"]
Sep 11 14:32:10.437395 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:10.437348    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74"
Sep 11 14:32:10.438504 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:10.438472    2581 patch_prober.go:28] interesting pod/e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Sep 11 14:32:10.438631 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:10.438557    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74" podUID="7b08b8a0-c2e7-4095-8eb3-090245fab2b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Sep 11 14:32:10.921289 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:10.921243    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74" podUID="7b08b8a0-c2e7-4095-8eb3-090245fab2b8" containerName="istio-proxy" containerID="cri-o://c62ec268bb77aaf53307dee95eb8768f10a14cd6b397dabd87cbdb7efa968474" gracePeriod=30
Sep 11 14:32:11.316640 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:11.316593    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx"]
Sep 11 14:32:11.317203 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:11.317167    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="346887af-6494-4c1a-b941-3d020d082b73" containerName="revoke-keys"
Sep 11 14:32:11.317203 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:11.317204    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="346887af-6494-4c1a-b941-3d020d082b73" containerName="revoke-keys"
Sep 11 14:32:11.317377 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:11.317290    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="346887af-6494-4c1a-b941-3d020d082b73" containerName="revoke-keys"
Sep 11 14:32:11.319585 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:11.319564    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx"
Sep 11 14:32:11.327825 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:11.327790    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-61c24488-openshift-default-dockercfg-bf98c\""
Sep 11 14:32:11.353481 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:11.353447    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx"]
Sep 11 14:32:11.381837 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:11.381796    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c796853a-ed1c-44e6-8994-92f782d6edb3-istio-podinfo\") pod \"e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx\" (UID: \"c796853a-ed1c-44e6-8994-92f782d6edb3\") " pod="openshift-ingress/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx"
Sep 11 14:32:11.382053 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:11.381868    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c796853a-ed1c-44e6-8994-92f782d6edb3-credential-socket\") pod \"e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx\" (UID: \"c796853a-ed1c-44e6-8994-92f782d6edb3\") " pod="openshift-ingress/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx"
Sep 11 14:32:11.382053 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:11.381999    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c796853a-ed1c-44e6-8994-92f782d6edb3-istio-envoy\") pod \"e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx\" (UID: \"c796853a-ed1c-44e6-8994-92f782d6edb3\") " pod="openshift-ingress/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx"
Sep 11 14:32:11.382053 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:11.382031    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljzvb\" (UniqueName: \"kubernetes.io/projected/c796853a-ed1c-44e6-8994-92f782d6edb3-kube-api-access-ljzvb\") pod \"e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx\" (UID: \"c796853a-ed1c-44e6-8994-92f782d6edb3\") " pod="openshift-ingress/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx"
Sep 11 14:32:11.382053 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:11.382055    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c796853a-ed1c-44e6-8994-92f782d6edb3-istio-token\") pod \"e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx\" (UID: \"c796853a-ed1c-44e6-8994-92f782d6edb3\") " pod="openshift-ingress/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx"
Sep 11 14:32:11.382300 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:11.382076    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c796853a-ed1c-44e6-8994-92f782d6edb3-workload-certs\") pod \"e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx\" (UID: \"c796853a-ed1c-44e6-8994-92f782d6edb3\") " pod="openshift-ingress/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx"
Sep 11 14:32:11.382300 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:11.382145    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c796853a-ed1c-44e6-8994-92f782d6edb3-istiod-ca-cert\") pod \"e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx\" (UID: \"c796853a-ed1c-44e6-8994-92f782d6edb3\") " pod="openshift-ingress/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx"
Sep 11 14:32:11.382300 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:11.382234    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c796853a-ed1c-44e6-8994-92f782d6edb3-istio-data\") pod \"e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx\" (UID: \"c796853a-ed1c-44e6-8994-92f782d6edb3\") " pod="openshift-ingress/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx"
Sep 11 14:32:11.382300 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:11.382278    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c796853a-ed1c-44e6-8994-92f782d6edb3-workload-socket\") pod \"e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx\" (UID: \"c796853a-ed1c-44e6-8994-92f782d6edb3\") " pod="openshift-ingress/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx"
Sep 11 14:32:11.483404 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:11.483372    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c796853a-ed1c-44e6-8994-92f782d6edb3-istiod-ca-cert\") pod \"e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx\" (UID: \"c796853a-ed1c-44e6-8994-92f782d6edb3\") " pod="openshift-ingress/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx"
Sep 11 14:32:11.483578 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:11.483437    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c796853a-ed1c-44e6-8994-92f782d6edb3-istio-data\") pod \"e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx\" (UID: \"c796853a-ed1c-44e6-8994-92f782d6edb3\") " pod="openshift-ingress/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx"
Sep 11 14:32:11.483578 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:11.483468    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c796853a-ed1c-44e6-8994-92f782d6edb3-workload-socket\") pod \"e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx\" (UID: \"c796853a-ed1c-44e6-8994-92f782d6edb3\") " pod="openshift-ingress/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx"
Sep 11 14:32:11.483578 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:11.483511    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c796853a-ed1c-44e6-8994-92f782d6edb3-istio-podinfo\") pod \"e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx\" (UID: \"c796853a-ed1c-44e6-8994-92f782d6edb3\") " pod="openshift-ingress/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx"
Sep 11 14:32:11.483578 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:11.483536    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c796853a-ed1c-44e6-8994-92f782d6edb3-credential-socket\") pod \"e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx\" (UID: \"c796853a-ed1c-44e6-8994-92f782d6edb3\") " pod="openshift-ingress/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx"
Sep 11 14:32:11.483793 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:11.483673    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c796853a-ed1c-44e6-8994-92f782d6edb3-istio-envoy\") pod \"e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx\" (UID: \"c796853a-ed1c-44e6-8994-92f782d6edb3\") " pod="openshift-ingress/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx"
Sep 11 14:32:11.483793 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:11.483701    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljzvb\" (UniqueName: \"kubernetes.io/projected/c796853a-ed1c-44e6-8994-92f782d6edb3-kube-api-access-ljzvb\") pod \"e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx\" (UID: \"c796853a-ed1c-44e6-8994-92f782d6edb3\") " pod="openshift-ingress/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx"
Sep 11 14:32:11.483904 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:11.483790    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c796853a-ed1c-44e6-8994-92f782d6edb3-istio-token\") pod \"e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx\" (UID: \"c796853a-ed1c-44e6-8994-92f782d6edb3\") " pod="openshift-ingress/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx"
Sep 11 14:32:11.483904 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:11.483825    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c796853a-ed1c-44e6-8994-92f782d6edb3-workload-certs\") pod \"e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx\" (UID: \"c796853a-ed1c-44e6-8994-92f782d6edb3\") " pod="openshift-ingress/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx"
Sep 11 14:32:11.484032 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:11.483989    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c796853a-ed1c-44e6-8994-92f782d6edb3-workload-socket\") pod \"e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx\" (UID: \"c796853a-ed1c-44e6-8994-92f782d6edb3\") " pod="openshift-ingress/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx"
Sep 11 14:32:11.484092 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:11.484028    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c796853a-ed1c-44e6-8994-92f782d6edb3-credential-socket\") pod \"e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx\" (UID: \"c796853a-ed1c-44e6-8994-92f782d6edb3\") " pod="openshift-ingress/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx"
Sep 11 14:32:11.484159 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:11.484140    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c796853a-ed1c-44e6-8994-92f782d6edb3-istio-data\") pod \"e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx\" (UID: \"c796853a-ed1c-44e6-8994-92f782d6edb3\") " pod="openshift-ingress/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx"
Sep 11 14:32:11.484271 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:11.484249    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c796853a-ed1c-44e6-8994-92f782d6edb3-workload-certs\") pod \"e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx\" (UID: \"c796853a-ed1c-44e6-8994-92f782d6edb3\") " pod="openshift-ingress/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx"
Sep 11 14:32:11.484357 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:11.484279    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c796853a-ed1c-44e6-8994-92f782d6edb3-istiod-ca-cert\") pod \"e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx\" (UID: \"c796853a-ed1c-44e6-8994-92f782d6edb3\") " pod="openshift-ingress/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx"
Sep 11 14:32:11.486015 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:11.485987    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c796853a-ed1c-44e6-8994-92f782d6edb3-istio-envoy\") pod \"e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx\" (UID: \"c796853a-ed1c-44e6-8994-92f782d6edb3\") " pod="openshift-ingress/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx"
Sep 11 14:32:11.486272 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:11.486255    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c796853a-ed1c-44e6-8994-92f782d6edb3-istio-podinfo\") pod \"e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx\" (UID: \"c796853a-ed1c-44e6-8994-92f782d6edb3\") " pod="openshift-ingress/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx"
Sep 11 14:32:11.503012 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:11.501628    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c796853a-ed1c-44e6-8994-92f782d6edb3-istio-token\") pod \"e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx\" (UID: \"c796853a-ed1c-44e6-8994-92f782d6edb3\") " pod="openshift-ingress/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx"
Sep 11 14:32:11.513001 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:11.511064    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="346887af-6494-4c1a-b941-3d020d082b73" path="/var/lib/kubelet/pods/346887af-6494-4c1a-b941-3d020d082b73/volumes"
Sep 11 14:32:11.518739 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:11.518604    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljzvb\" (UniqueName: \"kubernetes.io/projected/c796853a-ed1c-44e6-8994-92f782d6edb3-kube-api-access-ljzvb\") pod \"e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx\" (UID: \"c796853a-ed1c-44e6-8994-92f782d6edb3\") " pod="openshift-ingress/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx"
Sep 11 14:32:11.632880 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:11.632789    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx"
Sep 11 14:32:11.932935 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:11.932602    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx"]
Sep 11 14:32:11.935463 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:32:11.935437    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc796853a_ed1c_44e6_8994_92f782d6edb3.slice/crio-971316ec23f644384f94116744f28f07380b561cb12df804e30ea95757bcbc26 WatchSource:0}: Error finding container 971316ec23f644384f94116744f28f07380b561cb12df804e30ea95757bcbc26: Status 404 returned error can't find the container with id 971316ec23f644384f94116744f28f07380b561cb12df804e30ea95757bcbc26
Sep 11 14:32:11.937694 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:11.937662    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:32:11.937775 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:11.937729    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:32:11.937775 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:11.937767    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:32:12.739143 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:12.739106    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74"
Sep 11 14:32:12.932308 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:12.932270    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx" event={"ID":"c796853a-ed1c-44e6-8994-92f782d6edb3","Type":"ContainerStarted","Data":"037bbf19a399e3bd08301d396ead51d2f466b19141e1451c255108b0398ae7b0"}
Sep 11 14:32:12.932475 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:12.932324    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx" event={"ID":"c796853a-ed1c-44e6-8994-92f782d6edb3","Type":"ContainerStarted","Data":"971316ec23f644384f94116744f28f07380b561cb12df804e30ea95757bcbc26"}
Sep 11 14:32:12.987492 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:12.987440    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx" podStartSLOduration=1.987422396 podStartE2EDuration="1.987422396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:32:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:32:12.986484612 +0000 UTC m=+1036.137034700" watchObservedRunningTime="2026-09-11 14:32:12.987422396 +0000 UTC m=+1036.137972459"
Sep 11 14:32:13.633382 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:13.633346    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx"
Sep 11 14:32:13.639444 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:13.639423    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx"
Sep 11 14:32:13.937436 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:13.937351    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx"
Sep 11 14:32:13.938613 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:13.938592    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx"
Sep 11 14:32:20.669300 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:20.669273    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74"
Sep 11 14:32:20.784409 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:20.784376    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b08b8a0-c2e7-4095-8eb3-090245fab2b8-workload-certs\") pod \"7b08b8a0-c2e7-4095-8eb3-090245fab2b8\" (UID: \"7b08b8a0-c2e7-4095-8eb3-090245fab2b8\") "
Sep 11 14:32:20.784563 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:20.784443    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b08b8a0-c2e7-4095-8eb3-090245fab2b8-istio-podinfo\") pod \"7b08b8a0-c2e7-4095-8eb3-090245fab2b8\" (UID: \"7b08b8a0-c2e7-4095-8eb3-090245fab2b8\") "
Sep 11 14:32:20.784563 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:20.784473    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b08b8a0-c2e7-4095-8eb3-090245fab2b8-credential-socket\") pod \"7b08b8a0-c2e7-4095-8eb3-090245fab2b8\" (UID: \"7b08b8a0-c2e7-4095-8eb3-090245fab2b8\") "
Sep 11 14:32:20.784563 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:20.784512    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b08b8a0-c2e7-4095-8eb3-090245fab2b8-workload-socket\") pod \"7b08b8a0-c2e7-4095-8eb3-090245fab2b8\" (UID: \"7b08b8a0-c2e7-4095-8eb3-090245fab2b8\") "
Sep 11 14:32:20.784563 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:20.784536    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b08b8a0-c2e7-4095-8eb3-090245fab2b8-istio-envoy\") pod \"7b08b8a0-c2e7-4095-8eb3-090245fab2b8\" (UID: \"7b08b8a0-c2e7-4095-8eb3-090245fab2b8\") "
Sep 11 14:32:20.784788 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:20.784574    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b08b8a0-c2e7-4095-8eb3-090245fab2b8-istiod-ca-cert\") pod \"7b08b8a0-c2e7-4095-8eb3-090245fab2b8\" (UID: \"7b08b8a0-c2e7-4095-8eb3-090245fab2b8\") "
Sep 11 14:32:20.784788 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:20.784607    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b08b8a0-c2e7-4095-8eb3-090245fab2b8-istio-data\") pod \"7b08b8a0-c2e7-4095-8eb3-090245fab2b8\" (UID: \"7b08b8a0-c2e7-4095-8eb3-090245fab2b8\") "
Sep 11 14:32:20.784788 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:20.784711    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b08b8a0-c2e7-4095-8eb3-090245fab2b8-istio-token\") pod \"7b08b8a0-c2e7-4095-8eb3-090245fab2b8\" (UID: \"7b08b8a0-c2e7-4095-8eb3-090245fab2b8\") "
Sep 11 14:32:20.784788 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:20.784720    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b08b8a0-c2e7-4095-8eb3-090245fab2b8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7b08b8a0-c2e7-4095-8eb3-090245fab2b8" (UID: "7b08b8a0-c2e7-4095-8eb3-090245fab2b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:32:20.784788 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:20.784753    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-krcb6\" (UniqueName: \"kubernetes.io/projected/7b08b8a0-c2e7-4095-8eb3-090245fab2b8-kube-api-access-krcb6\") pod \"7b08b8a0-c2e7-4095-8eb3-090245fab2b8\" (UID: \"7b08b8a0-c2e7-4095-8eb3-090245fab2b8\") "
Sep 11 14:32:20.785393 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:20.784819    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b08b8a0-c2e7-4095-8eb3-090245fab2b8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7b08b8a0-c2e7-4095-8eb3-090245fab2b8" (UID: "7b08b8a0-c2e7-4095-8eb3-090245fab2b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:32:20.785393 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:20.784927    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b08b8a0-c2e7-4095-8eb3-090245fab2b8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7b08b8a0-c2e7-4095-8eb3-090245fab2b8" (UID: "7b08b8a0-c2e7-4095-8eb3-090245fab2b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:32:20.785393 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:20.785019    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b08b8a0-c2e7-4095-8eb3-090245fab2b8-istio-data" (OuterVolumeSpecName: "istio-data") pod "7b08b8a0-c2e7-4095-8eb3-090245fab2b8" (UID: "7b08b8a0-c2e7-4095-8eb3-090245fab2b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:32:20.785393 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:20.785057    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b08b8a0-c2e7-4095-8eb3-090245fab2b8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7b08b8a0-c2e7-4095-8eb3-090245fab2b8" (UID: "7b08b8a0-c2e7-4095-8eb3-090245fab2b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:32:20.785393 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:20.785184    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b08b8a0-c2e7-4095-8eb3-090245fab2b8-workload-certs\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:32:20.785393 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:20.785213    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b08b8a0-c2e7-4095-8eb3-090245fab2b8-credential-socket\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:32:20.785393 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:20.785230    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b08b8a0-c2e7-4095-8eb3-090245fab2b8-workload-socket\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:32:20.785393 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:20.785244    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b08b8a0-c2e7-4095-8eb3-090245fab2b8-istiod-ca-cert\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:32:20.785393 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:20.785257    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b08b8a0-c2e7-4095-8eb3-090245fab2b8-istio-data\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:32:20.786875 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:20.786850    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b08b8a0-c2e7-4095-8eb3-090245fab2b8-istio-token" (OuterVolumeSpecName: "istio-token") pod "7b08b8a0-c2e7-4095-8eb3-090245fab2b8" (UID: "7b08b8a0-c2e7-4095-8eb3-090245fab2b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:32:20.787017 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:20.786907    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b08b8a0-c2e7-4095-8eb3-090245fab2b8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7b08b8a0-c2e7-4095-8eb3-090245fab2b8" (UID: "7b08b8a0-c2e7-4095-8eb3-090245fab2b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 14:32:20.787213 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:20.787191    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b08b8a0-c2e7-4095-8eb3-090245fab2b8-kube-api-access-krcb6" (OuterVolumeSpecName: "kube-api-access-krcb6") pod "7b08b8a0-c2e7-4095-8eb3-090245fab2b8" (UID: "7b08b8a0-c2e7-4095-8eb3-090245fab2b8"). InnerVolumeSpecName "kube-api-access-krcb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:32:20.787273 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:20.787253    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b08b8a0-c2e7-4095-8eb3-090245fab2b8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7b08b8a0-c2e7-4095-8eb3-090245fab2b8" (UID: "7b08b8a0-c2e7-4095-8eb3-090245fab2b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:32:20.886521 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:20.886488    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b08b8a0-c2e7-4095-8eb3-090245fab2b8-istio-token\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:32:20.886521 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:20.886517    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krcb6\" (UniqueName: \"kubernetes.io/projected/7b08b8a0-c2e7-4095-8eb3-090245fab2b8-kube-api-access-krcb6\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:32:20.886521 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:20.886528    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b08b8a0-c2e7-4095-8eb3-090245fab2b8-istio-podinfo\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:32:20.886740 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:20.886536    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b08b8a0-c2e7-4095-8eb3-090245fab2b8-istio-envoy\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:32:20.978787 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:20.978751    2581 generic.go:356] "Generic (PLEG): container finished" podID="7b08b8a0-c2e7-4095-8eb3-090245fab2b8" containerID="c62ec268bb77aaf53307dee95eb8768f10a14cd6b397dabd87cbdb7efa968474" exitCode=0
Sep 11 14:32:20.979008 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:20.978805    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74"
Sep 11 14:32:20.979008 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:20.978831    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74" event={"ID":"7b08b8a0-c2e7-4095-8eb3-090245fab2b8","Type":"ContainerDied","Data":"c62ec268bb77aaf53307dee95eb8768f10a14cd6b397dabd87cbdb7efa968474"}
Sep 11 14:32:20.979008 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:20.978915    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74" event={"ID":"7b08b8a0-c2e7-4095-8eb3-090245fab2b8","Type":"ContainerDied","Data":"14fc57c21e9208600ca083cd9a75c349e48321c6d5409f9f37e887304494636a"}
Sep 11 14:32:20.979008 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:20.978947    2581 scope.go:117] "RemoveContainer" containerID="c62ec268bb77aaf53307dee95eb8768f10a14cd6b397dabd87cbdb7efa968474"
Sep 11 14:32:20.988122 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:20.988105    2581 scope.go:117] "RemoveContainer" containerID="32766f3857dac4e62846a8154af1e13de1c63ac540af1b97bf586a398e6d847d"
Sep 11 14:32:20.996511 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:20.996491    2581 scope.go:117] "RemoveContainer" containerID="c62ec268bb77aaf53307dee95eb8768f10a14cd6b397dabd87cbdb7efa968474"
Sep 11 14:32:20.996772 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:32:20.996755    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c62ec268bb77aaf53307dee95eb8768f10a14cd6b397dabd87cbdb7efa968474\": container with ID starting with c62ec268bb77aaf53307dee95eb8768f10a14cd6b397dabd87cbdb7efa968474 not found: ID does not exist" containerID="c62ec268bb77aaf53307dee95eb8768f10a14cd6b397dabd87cbdb7efa968474"
Sep 11 14:32:20.996825 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:20.996779    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c62ec268bb77aaf53307dee95eb8768f10a14cd6b397dabd87cbdb7efa968474"} err="failed to get container status \"c62ec268bb77aaf53307dee95eb8768f10a14cd6b397dabd87cbdb7efa968474\": rpc error: code = NotFound desc = could not find container \"c62ec268bb77aaf53307dee95eb8768f10a14cd6b397dabd87cbdb7efa968474\": container with ID starting with c62ec268bb77aaf53307dee95eb8768f10a14cd6b397dabd87cbdb7efa968474 not found: ID does not exist"
Sep 11 14:32:20.996825 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:20.996796    2581 scope.go:117] "RemoveContainer" containerID="32766f3857dac4e62846a8154af1e13de1c63ac540af1b97bf586a398e6d847d"
Sep 11 14:32:20.997064 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:32:20.997047    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32766f3857dac4e62846a8154af1e13de1c63ac540af1b97bf586a398e6d847d\": container with ID starting with 32766f3857dac4e62846a8154af1e13de1c63ac540af1b97bf586a398e6d847d not found: ID does not exist" containerID="32766f3857dac4e62846a8154af1e13de1c63ac540af1b97bf586a398e6d847d"
Sep 11 14:32:20.997131 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:20.997066    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32766f3857dac4e62846a8154af1e13de1c63ac540af1b97bf586a398e6d847d"} err="failed to get container status \"32766f3857dac4e62846a8154af1e13de1c63ac540af1b97bf586a398e6d847d\": rpc error: code = NotFound desc = could not find container \"32766f3857dac4e62846a8154af1e13de1c63ac540af1b97bf586a398e6d847d\": container with ID starting with 32766f3857dac4e62846a8154af1e13de1c63ac540af1b97bf586a398e6d847d not found: ID does not exist"
Sep 11 14:32:21.023873 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:21.023846    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74"]
Sep 11 14:32:21.068357 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:21.068275    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-a6adb955-gw-openshift-default-7d6fc5db8-gtf74"]
Sep 11 14:32:21.499924 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:21.499896    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b08b8a0-c2e7-4095-8eb3-090245fab2b8" path="/var/lib/kubelet/pods/7b08b8a0-c2e7-4095-8eb3-090245fab2b8/volumes"
Sep 11 14:32:34.052538 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:34.052505    2581 generic.go:356] "Generic (PLEG): container finished" podID="c796853a-ed1c-44e6-8994-92f782d6edb3" containerID="037bbf19a399e3bd08301d396ead51d2f466b19141e1451c255108b0398ae7b0" exitCode=0
Sep 11 14:32:34.052942 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:34.052563    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx" event={"ID":"c796853a-ed1c-44e6-8994-92f782d6edb3","Type":"ContainerDied","Data":"037bbf19a399e3bd08301d396ead51d2f466b19141e1451c255108b0398ae7b0"}
Sep 11 14:32:34.053002 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:34.052963    2581 scope.go:117] "RemoveContainer" containerID="037bbf19a399e3bd08301d396ead51d2f466b19141e1451c255108b0398ae7b0"
Sep 11 14:32:34.054336 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:34.054305    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:32:34.054423 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:34.054376    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:32:34.054423 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:34.054409    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:32:35.065924 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:35.065891    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx" event={"ID":"c796853a-ed1c-44e6-8994-92f782d6edb3","Type":"ContainerStarted","Data":"7cbcff64b4ee3a086fafb996bb7139067f3b74d2e90a95c37a1b04f5d5cb0121"}
Sep 11 14:32:35.633409 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:35.633370    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx"
Sep 11 14:32:35.633654 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:35.633625    2581 patch_prober.go:28] interesting pod/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Sep 11 14:32:35.633713 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:35.633694    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx" podUID="c796853a-ed1c-44e6-8994-92f782d6edb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Sep 11 14:32:36.633537 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:36.633496    2581 patch_prober.go:28] interesting pod/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Sep 11 14:32:36.634003 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:36.633574    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx" podUID="c796853a-ed1c-44e6-8994-92f782d6edb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Sep 11 14:32:37.633849 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:37.633813    2581 patch_prober.go:28] interesting pod/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Sep 11 14:32:37.634276 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:37.633872    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx" podUID="c796853a-ed1c-44e6-8994-92f782d6edb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Sep 11 14:32:38.634146 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:38.634108    2581 patch_prober.go:28] interesting pod/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Sep 11 14:32:38.634583 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:38.634171    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx" podUID="c796853a-ed1c-44e6-8994-92f782d6edb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Sep 11 14:32:39.137148 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:39.137105    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-ca8d752a-kserve-54d7d48496-lvk89"]
Sep 11 14:32:39.137572 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:39.137551    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b08b8a0-c2e7-4095-8eb3-090245fab2b8" containerName="istio-proxy"
Sep 11 14:32:39.137667 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:39.137574    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b08b8a0-c2e7-4095-8eb3-090245fab2b8" containerName="istio-proxy"
Sep 11 14:32:39.137667 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:39.137593    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b08b8a0-c2e7-4095-8eb3-090245fab2b8" containerName="istio-proxy"
Sep 11 14:32:39.137667 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:39.137602    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b08b8a0-c2e7-4095-8eb3-090245fab2b8" containerName="istio-proxy"
Sep 11 14:32:39.137816 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:39.137710    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b08b8a0-c2e7-4095-8eb3-090245fab2b8" containerName="istio-proxy"
Sep 11 14:32:39.137816 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:39.137725    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b08b8a0-c2e7-4095-8eb3-090245fab2b8" containerName="istio-proxy"
Sep 11 14:32:39.140731 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:39.140709    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-ca8d752a-kserve-54d7d48496-lvk89"
Sep 11 14:32:39.143645 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:39.143625    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-ca8d752a-kserve-self-signed-certs\""
Sep 11 14:32:39.162077 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:39.162056    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-ca8d752a-kserve-54d7d48496-lvk89"]
Sep 11 14:32:39.272659 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:39.272623    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26a3b068-3f7c-4b11-97df-1494cb551008-tmp-dir\") pod \"e2e-conflict-b-ca8d752a-kserve-54d7d48496-lvk89\" (UID: \"26a3b068-3f7c-4b11-97df-1494cb551008\") " pod="llm/e2e-conflict-b-ca8d752a-kserve-54d7d48496-lvk89"
Sep 11 14:32:39.272849 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:39.272689    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26a3b068-3f7c-4b11-97df-1494cb551008-model-cache\") pod \"e2e-conflict-b-ca8d752a-kserve-54d7d48496-lvk89\" (UID: \"26a3b068-3f7c-4b11-97df-1494cb551008\") " pod="llm/e2e-conflict-b-ca8d752a-kserve-54d7d48496-lvk89"
Sep 11 14:32:39.272849 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:39.272763    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26a3b068-3f7c-4b11-97df-1494cb551008-tls-certs\") pod \"e2e-conflict-b-ca8d752a-kserve-54d7d48496-lvk89\" (UID: \"26a3b068-3f7c-4b11-97df-1494cb551008\") " pod="llm/e2e-conflict-b-ca8d752a-kserve-54d7d48496-lvk89"
Sep 11 14:32:39.272849 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:39.272817    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26a3b068-3f7c-4b11-97df-1494cb551008-home\") pod \"e2e-conflict-b-ca8d752a-kserve-54d7d48496-lvk89\" (UID: \"26a3b068-3f7c-4b11-97df-1494cb551008\") " pod="llm/e2e-conflict-b-ca8d752a-kserve-54d7d48496-lvk89"
Sep 11 14:32:39.273021 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:39.272867    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26a3b068-3f7c-4b11-97df-1494cb551008-dshm\") pod \"e2e-conflict-b-ca8d752a-kserve-54d7d48496-lvk89\" (UID: \"26a3b068-3f7c-4b11-97df-1494cb551008\") " pod="llm/e2e-conflict-b-ca8d752a-kserve-54d7d48496-lvk89"
Sep 11 14:32:39.273021 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:39.272893    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgw2b\" (UniqueName: \"kubernetes.io/projected/26a3b068-3f7c-4b11-97df-1494cb551008-kube-api-access-vgw2b\") pod \"e2e-conflict-b-ca8d752a-kserve-54d7d48496-lvk89\" (UID: \"26a3b068-3f7c-4b11-97df-1494cb551008\") " pod="llm/e2e-conflict-b-ca8d752a-kserve-54d7d48496-lvk89"
Sep 11 14:32:39.374067 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:39.374036    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26a3b068-3f7c-4b11-97df-1494cb551008-model-cache\") pod \"e2e-conflict-b-ca8d752a-kserve-54d7d48496-lvk89\" (UID: \"26a3b068-3f7c-4b11-97df-1494cb551008\") " pod="llm/e2e-conflict-b-ca8d752a-kserve-54d7d48496-lvk89"
Sep 11 14:32:39.374407 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:39.374382    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26a3b068-3f7c-4b11-97df-1494cb551008-tls-certs\") pod \"e2e-conflict-b-ca8d752a-kserve-54d7d48496-lvk89\" (UID: \"26a3b068-3f7c-4b11-97df-1494cb551008\") " pod="llm/e2e-conflict-b-ca8d752a-kserve-54d7d48496-lvk89"
Sep 11 14:32:39.374549 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:39.374525    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26a3b068-3f7c-4b11-97df-1494cb551008-model-cache\") pod \"e2e-conflict-b-ca8d752a-kserve-54d7d48496-lvk89\" (UID: \"26a3b068-3f7c-4b11-97df-1494cb551008\") " pod="llm/e2e-conflict-b-ca8d752a-kserve-54d7d48496-lvk89"
Sep 11 14:32:39.374726 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:39.374704    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26a3b068-3f7c-4b11-97df-1494cb551008-home\") pod \"e2e-conflict-b-ca8d752a-kserve-54d7d48496-lvk89\" (UID: \"26a3b068-3f7c-4b11-97df-1494cb551008\") " pod="llm/e2e-conflict-b-ca8d752a-kserve-54d7d48496-lvk89"
Sep 11 14:32:39.374913 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:39.374896    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26a3b068-3f7c-4b11-97df-1494cb551008-dshm\") pod \"e2e-conflict-b-ca8d752a-kserve-54d7d48496-lvk89\" (UID: \"26a3b068-3f7c-4b11-97df-1494cb551008\") " pod="llm/e2e-conflict-b-ca8d752a-kserve-54d7d48496-lvk89"
Sep 11 14:32:39.375071 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:39.375054    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgw2b\" (UniqueName: \"kubernetes.io/projected/26a3b068-3f7c-4b11-97df-1494cb551008-kube-api-access-vgw2b\") pod \"e2e-conflict-b-ca8d752a-kserve-54d7d48496-lvk89\" (UID: \"26a3b068-3f7c-4b11-97df-1494cb551008\") " pod="llm/e2e-conflict-b-ca8d752a-kserve-54d7d48496-lvk89"
Sep 11 14:32:39.375220 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:39.375000    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26a3b068-3f7c-4b11-97df-1494cb551008-home\") pod \"e2e-conflict-b-ca8d752a-kserve-54d7d48496-lvk89\" (UID: \"26a3b068-3f7c-4b11-97df-1494cb551008\") " pod="llm/e2e-conflict-b-ca8d752a-kserve-54d7d48496-lvk89"
Sep 11 14:32:39.375325 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:39.375308    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26a3b068-3f7c-4b11-97df-1494cb551008-tmp-dir\") pod \"e2e-conflict-b-ca8d752a-kserve-54d7d48496-lvk89\" (UID: \"26a3b068-3f7c-4b11-97df-1494cb551008\") " pod="llm/e2e-conflict-b-ca8d752a-kserve-54d7d48496-lvk89"
Sep 11 14:32:39.375718 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:39.375696    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26a3b068-3f7c-4b11-97df-1494cb551008-tmp-dir\") pod \"e2e-conflict-b-ca8d752a-kserve-54d7d48496-lvk89\" (UID: \"26a3b068-3f7c-4b11-97df-1494cb551008\") " pod="llm/e2e-conflict-b-ca8d752a-kserve-54d7d48496-lvk89"
Sep 11 14:32:39.377628 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:39.377594    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26a3b068-3f7c-4b11-97df-1494cb551008-tls-certs\") pod \"e2e-conflict-b-ca8d752a-kserve-54d7d48496-lvk89\" (UID: \"26a3b068-3f7c-4b11-97df-1494cb551008\") " pod="llm/e2e-conflict-b-ca8d752a-kserve-54d7d48496-lvk89"
Sep 11 14:32:39.378415 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:39.378390    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26a3b068-3f7c-4b11-97df-1494cb551008-dshm\") pod \"e2e-conflict-b-ca8d752a-kserve-54d7d48496-lvk89\" (UID: \"26a3b068-3f7c-4b11-97df-1494cb551008\") " pod="llm/e2e-conflict-b-ca8d752a-kserve-54d7d48496-lvk89"
Sep 11 14:32:39.405406 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:39.405295    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgw2b\" (UniqueName: \"kubernetes.io/projected/26a3b068-3f7c-4b11-97df-1494cb551008-kube-api-access-vgw2b\") pod \"e2e-conflict-b-ca8d752a-kserve-54d7d48496-lvk89\" (UID: \"26a3b068-3f7c-4b11-97df-1494cb551008\") " pod="llm/e2e-conflict-b-ca8d752a-kserve-54d7d48496-lvk89"
Sep 11 14:32:39.449825 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:39.449792    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-ca8d752a-kserve-54d7d48496-lvk89"
Sep 11 14:32:39.633186 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:39.633151    2581 patch_prober.go:28] interesting pod/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Sep 11 14:32:39.633363 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:39.633207    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx" podUID="c796853a-ed1c-44e6-8994-92f782d6edb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Sep 11 14:32:39.760914 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:39.760892    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-ca8d752a-kserve-54d7d48496-lvk89"]
Sep 11 14:32:39.763011 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:32:39.762957    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod26a3b068_3f7c_4b11_97df_1494cb551008.slice/crio-d979cc518cb50dcc9522a2f4c6fb7cc71cab917f706d042d2df768a8fe2ff0b9 WatchSource:0}: Error finding container d979cc518cb50dcc9522a2f4c6fb7cc71cab917f706d042d2df768a8fe2ff0b9: Status 404 returned error can't find the container with id d979cc518cb50dcc9522a2f4c6fb7cc71cab917f706d042d2df768a8fe2ff0b9
Sep 11 14:32:40.092839 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:40.092748    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-ca8d752a-kserve-54d7d48496-lvk89" event={"ID":"26a3b068-3f7c-4b11-97df-1494cb551008","Type":"ContainerStarted","Data":"6a09b3d5489f164564b94de34a6278a8b637c99e57698ea970fad8f273e4e799"}
Sep 11 14:32:40.092839 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:40.092795    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-ca8d752a-kserve-54d7d48496-lvk89" event={"ID":"26a3b068-3f7c-4b11-97df-1494cb551008","Type":"ContainerStarted","Data":"d979cc518cb50dcc9522a2f4c6fb7cc71cab917f706d042d2df768a8fe2ff0b9"}
Sep 11 14:32:40.120213 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:40.120153    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-ca8d752a-kserve-54d7d48496-lvk89" podStartSLOduration=1.120130719 podStartE2EDuration="1.120130719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:32:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:32:40.117269472 +0000 UTC m=+1063.267819541" watchObservedRunningTime="2026-09-11 14:32:40.120130719 +0000 UTC m=+1063.270680785"
Sep 11 14:32:40.637120 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:40.637086    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx"
Sep 11 14:32:40.637402 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:40.637372    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx"
Sep 11 14:32:40.638072 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:40.638054    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx"
Sep 11 14:32:45.638031 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:45.637986    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61c24488-6fvq6"]
Sep 11 14:32:45.641678 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:45.641659    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61c24488-6fvq6"
Sep 11 14:32:45.644687 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:45.644659    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 14:32:45.644784 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:45.644721    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hcm4g\""
Sep 11 14:32:45.655348 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:45.655326    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61c24488-6fvq6"]
Sep 11 14:32:45.742785 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:45.742740    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bded667d-3e50-4f2f-b5f6-b848b5499ff6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-61c24488-6fvq6\" (UID: \"bded667d-3e50-4f2f-b5f6-b848b5499ff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61c24488-6fvq6"
Sep 11 14:32:45.843518 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:45.843483    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bded667d-3e50-4f2f-b5f6-b848b5499ff6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-61c24488-6fvq6\" (UID: \"bded667d-3e50-4f2f-b5f6-b848b5499ff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61c24488-6fvq6"
Sep 11 14:32:45.844154 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:45.844132    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bded667d-3e50-4f2f-b5f6-b848b5499ff6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-61c24488-6fvq6\" (UID: \"bded667d-3e50-4f2f-b5f6-b848b5499ff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61c24488-6fvq6"
Sep 11 14:32:45.952082 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:45.951962    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61c24488-6fvq6"
Sep 11 14:32:46.223623 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:46.223528    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61c24488-6fvq6"]
Sep 11 14:32:46.226436 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:32:46.226408    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbded667d_3e50_4f2f_b5f6_b848b5499ff6.slice/crio-ecc1505b12f189864adae0f99f9163c339d1fa72ca30f0a4ac040eabdb0f2779 WatchSource:0}: Error finding container ecc1505b12f189864adae0f99f9163c339d1fa72ca30f0a4ac040eabdb0f2779: Status 404 returned error can't find the container with id ecc1505b12f189864adae0f99f9163c339d1fa72ca30f0a4ac040eabdb0f2779
Sep 11 14:32:47.129124 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:47.129085    2581 generic.go:356] "Generic (PLEG): container finished" podID="bded667d-3e50-4f2f-b5f6-b848b5499ff6" containerID="768d9479b2f64b80b09c9c97d92d1fda8c8a93b3fcaef1e645c0f3ac1fe62119" exitCode=0
Sep 11 14:32:47.129555 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:47.129178    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61c24488-6fvq6" event={"ID":"bded667d-3e50-4f2f-b5f6-b848b5499ff6","Type":"ContainerDied","Data":"768d9479b2f64b80b09c9c97d92d1fda8c8a93b3fcaef1e645c0f3ac1fe62119"}
Sep 11 14:32:47.129555 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:47.129240    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61c24488-6fvq6" event={"ID":"bded667d-3e50-4f2f-b5f6-b848b5499ff6","Type":"ContainerStarted","Data":"ecc1505b12f189864adae0f99f9163c339d1fa72ca30f0a4ac040eabdb0f2779"}
Sep 11 14:32:48.264959 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:48.264928    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61c24488-6fvq6"
Sep 11 14:32:48.372175 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:48.372137    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bded667d-3e50-4f2f-b5f6-b848b5499ff6-maas-api-service-ca\") pod \"bded667d-3e50-4f2f-b5f6-b848b5499ff6\" (UID: \"bded667d-3e50-4f2f-b5f6-b848b5499ff6\") "
Sep 11 14:32:48.372500 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:48.372477    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bded667d-3e50-4f2f-b5f6-b848b5499ff6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bded667d-3e50-4f2f-b5f6-b848b5499ff6" (UID: "bded667d-3e50-4f2f-b5f6-b848b5499ff6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:32:48.473570 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:48.473490    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bded667d-3e50-4f2f-b5f6-b848b5499ff6-maas-api-service-ca\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:32:49.140046 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:49.140017    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61c24488-6fvq6"
Sep 11 14:32:49.140046 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:49.140032    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61c24488-6fvq6" event={"ID":"bded667d-3e50-4f2f-b5f6-b848b5499ff6","Type":"ContainerDied","Data":"ecc1505b12f189864adae0f99f9163c339d1fa72ca30f0a4ac040eabdb0f2779"}
Sep 11 14:32:49.140268 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:49.140084    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ecc1505b12f189864adae0f99f9163c339d1fa72ca30f0a4ac040eabdb0f2779"
Sep 11 14:32:49.449932 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:49.449846    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-ca8d752a-kserve-54d7d48496-lvk89"
Sep 11 14:32:49.449932 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:49.449898    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-ca8d752a-kserve-54d7d48496-lvk89"
Sep 11 14:32:49.462664 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:49.462635    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-ca8d752a-kserve-54d7d48496-lvk89"
Sep 11 14:32:49.579336 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:49.579254    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-84d3a8/autores-model-84d3a8-kserve-6494496fc9-xvg8t"]
Sep 11 14:32:49.579823 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:49.579796    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bded667d-3e50-4f2f-b5f6-b848b5499ff6" containerName="revoke-keys"
Sep 11 14:32:49.579926 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:49.579825    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="bded667d-3e50-4f2f-b5f6-b848b5499ff6" containerName="revoke-keys"
Sep 11 14:32:49.579989 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:49.579952    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="bded667d-3e50-4f2f-b5f6-b848b5499ff6" containerName="revoke-keys"
Sep 11 14:32:49.583374 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:49.583352    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-84d3a8/autores-model-84d3a8-kserve-6494496fc9-xvg8t"
Sep 11 14:32:49.587983 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:49.587948    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-84d3a8\"/\"openshift-service-ca.crt\""
Sep 11 14:32:49.588083 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:49.588015    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-84d3a8\"/\"default-dockercfg-gntvx\""
Sep 11 14:32:49.588083 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:49.588037    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-84d3a8\"/\"kube-root-ca.crt\""
Sep 11 14:32:49.588217 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:49.588111    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-84d3a8\"/\"autores-model-84d3a8-kserve-self-signed-certs\""
Sep 11 14:32:49.600304 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:49.600277    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-84d3a8/autores-model-84d3a8-kserve-6494496fc9-xvg8t"]
Sep 11 14:32:49.685211 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:49.685168    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f16836a9-2f8c-43d6-834e-1f0c9dd93ec3-home\") pod \"autores-model-84d3a8-kserve-6494496fc9-xvg8t\" (UID: \"f16836a9-2f8c-43d6-834e-1f0c9dd93ec3\") " pod="ai-tenant-e2e-autores-84d3a8/autores-model-84d3a8-kserve-6494496fc9-xvg8t"
Sep 11 14:32:49.685427 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:49.685234    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f16836a9-2f8c-43d6-834e-1f0c9dd93ec3-model-cache\") pod \"autores-model-84d3a8-kserve-6494496fc9-xvg8t\" (UID: \"f16836a9-2f8c-43d6-834e-1f0c9dd93ec3\") " pod="ai-tenant-e2e-autores-84d3a8/autores-model-84d3a8-kserve-6494496fc9-xvg8t"
Sep 11 14:32:49.685427 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:49.685330    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f16836a9-2f8c-43d6-834e-1f0c9dd93ec3-tmp-dir\") pod \"autores-model-84d3a8-kserve-6494496fc9-xvg8t\" (UID: \"f16836a9-2f8c-43d6-834e-1f0c9dd93ec3\") " pod="ai-tenant-e2e-autores-84d3a8/autores-model-84d3a8-kserve-6494496fc9-xvg8t"
Sep 11 14:32:49.685557 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:49.685458    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f16836a9-2f8c-43d6-834e-1f0c9dd93ec3-tls-certs\") pod \"autores-model-84d3a8-kserve-6494496fc9-xvg8t\" (UID: \"f16836a9-2f8c-43d6-834e-1f0c9dd93ec3\") " pod="ai-tenant-e2e-autores-84d3a8/autores-model-84d3a8-kserve-6494496fc9-xvg8t"
Sep 11 14:32:49.685557 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:49.685519    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f16836a9-2f8c-43d6-834e-1f0c9dd93ec3-dshm\") pod \"autores-model-84d3a8-kserve-6494496fc9-xvg8t\" (UID: \"f16836a9-2f8c-43d6-834e-1f0c9dd93ec3\") " pod="ai-tenant-e2e-autores-84d3a8/autores-model-84d3a8-kserve-6494496fc9-xvg8t"
Sep 11 14:32:49.685677 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:49.685558    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjmb8\" (UniqueName: \"kubernetes.io/projected/f16836a9-2f8c-43d6-834e-1f0c9dd93ec3-kube-api-access-tjmb8\") pod \"autores-model-84d3a8-kserve-6494496fc9-xvg8t\" (UID: \"f16836a9-2f8c-43d6-834e-1f0c9dd93ec3\") " pod="ai-tenant-e2e-autores-84d3a8/autores-model-84d3a8-kserve-6494496fc9-xvg8t"
Sep 11 14:32:49.786348 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:49.786319    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f16836a9-2f8c-43d6-834e-1f0c9dd93ec3-tls-certs\") pod \"autores-model-84d3a8-kserve-6494496fc9-xvg8t\" (UID: \"f16836a9-2f8c-43d6-834e-1f0c9dd93ec3\") " pod="ai-tenant-e2e-autores-84d3a8/autores-model-84d3a8-kserve-6494496fc9-xvg8t"
Sep 11 14:32:49.786530 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:49.786365    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f16836a9-2f8c-43d6-834e-1f0c9dd93ec3-dshm\") pod \"autores-model-84d3a8-kserve-6494496fc9-xvg8t\" (UID: \"f16836a9-2f8c-43d6-834e-1f0c9dd93ec3\") " pod="ai-tenant-e2e-autores-84d3a8/autores-model-84d3a8-kserve-6494496fc9-xvg8t"
Sep 11 14:32:49.786530 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:49.786390    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjmb8\" (UniqueName: \"kubernetes.io/projected/f16836a9-2f8c-43d6-834e-1f0c9dd93ec3-kube-api-access-tjmb8\") pod \"autores-model-84d3a8-kserve-6494496fc9-xvg8t\" (UID: \"f16836a9-2f8c-43d6-834e-1f0c9dd93ec3\") " pod="ai-tenant-e2e-autores-84d3a8/autores-model-84d3a8-kserve-6494496fc9-xvg8t"
Sep 11 14:32:49.786530 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:49.786421    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f16836a9-2f8c-43d6-834e-1f0c9dd93ec3-home\") pod \"autores-model-84d3a8-kserve-6494496fc9-xvg8t\" (UID: \"f16836a9-2f8c-43d6-834e-1f0c9dd93ec3\") " pod="ai-tenant-e2e-autores-84d3a8/autores-model-84d3a8-kserve-6494496fc9-xvg8t"
Sep 11 14:32:49.786530 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:49.786443    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f16836a9-2f8c-43d6-834e-1f0c9dd93ec3-model-cache\") pod \"autores-model-84d3a8-kserve-6494496fc9-xvg8t\" (UID: \"f16836a9-2f8c-43d6-834e-1f0c9dd93ec3\") " pod="ai-tenant-e2e-autores-84d3a8/autores-model-84d3a8-kserve-6494496fc9-xvg8t"
Sep 11 14:32:49.786744 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:49.786592    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f16836a9-2f8c-43d6-834e-1f0c9dd93ec3-tmp-dir\") pod \"autores-model-84d3a8-kserve-6494496fc9-xvg8t\" (UID: \"f16836a9-2f8c-43d6-834e-1f0c9dd93ec3\") " pod="ai-tenant-e2e-autores-84d3a8/autores-model-84d3a8-kserve-6494496fc9-xvg8t"
Sep 11 14:32:49.786823 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:49.786807    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f16836a9-2f8c-43d6-834e-1f0c9dd93ec3-home\") pod \"autores-model-84d3a8-kserve-6494496fc9-xvg8t\" (UID: \"f16836a9-2f8c-43d6-834e-1f0c9dd93ec3\") " pod="ai-tenant-e2e-autores-84d3a8/autores-model-84d3a8-kserve-6494496fc9-xvg8t"
Sep 11 14:32:49.786903 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:49.786858    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f16836a9-2f8c-43d6-834e-1f0c9dd93ec3-model-cache\") pod \"autores-model-84d3a8-kserve-6494496fc9-xvg8t\" (UID: \"f16836a9-2f8c-43d6-834e-1f0c9dd93ec3\") " pod="ai-tenant-e2e-autores-84d3a8/autores-model-84d3a8-kserve-6494496fc9-xvg8t"
Sep 11 14:32:49.786962 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:49.786944    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f16836a9-2f8c-43d6-834e-1f0c9dd93ec3-tmp-dir\") pod \"autores-model-84d3a8-kserve-6494496fc9-xvg8t\" (UID: \"f16836a9-2f8c-43d6-834e-1f0c9dd93ec3\") " pod="ai-tenant-e2e-autores-84d3a8/autores-model-84d3a8-kserve-6494496fc9-xvg8t"
Sep 11 14:32:49.788765 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:49.788740    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f16836a9-2f8c-43d6-834e-1f0c9dd93ec3-dshm\") pod \"autores-model-84d3a8-kserve-6494496fc9-xvg8t\" (UID: \"f16836a9-2f8c-43d6-834e-1f0c9dd93ec3\") " pod="ai-tenant-e2e-autores-84d3a8/autores-model-84d3a8-kserve-6494496fc9-xvg8t"
Sep 11 14:32:49.789054 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:49.789037    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f16836a9-2f8c-43d6-834e-1f0c9dd93ec3-tls-certs\") pod \"autores-model-84d3a8-kserve-6494496fc9-xvg8t\" (UID: \"f16836a9-2f8c-43d6-834e-1f0c9dd93ec3\") " pod="ai-tenant-e2e-autores-84d3a8/autores-model-84d3a8-kserve-6494496fc9-xvg8t"
Sep 11 14:32:49.799820 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:49.799800    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjmb8\" (UniqueName: \"kubernetes.io/projected/f16836a9-2f8c-43d6-834e-1f0c9dd93ec3-kube-api-access-tjmb8\") pod \"autores-model-84d3a8-kserve-6494496fc9-xvg8t\" (UID: \"f16836a9-2f8c-43d6-834e-1f0c9dd93ec3\") " pod="ai-tenant-e2e-autores-84d3a8/autores-model-84d3a8-kserve-6494496fc9-xvg8t"
Sep 11 14:32:49.893621 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:49.893571    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-84d3a8/autores-model-84d3a8-kserve-6494496fc9-xvg8t"
Sep 11 14:32:50.158694 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:50.158600    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-ca8d752a-kserve-54d7d48496-lvk89"
Sep 11 14:32:50.176887 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:50.176863    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-84d3a8/autores-model-84d3a8-kserve-6494496fc9-xvg8t"]
Sep 11 14:32:50.179161 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:32:50.179127    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf16836a9_2f8c_43d6_834e_1f0c9dd93ec3.slice/crio-540fe7eafa5ae5f721b46cc77151ccd648137049eaa12e5cf54b6ad4963715e3 WatchSource:0}: Error finding container 540fe7eafa5ae5f721b46cc77151ccd648137049eaa12e5cf54b6ad4963715e3: Status 404 returned error can't find the container with id 540fe7eafa5ae5f721b46cc77151ccd648137049eaa12e5cf54b6ad4963715e3
Sep 11 14:32:51.153220 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:51.153182    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-84d3a8/autores-model-84d3a8-kserve-6494496fc9-xvg8t" event={"ID":"f16836a9-2f8c-43d6-834e-1f0c9dd93ec3","Type":"ContainerStarted","Data":"d99964b89e07257312fac69beea29a13d1f371ed519bc0f351590c60d3d1df29"}
Sep 11 14:32:51.153638 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:51.153230    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-84d3a8/autores-model-84d3a8-kserve-6494496fc9-xvg8t" event={"ID":"f16836a9-2f8c-43d6-834e-1f0c9dd93ec3","Type":"ContainerStarted","Data":"540fe7eafa5ae5f721b46cc77151ccd648137049eaa12e5cf54b6ad4963715e3"}
Sep 11 14:32:51.174811 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:51.174726    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-84d3a8/autores-model-84d3a8-kserve-6494496fc9-xvg8t" podStartSLOduration=2.174708098 podStartE2EDuration="2.174708098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:32:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:32:51.173722821 +0000 UTC m=+1074.324272885" watchObservedRunningTime="2026-09-11 14:32:51.174708098 +0000 UTC m=+1074.325258179"
Sep 11 14:32:57.872636 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:57.872595    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61c24488-6fvq6"]
Sep 11 14:32:57.882302 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:57.882267    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61c24488-6fvq6"]
Sep 11 14:32:59.247769 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:59.247734    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx"]
Sep 11 14:32:59.248178 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:59.247978    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx" podUID="c796853a-ed1c-44e6-8994-92f782d6edb3" containerName="istio-proxy" containerID="cri-o://7cbcff64b4ee3a086fafb996bb7139067f3b74d2e90a95c37a1b04f5d5cb0121" gracePeriod=30
Sep 11 14:32:59.499856 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:59.499775    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bded667d-3e50-4f2f-b5f6-b848b5499ff6" path="/var/lib/kubelet/pods/bded667d-3e50-4f2f-b5f6-b848b5499ff6/volumes"
Sep 11 14:32:59.894324 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:59.894218    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-84d3a8/autores-model-84d3a8-kserve-6494496fc9-xvg8t"
Sep 11 14:32:59.894324 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:59.894280    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-84d3a8/autores-model-84d3a8-kserve-6494496fc9-xvg8t"
Sep 11 14:32:59.906543 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:32:59.906520    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-84d3a8/autores-model-84d3a8-kserve-6494496fc9-xvg8t"
Sep 11 14:33:00.232809 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:00.232420    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-84d3a8/autores-model-84d3a8-kserve-6494496fc9-xvg8t"
Sep 11 14:33:04.492445 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:04.492413    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx"
Sep 11 14:33:04.516500 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:04.516421    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c796853a-ed1c-44e6-8994-92f782d6edb3-workload-socket\") pod \"c796853a-ed1c-44e6-8994-92f782d6edb3\" (UID: \"c796853a-ed1c-44e6-8994-92f782d6edb3\") "
Sep 11 14:33:04.516500 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:04.516468    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c796853a-ed1c-44e6-8994-92f782d6edb3-istio-podinfo\") pod \"c796853a-ed1c-44e6-8994-92f782d6edb3\" (UID: \"c796853a-ed1c-44e6-8994-92f782d6edb3\") "
Sep 11 14:33:04.516718 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:04.516507    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c796853a-ed1c-44e6-8994-92f782d6edb3-istio-token\") pod \"c796853a-ed1c-44e6-8994-92f782d6edb3\" (UID: \"c796853a-ed1c-44e6-8994-92f782d6edb3\") "
Sep 11 14:33:04.516718 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:04.516528    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljzvb\" (UniqueName: \"kubernetes.io/projected/c796853a-ed1c-44e6-8994-92f782d6edb3-kube-api-access-ljzvb\") pod \"c796853a-ed1c-44e6-8994-92f782d6edb3\" (UID: \"c796853a-ed1c-44e6-8994-92f782d6edb3\") "
Sep 11 14:33:04.516718 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:04.516578    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c796853a-ed1c-44e6-8994-92f782d6edb3-credential-socket\") pod \"c796853a-ed1c-44e6-8994-92f782d6edb3\" (UID: \"c796853a-ed1c-44e6-8994-92f782d6edb3\") "
Sep 11 14:33:04.516718 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:04.516606    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c796853a-ed1c-44e6-8994-92f782d6edb3-istiod-ca-cert\") pod \"c796853a-ed1c-44e6-8994-92f782d6edb3\" (UID: \"c796853a-ed1c-44e6-8994-92f782d6edb3\") "
Sep 11 14:33:04.516718 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:04.516680    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c796853a-ed1c-44e6-8994-92f782d6edb3-workload-certs\") pod \"c796853a-ed1c-44e6-8994-92f782d6edb3\" (UID: \"c796853a-ed1c-44e6-8994-92f782d6edb3\") "
Sep 11 14:33:04.516718 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:04.516715    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c796853a-ed1c-44e6-8994-92f782d6edb3-istio-envoy\") pod \"c796853a-ed1c-44e6-8994-92f782d6edb3\" (UID: \"c796853a-ed1c-44e6-8994-92f782d6edb3\") "
Sep 11 14:33:04.517059 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:04.516745    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c796853a-ed1c-44e6-8994-92f782d6edb3-istio-data\") pod \"c796853a-ed1c-44e6-8994-92f782d6edb3\" (UID: \"c796853a-ed1c-44e6-8994-92f782d6edb3\") "
Sep 11 14:33:04.517059 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:04.516737    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c796853a-ed1c-44e6-8994-92f782d6edb3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c796853a-ed1c-44e6-8994-92f782d6edb3" (UID: "c796853a-ed1c-44e6-8994-92f782d6edb3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:33:04.517165 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:04.516948    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c796853a-ed1c-44e6-8994-92f782d6edb3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c796853a-ed1c-44e6-8994-92f782d6edb3" (UID: "c796853a-ed1c-44e6-8994-92f782d6edb3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:33:04.517165 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:04.516954    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c796853a-ed1c-44e6-8994-92f782d6edb3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c796853a-ed1c-44e6-8994-92f782d6edb3" (UID: "c796853a-ed1c-44e6-8994-92f782d6edb3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:33:04.517165 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:04.517126    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c796853a-ed1c-44e6-8994-92f782d6edb3-istio-data" (OuterVolumeSpecName: "istio-data") pod "c796853a-ed1c-44e6-8994-92f782d6edb3" (UID: "c796853a-ed1c-44e6-8994-92f782d6edb3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:33:04.517316 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:04.517185    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c796853a-ed1c-44e6-8994-92f782d6edb3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c796853a-ed1c-44e6-8994-92f782d6edb3" (UID: "c796853a-ed1c-44e6-8994-92f782d6edb3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:33:04.518983 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:04.518936    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c796853a-ed1c-44e6-8994-92f782d6edb3-istio-token" (OuterVolumeSpecName: "istio-token") pod "c796853a-ed1c-44e6-8994-92f782d6edb3" (UID: "c796853a-ed1c-44e6-8994-92f782d6edb3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:33:04.519092 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:04.519024    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c796853a-ed1c-44e6-8994-92f782d6edb3-kube-api-access-ljzvb" (OuterVolumeSpecName: "kube-api-access-ljzvb") pod "c796853a-ed1c-44e6-8994-92f782d6edb3" (UID: "c796853a-ed1c-44e6-8994-92f782d6edb3"). InnerVolumeSpecName "kube-api-access-ljzvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:33:04.519136 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:04.519116    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c796853a-ed1c-44e6-8994-92f782d6edb3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c796853a-ed1c-44e6-8994-92f782d6edb3" (UID: "c796853a-ed1c-44e6-8994-92f782d6edb3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 14:33:04.519170 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:04.519135    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c796853a-ed1c-44e6-8994-92f782d6edb3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c796853a-ed1c-44e6-8994-92f782d6edb3" (UID: "c796853a-ed1c-44e6-8994-92f782d6edb3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:33:04.618262 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:04.618225    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c796853a-ed1c-44e6-8994-92f782d6edb3-workload-socket\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:33:04.618262 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:04.618260    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c796853a-ed1c-44e6-8994-92f782d6edb3-istio-podinfo\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:33:04.618473 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:04.618276    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c796853a-ed1c-44e6-8994-92f782d6edb3-istio-token\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:33:04.618473 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:04.618291    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljzvb\" (UniqueName: \"kubernetes.io/projected/c796853a-ed1c-44e6-8994-92f782d6edb3-kube-api-access-ljzvb\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:33:04.618473 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:04.618305    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c796853a-ed1c-44e6-8994-92f782d6edb3-credential-socket\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:33:04.618473 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:04.618318    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c796853a-ed1c-44e6-8994-92f782d6edb3-istiod-ca-cert\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:33:04.618473 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:04.618331    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c796853a-ed1c-44e6-8994-92f782d6edb3-workload-certs\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:33:04.618473 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:04.618344    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c796853a-ed1c-44e6-8994-92f782d6edb3-istio-envoy\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:33:04.618473 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:04.618355    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c796853a-ed1c-44e6-8994-92f782d6edb3-istio-data\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:33:05.236705 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:05.236673    2581 generic.go:356] "Generic (PLEG): container finished" podID="c796853a-ed1c-44e6-8994-92f782d6edb3" containerID="7cbcff64b4ee3a086fafb996bb7139067f3b74d2e90a95c37a1b04f5d5cb0121" exitCode=0
Sep 11 14:33:05.236962 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:05.236725    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx"
Sep 11 14:33:05.236962 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:05.236766    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx" event={"ID":"c796853a-ed1c-44e6-8994-92f782d6edb3","Type":"ContainerDied","Data":"7cbcff64b4ee3a086fafb996bb7139067f3b74d2e90a95c37a1b04f5d5cb0121"}
Sep 11 14:33:05.236962 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:05.236821    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx" event={"ID":"c796853a-ed1c-44e6-8994-92f782d6edb3","Type":"ContainerDied","Data":"971316ec23f644384f94116744f28f07380b561cb12df804e30ea95757bcbc26"}
Sep 11 14:33:05.236962 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:05.236850    2581 scope.go:117] "RemoveContainer" containerID="7cbcff64b4ee3a086fafb996bb7139067f3b74d2e90a95c37a1b04f5d5cb0121"
Sep 11 14:33:05.248864 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:05.248843    2581 scope.go:117] "RemoveContainer" containerID="037bbf19a399e3bd08301d396ead51d2f466b19141e1451c255108b0398ae7b0"
Sep 11 14:33:05.257655 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:05.257637    2581 scope.go:117] "RemoveContainer" containerID="7cbcff64b4ee3a086fafb996bb7139067f3b74d2e90a95c37a1b04f5d5cb0121"
Sep 11 14:33:05.257925 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:33:05.257904    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cbcff64b4ee3a086fafb996bb7139067f3b74d2e90a95c37a1b04f5d5cb0121\": container with ID starting with 7cbcff64b4ee3a086fafb996bb7139067f3b74d2e90a95c37a1b04f5d5cb0121 not found: ID does not exist" containerID="7cbcff64b4ee3a086fafb996bb7139067f3b74d2e90a95c37a1b04f5d5cb0121"
Sep 11 14:33:05.257993 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:05.257934    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cbcff64b4ee3a086fafb996bb7139067f3b74d2e90a95c37a1b04f5d5cb0121"} err="failed to get container status \"7cbcff64b4ee3a086fafb996bb7139067f3b74d2e90a95c37a1b04f5d5cb0121\": rpc error: code = NotFound desc = could not find container \"7cbcff64b4ee3a086fafb996bb7139067f3b74d2e90a95c37a1b04f5d5cb0121\": container with ID starting with 7cbcff64b4ee3a086fafb996bb7139067f3b74d2e90a95c37a1b04f5d5cb0121 not found: ID does not exist"
Sep 11 14:33:05.257993 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:05.257951    2581 scope.go:117] "RemoveContainer" containerID="037bbf19a399e3bd08301d396ead51d2f466b19141e1451c255108b0398ae7b0"
Sep 11 14:33:05.258221 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:33:05.258203    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"037bbf19a399e3bd08301d396ead51d2f466b19141e1451c255108b0398ae7b0\": container with ID starting with 037bbf19a399e3bd08301d396ead51d2f466b19141e1451c255108b0398ae7b0 not found: ID does not exist" containerID="037bbf19a399e3bd08301d396ead51d2f466b19141e1451c255108b0398ae7b0"
Sep 11 14:33:05.258281 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:05.258231    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"037bbf19a399e3bd08301d396ead51d2f466b19141e1451c255108b0398ae7b0"} err="failed to get container status \"037bbf19a399e3bd08301d396ead51d2f466b19141e1451c255108b0398ae7b0\": rpc error: code = NotFound desc = could not find container \"037bbf19a399e3bd08301d396ead51d2f466b19141e1451c255108b0398ae7b0\": container with ID starting with 037bbf19a399e3bd08301d396ead51d2f466b19141e1451c255108b0398ae7b0 not found: ID does not exist"
Sep 11 14:33:05.268639 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:05.268615    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx"]
Sep 11 14:33:05.276950 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:05.276926    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-61c24488-openshift-default-6b96d66d96-jltbx"]
Sep 11 14:33:05.500127 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:05.500096    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c796853a-ed1c-44e6-8994-92f782d6edb3" path="/var/lib/kubelet/pods/c796853a-ed1c-44e6-8994-92f782d6edb3/volumes"
Sep 11 14:33:23.636828 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:23.636795    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-e85961/test-model-e85961-kserve-544759b8b5-xqh2b"]
Sep 11 14:33:23.637371 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:23.637353    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c796853a-ed1c-44e6-8994-92f782d6edb3" containerName="istio-proxy"
Sep 11 14:33:23.637423 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:23.637375    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c796853a-ed1c-44e6-8994-92f782d6edb3" containerName="istio-proxy"
Sep 11 14:33:23.637423 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:23.637398    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c796853a-ed1c-44e6-8994-92f782d6edb3" containerName="istio-proxy"
Sep 11 14:33:23.637423 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:23.637406    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c796853a-ed1c-44e6-8994-92f782d6edb3" containerName="istio-proxy"
Sep 11 14:33:23.637530 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:23.637502    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c796853a-ed1c-44e6-8994-92f782d6edb3" containerName="istio-proxy"
Sep 11 14:33:23.637530 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:23.637514    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c796853a-ed1c-44e6-8994-92f782d6edb3" containerName="istio-proxy"
Sep 11 14:33:23.642103 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:23.642057    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e85961/test-model-e85961-kserve-544759b8b5-xqh2b"
Sep 11 14:33:23.645356 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:23.645135    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e85961\"/\"test-model-e85961-kserve-self-signed-certs\""
Sep 11 14:33:23.645356 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:23.645205    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e85961\"/\"kube-root-ca.crt\""
Sep 11 14:33:23.645356 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:23.645245    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e85961\"/\"openshift-service-ca.crt\""
Sep 11 14:33:23.645615 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:23.645445    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e85961\"/\"default-dockercfg-p8m79\""
Sep 11 14:33:23.657906 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:23.657879    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e85961/test-model-e85961-kserve-544759b8b5-xqh2b"]
Sep 11 14:33:23.697399 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:23.697358    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq8b9\" (UniqueName: \"kubernetes.io/projected/f5685f06-073d-48b6-8d7b-b332528573c1-kube-api-access-nq8b9\") pod \"test-model-e85961-kserve-544759b8b5-xqh2b\" (UID: \"f5685f06-073d-48b6-8d7b-b332528573c1\") " pod="ai-tenant-e2e-inf-a-e85961/test-model-e85961-kserve-544759b8b5-xqh2b"
Sep 11 14:33:23.697579 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:23.697429    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5685f06-073d-48b6-8d7b-b332528573c1-tls-certs\") pod \"test-model-e85961-kserve-544759b8b5-xqh2b\" (UID: \"f5685f06-073d-48b6-8d7b-b332528573c1\") " pod="ai-tenant-e2e-inf-a-e85961/test-model-e85961-kserve-544759b8b5-xqh2b"
Sep 11 14:33:23.697579 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:23.697447    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5685f06-073d-48b6-8d7b-b332528573c1-model-cache\") pod \"test-model-e85961-kserve-544759b8b5-xqh2b\" (UID: \"f5685f06-073d-48b6-8d7b-b332528573c1\") " pod="ai-tenant-e2e-inf-a-e85961/test-model-e85961-kserve-544759b8b5-xqh2b"
Sep 11 14:33:23.697579 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:23.697482    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5685f06-073d-48b6-8d7b-b332528573c1-dshm\") pod \"test-model-e85961-kserve-544759b8b5-xqh2b\" (UID: \"f5685f06-073d-48b6-8d7b-b332528573c1\") " pod="ai-tenant-e2e-inf-a-e85961/test-model-e85961-kserve-544759b8b5-xqh2b"
Sep 11 14:33:23.697579 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:23.697512    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5685f06-073d-48b6-8d7b-b332528573c1-tmp-dir\") pod \"test-model-e85961-kserve-544759b8b5-xqh2b\" (UID: \"f5685f06-073d-48b6-8d7b-b332528573c1\") " pod="ai-tenant-e2e-inf-a-e85961/test-model-e85961-kserve-544759b8b5-xqh2b"
Sep 11 14:33:23.697579 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:23.697547    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5685f06-073d-48b6-8d7b-b332528573c1-home\") pod \"test-model-e85961-kserve-544759b8b5-xqh2b\" (UID: \"f5685f06-073d-48b6-8d7b-b332528573c1\") " pod="ai-tenant-e2e-inf-a-e85961/test-model-e85961-kserve-544759b8b5-xqh2b"
Sep 11 14:33:23.798360 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:23.798323    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5685f06-073d-48b6-8d7b-b332528573c1-dshm\") pod \"test-model-e85961-kserve-544759b8b5-xqh2b\" (UID: \"f5685f06-073d-48b6-8d7b-b332528573c1\") " pod="ai-tenant-e2e-inf-a-e85961/test-model-e85961-kserve-544759b8b5-xqh2b"
Sep 11 14:33:23.798564 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:23.798386    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5685f06-073d-48b6-8d7b-b332528573c1-tmp-dir\") pod \"test-model-e85961-kserve-544759b8b5-xqh2b\" (UID: \"f5685f06-073d-48b6-8d7b-b332528573c1\") " pod="ai-tenant-e2e-inf-a-e85961/test-model-e85961-kserve-544759b8b5-xqh2b"
Sep 11 14:33:23.798564 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:23.798433    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5685f06-073d-48b6-8d7b-b332528573c1-home\") pod \"test-model-e85961-kserve-544759b8b5-xqh2b\" (UID: \"f5685f06-073d-48b6-8d7b-b332528573c1\") " pod="ai-tenant-e2e-inf-a-e85961/test-model-e85961-kserve-544759b8b5-xqh2b"
Sep 11 14:33:23.798564 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:23.798475    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq8b9\" (UniqueName: \"kubernetes.io/projected/f5685f06-073d-48b6-8d7b-b332528573c1-kube-api-access-nq8b9\") pod \"test-model-e85961-kserve-544759b8b5-xqh2b\" (UID: \"f5685f06-073d-48b6-8d7b-b332528573c1\") " pod="ai-tenant-e2e-inf-a-e85961/test-model-e85961-kserve-544759b8b5-xqh2b"
Sep 11 14:33:23.798564 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:23.798550    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5685f06-073d-48b6-8d7b-b332528573c1-tls-certs\") pod \"test-model-e85961-kserve-544759b8b5-xqh2b\" (UID: \"f5685f06-073d-48b6-8d7b-b332528573c1\") " pod="ai-tenant-e2e-inf-a-e85961/test-model-e85961-kserve-544759b8b5-xqh2b"
Sep 11 14:33:23.798764 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:23.798576    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5685f06-073d-48b6-8d7b-b332528573c1-model-cache\") pod \"test-model-e85961-kserve-544759b8b5-xqh2b\" (UID: \"f5685f06-073d-48b6-8d7b-b332528573c1\") " pod="ai-tenant-e2e-inf-a-e85961/test-model-e85961-kserve-544759b8b5-xqh2b"
Sep 11 14:33:23.798873 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:23.798849    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5685f06-073d-48b6-8d7b-b332528573c1-home\") pod \"test-model-e85961-kserve-544759b8b5-xqh2b\" (UID: \"f5685f06-073d-48b6-8d7b-b332528573c1\") " pod="ai-tenant-e2e-inf-a-e85961/test-model-e85961-kserve-544759b8b5-xqh2b"
Sep 11 14:33:23.798935 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:23.798859    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5685f06-073d-48b6-8d7b-b332528573c1-tmp-dir\") pod \"test-model-e85961-kserve-544759b8b5-xqh2b\" (UID: \"f5685f06-073d-48b6-8d7b-b332528573c1\") " pod="ai-tenant-e2e-inf-a-e85961/test-model-e85961-kserve-544759b8b5-xqh2b"
Sep 11 14:33:23.798935 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:23.798903    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5685f06-073d-48b6-8d7b-b332528573c1-model-cache\") pod \"test-model-e85961-kserve-544759b8b5-xqh2b\" (UID: \"f5685f06-073d-48b6-8d7b-b332528573c1\") " pod="ai-tenant-e2e-inf-a-e85961/test-model-e85961-kserve-544759b8b5-xqh2b"
Sep 11 14:33:23.800690 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:23.800671    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5685f06-073d-48b6-8d7b-b332528573c1-dshm\") pod \"test-model-e85961-kserve-544759b8b5-xqh2b\" (UID: \"f5685f06-073d-48b6-8d7b-b332528573c1\") " pod="ai-tenant-e2e-inf-a-e85961/test-model-e85961-kserve-544759b8b5-xqh2b"
Sep 11 14:33:23.800846 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:23.800830    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5685f06-073d-48b6-8d7b-b332528573c1-tls-certs\") pod \"test-model-e85961-kserve-544759b8b5-xqh2b\" (UID: \"f5685f06-073d-48b6-8d7b-b332528573c1\") " pod="ai-tenant-e2e-inf-a-e85961/test-model-e85961-kserve-544759b8b5-xqh2b"
Sep 11 14:33:23.814139 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:23.814115    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq8b9\" (UniqueName: \"kubernetes.io/projected/f5685f06-073d-48b6-8d7b-b332528573c1-kube-api-access-nq8b9\") pod \"test-model-e85961-kserve-544759b8b5-xqh2b\" (UID: \"f5685f06-073d-48b6-8d7b-b332528573c1\") " pod="ai-tenant-e2e-inf-a-e85961/test-model-e85961-kserve-544759b8b5-xqh2b"
Sep 11 14:33:23.952262 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:23.952169    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e85961/test-model-e85961-kserve-544759b8b5-xqh2b"
Sep 11 14:33:24.241597 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:24.241571    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e85961/test-model-e85961-kserve-544759b8b5-xqh2b"]
Sep 11 14:33:24.244098 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:33:24.244075    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5685f06_073d_48b6_8d7b_b332528573c1.slice/crio-ffa363b4a8648aa26f2469238ad186864a2838f6eb02ae4940e8d000bc5f8d26 WatchSource:0}: Error finding container ffa363b4a8648aa26f2469238ad186864a2838f6eb02ae4940e8d000bc5f8d26: Status 404 returned error can't find the container with id ffa363b4a8648aa26f2469238ad186864a2838f6eb02ae4940e8d000bc5f8d26
Sep 11 14:33:24.336608 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:24.336573    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e85961/test-model-e85961-kserve-544759b8b5-xqh2b" event={"ID":"f5685f06-073d-48b6-8d7b-b332528573c1","Type":"ContainerStarted","Data":"65baeffb2ea960736fd9a375823267d068b958c4a16d337e174807bd7ce58bbc"}
Sep 11 14:33:24.336608 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:24.336615    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e85961/test-model-e85961-kserve-544759b8b5-xqh2b" event={"ID":"f5685f06-073d-48b6-8d7b-b332528573c1","Type":"ContainerStarted","Data":"ffa363b4a8648aa26f2469238ad186864a2838f6eb02ae4940e8d000bc5f8d26"}
Sep 11 14:33:24.378116 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:24.378061    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-e85961/test-model-e85961-kserve-544759b8b5-xqh2b" podStartSLOduration=1.378043013 podStartE2EDuration="1.378043013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:33:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:33:24.376525402 +0000 UTC m=+1107.527075463" watchObservedRunningTime="2026-09-11 14:33:24.378043013 +0000 UTC m=+1107.528593074"
Sep 11 14:33:25.237152 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:25.237118    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-84d3a8-mh94g"]
Sep 11 14:33:25.240484 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:25.240465    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-84d3a8-mh94g"
Sep 11 14:33:25.243756 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:25.243736    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 14:33:25.243876 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:25.243796    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hcm4g\""
Sep 11 14:33:25.255131 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:25.255104    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-84d3a8-mh94g"]
Sep 11 14:33:25.316247 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:25.316208    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c2ebc05-4fde-4d34-bf50-160665efdf5c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-84d3a8-mh94g\" (UID: \"0c2ebc05-4fde-4d34-bf50-160665efdf5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-84d3a8-mh94g"
Sep 11 14:33:25.417099 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:25.417062    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c2ebc05-4fde-4d34-bf50-160665efdf5c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-84d3a8-mh94g\" (UID: \"0c2ebc05-4fde-4d34-bf50-160665efdf5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-84d3a8-mh94g"
Sep 11 14:33:25.417804 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:25.417779    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c2ebc05-4fde-4d34-bf50-160665efdf5c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-84d3a8-mh94g\" (UID: \"0c2ebc05-4fde-4d34-bf50-160665efdf5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-84d3a8-mh94g"
Sep 11 14:33:25.551515 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:25.551483    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-84d3a8-mh94g"
Sep 11 14:33:25.843675 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:25.843651    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-84d3a8-mh94g"]
Sep 11 14:33:25.845956 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:33:25.845931    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c2ebc05_4fde_4d34_bf50_160665efdf5c.slice/crio-a7ed05b85cce4833f2791821c31564898fa501970c34d562c003babd03435fad WatchSource:0}: Error finding container a7ed05b85cce4833f2791821c31564898fa501970c34d562c003babd03435fad: Status 404 returned error can't find the container with id a7ed05b85cce4833f2791821c31564898fa501970c34d562c003babd03435fad
Sep 11 14:33:26.293841 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:26.293809    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-ca8d752a-kserve-54d7d48496-lvk89"]
Sep 11 14:33:26.347384 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:26.347346    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-84d3a8-mh94g" event={"ID":"0c2ebc05-4fde-4d34-bf50-160665efdf5c","Type":"ContainerStarted","Data":"e9a4580fe27c714b8b3a3d21b9edb5303fe110d46fd1b316c9ae14ce9c99d220"}
Sep 11 14:33:26.347584 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:26.347399    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-84d3a8-mh94g" event={"ID":"0c2ebc05-4fde-4d34-bf50-160665efdf5c","Type":"ContainerStarted","Data":"a7ed05b85cce4833f2791821c31564898fa501970c34d562c003babd03435fad"}
Sep 11 14:33:26.391299 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:26.391249    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-84d3a8-mh94g" podStartSLOduration=1.3912314129999999 podStartE2EDuration="1.391231413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:33:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:33:26.389933279 +0000 UTC m=+1109.540483354" watchObservedRunningTime="2026-09-11 14:33:26.391231413 +0000 UTC m=+1109.541781478"
Sep 11 14:33:27.356469 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:27.356430    2581 generic.go:356] "Generic (PLEG): container finished" podID="0c2ebc05-4fde-4d34-bf50-160665efdf5c" containerID="e9a4580fe27c714b8b3a3d21b9edb5303fe110d46fd1b316c9ae14ce9c99d220" exitCode=0
Sep 11 14:33:27.356860 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:27.356520    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-84d3a8-mh94g" event={"ID":"0c2ebc05-4fde-4d34-bf50-160665efdf5c","Type":"ContainerDied","Data":"e9a4580fe27c714b8b3a3d21b9edb5303fe110d46fd1b316c9ae14ce9c99d220"}
Sep 11 14:33:27.565242 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:27.565210    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-84d3a8/autores-model-84d3a8-kserve-6494496fc9-xvg8t"]
Sep 11 14:33:28.492116 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:28.492092    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-84d3a8-mh94g"
Sep 11 14:33:28.550702 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:28.550670    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c2ebc05-4fde-4d34-bf50-160665efdf5c-maas-api-service-ca\") pod \"0c2ebc05-4fde-4d34-bf50-160665efdf5c\" (UID: \"0c2ebc05-4fde-4d34-bf50-160665efdf5c\") "
Sep 11 14:33:28.551052 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:28.551027    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c2ebc05-4fde-4d34-bf50-160665efdf5c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0c2ebc05-4fde-4d34-bf50-160665efdf5c" (UID: "0c2ebc05-4fde-4d34-bf50-160665efdf5c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:33:28.647146 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:28.647065    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-ca8d752a-kserve-5f75475f98-5bmm9"]
Sep 11 14:33:28.652471 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:28.652450    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c2ebc05-4fde-4d34-bf50-160665efdf5c-maas-api-service-ca\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:33:29.369561 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:29.369534    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-84d3a8-mh94g"
Sep 11 14:33:29.369750 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:29.369576    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-84d3a8-mh94g" event={"ID":"0c2ebc05-4fde-4d34-bf50-160665efdf5c","Type":"ContainerDied","Data":"a7ed05b85cce4833f2791821c31564898fa501970c34d562c003babd03435fad"}
Sep 11 14:33:29.369750 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:29.369649    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7ed05b85cce4833f2791821c31564898fa501970c34d562c003babd03435fad"
Sep 11 14:33:32.888149 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:32.888117    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93c38c4c-cllbh"]
Sep 11 14:33:32.888540 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:32.888485    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c2ebc05-4fde-4d34-bf50-160665efdf5c" containerName="revoke-keys"
Sep 11 14:33:32.888540 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:32.888498    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c2ebc05-4fde-4d34-bf50-160665efdf5c" containerName="revoke-keys"
Sep 11 14:33:32.888613 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:32.888558    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c2ebc05-4fde-4d34-bf50-160665efdf5c" containerName="revoke-keys"
Sep 11 14:33:32.891596 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:32.891579    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93c38c4c-cllbh"
Sep 11 14:33:32.916831 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:32.916806    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hcm4g\""
Sep 11 14:33:32.917000 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:32.916812    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 14:33:32.928431 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:32.928397    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93c38c4c-cllbh"]
Sep 11 14:33:32.995140 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:32.995110    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fa06311-1a87-4419-952d-71be0a1151e6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-93c38c4c-cllbh\" (UID: \"0fa06311-1a87-4419-952d-71be0a1151e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93c38c4c-cllbh"
Sep 11 14:33:33.096219 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:33.096176    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fa06311-1a87-4419-952d-71be0a1151e6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-93c38c4c-cllbh\" (UID: \"0fa06311-1a87-4419-952d-71be0a1151e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93c38c4c-cllbh"
Sep 11 14:33:33.096785 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:33.096763    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fa06311-1a87-4419-952d-71be0a1151e6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-93c38c4c-cllbh\" (UID: \"0fa06311-1a87-4419-952d-71be0a1151e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93c38c4c-cllbh"
Sep 11 14:33:33.209108 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:33.209017    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93c38c4c-cllbh"
Sep 11 14:33:33.515451 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:33.515426    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93c38c4c-cllbh"]
Sep 11 14:33:33.517262 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:33:33.517235    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fa06311_1a87_4419_952d_71be0a1151e6.slice/crio-2149cdca118fcd454be1142645c986bfc9912d4a35cb04155680fe7633b04eda WatchSource:0}: Error finding container 2149cdca118fcd454be1142645c986bfc9912d4a35cb04155680fe7633b04eda: Status 404 returned error can't find the container with id 2149cdca118fcd454be1142645c986bfc9912d4a35cb04155680fe7633b04eda
Sep 11 14:33:33.953347 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:33.953257    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-e85961/test-model-e85961-kserve-544759b8b5-xqh2b"
Sep 11 14:33:33.953347 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:33.953309    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-e85961/test-model-e85961-kserve-544759b8b5-xqh2b"
Sep 11 14:33:33.965718 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:33.965689    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-e85961/test-model-e85961-kserve-544759b8b5-xqh2b"
Sep 11 14:33:34.397358 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:34.397327    2581 generic.go:356] "Generic (PLEG): container finished" podID="0fa06311-1a87-4419-952d-71be0a1151e6" containerID="f0ff2ac905bd6941bd1ab4570b5b55ece4f310a6006268e70d84aa2808f2be81" exitCode=0
Sep 11 14:33:34.397511 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:34.397388    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93c38c4c-cllbh" event={"ID":"0fa06311-1a87-4419-952d-71be0a1151e6","Type":"ContainerDied","Data":"f0ff2ac905bd6941bd1ab4570b5b55ece4f310a6006268e70d84aa2808f2be81"}
Sep 11 14:33:34.397511 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:34.397426    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93c38c4c-cllbh" event={"ID":"0fa06311-1a87-4419-952d-71be0a1151e6","Type":"ContainerStarted","Data":"2149cdca118fcd454be1142645c986bfc9912d4a35cb04155680fe7633b04eda"}
Sep 11 14:33:34.408991 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:34.408941    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-e85961/test-model-e85961-kserve-544759b8b5-xqh2b"
Sep 11 14:33:35.534696 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:35.534672    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93c38c4c-cllbh"
Sep 11 14:33:35.622315 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:35.622286    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fa06311-1a87-4419-952d-71be0a1151e6-maas-api-service-ca\") pod \"0fa06311-1a87-4419-952d-71be0a1151e6\" (UID: \"0fa06311-1a87-4419-952d-71be0a1151e6\") "
Sep 11 14:33:35.622671 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:35.622637    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fa06311-1a87-4419-952d-71be0a1151e6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0fa06311-1a87-4419-952d-71be0a1151e6" (UID: "0fa06311-1a87-4419-952d-71be0a1151e6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:33:35.622884 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:35.622859    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fa06311-1a87-4419-952d-71be0a1151e6-maas-api-service-ca\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:33:36.407857 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:36.407828    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93c38c4c-cllbh"
Sep 11 14:33:36.407857 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:36.407856    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93c38c4c-cllbh" event={"ID":"0fa06311-1a87-4419-952d-71be0a1151e6","Type":"ContainerDied","Data":"2149cdca118fcd454be1142645c986bfc9912d4a35cb04155680fe7633b04eda"}
Sep 11 14:33:36.408266 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:36.407897    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2149cdca118fcd454be1142645c986bfc9912d4a35cb04155680fe7633b04eda"
Sep 11 14:33:41.312840 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:41.312800    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-ca8d752a-kserve-54d7d48496-lvk89" podUID="26a3b068-3f7c-4b11-97df-1494cb551008" containerName="main" containerID="cri-o://6a09b3d5489f164564b94de34a6278a8b637c99e57698ea970fad8f273e4e799" gracePeriod=45
Sep 11 14:33:41.398624 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:41.398589    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-84d3a8-mh94g"]
Sep 11 14:33:41.403045 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:41.403018    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-84d3a8-mh94g"]
Sep 11 14:33:41.438961 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:41.438930    2581 generic.go:356] "Generic (PLEG): container finished" podID="26a3b068-3f7c-4b11-97df-1494cb551008" containerID="6a09b3d5489f164564b94de34a6278a8b637c99e57698ea970fad8f273e4e799" exitCode=0
Sep 11 14:33:41.439141 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:41.439004    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-ca8d752a-kserve-54d7d48496-lvk89" event={"ID":"26a3b068-3f7c-4b11-97df-1494cb551008","Type":"ContainerDied","Data":"6a09b3d5489f164564b94de34a6278a8b637c99e57698ea970fad8f273e4e799"}
Sep 11 14:33:41.501323 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:41.501289    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c2ebc05-4fde-4d34-bf50-160665efdf5c" path="/var/lib/kubelet/pods/0c2ebc05-4fde-4d34-bf50-160665efdf5c/volumes"
Sep 11 14:33:41.565990 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:41.565902    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-ca8d752a-kserve-54d7d48496-lvk89"
Sep 11 14:33:41.682270 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:41.682233    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26a3b068-3f7c-4b11-97df-1494cb551008-tmp-dir\") pod \"26a3b068-3f7c-4b11-97df-1494cb551008\" (UID: \"26a3b068-3f7c-4b11-97df-1494cb551008\") "
Sep 11 14:33:41.682270 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:41.682275    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26a3b068-3f7c-4b11-97df-1494cb551008-tls-certs\") pod \"26a3b068-3f7c-4b11-97df-1494cb551008\" (UID: \"26a3b068-3f7c-4b11-97df-1494cb551008\") "
Sep 11 14:33:41.682490 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:41.682342    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26a3b068-3f7c-4b11-97df-1494cb551008-dshm\") pod \"26a3b068-3f7c-4b11-97df-1494cb551008\" (UID: \"26a3b068-3f7c-4b11-97df-1494cb551008\") "
Sep 11 14:33:41.682546 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:41.682501    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26a3b068-3f7c-4b11-97df-1494cb551008-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "26a3b068-3f7c-4b11-97df-1494cb551008" (UID: "26a3b068-3f7c-4b11-97df-1494cb551008"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:33:41.682603 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:41.682555    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26a3b068-3f7c-4b11-97df-1494cb551008-home\") pod \"26a3b068-3f7c-4b11-97df-1494cb551008\" (UID: \"26a3b068-3f7c-4b11-97df-1494cb551008\") "
Sep 11 14:33:41.682653 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:41.682612    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26a3b068-3f7c-4b11-97df-1494cb551008-model-cache\") pod \"26a3b068-3f7c-4b11-97df-1494cb551008\" (UID: \"26a3b068-3f7c-4b11-97df-1494cb551008\") "
Sep 11 14:33:41.682708 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:41.682654    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vgw2b\" (UniqueName: \"kubernetes.io/projected/26a3b068-3f7c-4b11-97df-1494cb551008-kube-api-access-vgw2b\") pod \"26a3b068-3f7c-4b11-97df-1494cb551008\" (UID: \"26a3b068-3f7c-4b11-97df-1494cb551008\") "
Sep 11 14:33:41.682758 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:41.682732    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26a3b068-3f7c-4b11-97df-1494cb551008-home" (OuterVolumeSpecName: "home") pod "26a3b068-3f7c-4b11-97df-1494cb551008" (UID: "26a3b068-3f7c-4b11-97df-1494cb551008"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:33:41.682879 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:41.682856    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26a3b068-3f7c-4b11-97df-1494cb551008-model-cache" (OuterVolumeSpecName: "model-cache") pod "26a3b068-3f7c-4b11-97df-1494cb551008" (UID: "26a3b068-3f7c-4b11-97df-1494cb551008"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:33:41.683076 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:41.683059    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26a3b068-3f7c-4b11-97df-1494cb551008-home\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:33:41.683149 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:41.683080    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26a3b068-3f7c-4b11-97df-1494cb551008-model-cache\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:33:41.683149 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:41.683095    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26a3b068-3f7c-4b11-97df-1494cb551008-tmp-dir\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:33:41.684675 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:41.684653    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26a3b068-3f7c-4b11-97df-1494cb551008-kube-api-access-vgw2b" (OuterVolumeSpecName: "kube-api-access-vgw2b") pod "26a3b068-3f7c-4b11-97df-1494cb551008" (UID: "26a3b068-3f7c-4b11-97df-1494cb551008"). InnerVolumeSpecName "kube-api-access-vgw2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:33:41.684723 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:41.684674    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26a3b068-3f7c-4b11-97df-1494cb551008-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "26a3b068-3f7c-4b11-97df-1494cb551008" (UID: "26a3b068-3f7c-4b11-97df-1494cb551008"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:33:41.684759 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:41.684729    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26a3b068-3f7c-4b11-97df-1494cb551008-dshm" (OuterVolumeSpecName: "dshm") pod "26a3b068-3f7c-4b11-97df-1494cb551008" (UID: "26a3b068-3f7c-4b11-97df-1494cb551008"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:33:41.784537 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:41.784497    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26a3b068-3f7c-4b11-97df-1494cb551008-tls-certs\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:33:41.784537 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:41.784530    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26a3b068-3f7c-4b11-97df-1494cb551008-dshm\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:33:41.784537 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:41.784540    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgw2b\" (UniqueName: \"kubernetes.io/projected/26a3b068-3f7c-4b11-97df-1494cb551008-kube-api-access-vgw2b\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:33:42.445655 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:42.445619    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-ca8d752a-kserve-54d7d48496-lvk89" event={"ID":"26a3b068-3f7c-4b11-97df-1494cb551008","Type":"ContainerDied","Data":"d979cc518cb50dcc9522a2f4c6fb7cc71cab917f706d042d2df768a8fe2ff0b9"}
Sep 11 14:33:42.446086 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:42.445670    2581 scope.go:117] "RemoveContainer" containerID="6a09b3d5489f164564b94de34a6278a8b637c99e57698ea970fad8f273e4e799"
Sep 11 14:33:42.446086 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:42.445670    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-ca8d752a-kserve-54d7d48496-lvk89"
Sep 11 14:33:42.477496 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:42.477464    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-ca8d752a-kserve-54d7d48496-lvk89"]
Sep 11 14:33:42.490007 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:42.489979    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-ca8d752a-kserve-54d7d48496-lvk89"]
Sep 11 14:33:42.584292 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:42.584251    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-84d3a8/autores-model-84d3a8-kserve-6494496fc9-xvg8t" podUID="f16836a9-2f8c-43d6-834e-1f0c9dd93ec3" containerName="main" containerID="cri-o://d99964b89e07257312fac69beea29a13d1f371ed519bc0f351590c60d3d1df29" gracePeriod=45
Sep 11 14:33:42.834348 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:42.834325    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-84d3a8/autores-model-84d3a8-kserve-6494496fc9-xvg8t"
Sep 11 14:33:42.997556 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:42.997525    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjmb8\" (UniqueName: \"kubernetes.io/projected/f16836a9-2f8c-43d6-834e-1f0c9dd93ec3-kube-api-access-tjmb8\") pod \"f16836a9-2f8c-43d6-834e-1f0c9dd93ec3\" (UID: \"f16836a9-2f8c-43d6-834e-1f0c9dd93ec3\") "
Sep 11 14:33:42.997725 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:42.997590    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f16836a9-2f8c-43d6-834e-1f0c9dd93ec3-dshm\") pod \"f16836a9-2f8c-43d6-834e-1f0c9dd93ec3\" (UID: \"f16836a9-2f8c-43d6-834e-1f0c9dd93ec3\") "
Sep 11 14:33:42.997725 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:42.997633    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f16836a9-2f8c-43d6-834e-1f0c9dd93ec3-model-cache\") pod \"f16836a9-2f8c-43d6-834e-1f0c9dd93ec3\" (UID: \"f16836a9-2f8c-43d6-834e-1f0c9dd93ec3\") "
Sep 11 14:33:42.997725 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:42.997669    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f16836a9-2f8c-43d6-834e-1f0c9dd93ec3-tmp-dir\") pod \"f16836a9-2f8c-43d6-834e-1f0c9dd93ec3\" (UID: \"f16836a9-2f8c-43d6-834e-1f0c9dd93ec3\") "
Sep 11 14:33:42.997725 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:42.997713    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f16836a9-2f8c-43d6-834e-1f0c9dd93ec3-tls-certs\") pod \"f16836a9-2f8c-43d6-834e-1f0c9dd93ec3\" (UID: \"f16836a9-2f8c-43d6-834e-1f0c9dd93ec3\") "
Sep 11 14:33:42.997903 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:42.997750    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f16836a9-2f8c-43d6-834e-1f0c9dd93ec3-home\") pod \"f16836a9-2f8c-43d6-834e-1f0c9dd93ec3\" (UID: \"f16836a9-2f8c-43d6-834e-1f0c9dd93ec3\") "
Sep 11 14:33:42.997965 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:42.997907    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f16836a9-2f8c-43d6-834e-1f0c9dd93ec3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f16836a9-2f8c-43d6-834e-1f0c9dd93ec3" (UID: "f16836a9-2f8c-43d6-834e-1f0c9dd93ec3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:33:42.997965 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:42.997921    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f16836a9-2f8c-43d6-834e-1f0c9dd93ec3-model-cache" (OuterVolumeSpecName: "model-cache") pod "f16836a9-2f8c-43d6-834e-1f0c9dd93ec3" (UID: "f16836a9-2f8c-43d6-834e-1f0c9dd93ec3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:33:42.998166 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:42.998136    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f16836a9-2f8c-43d6-834e-1f0c9dd93ec3-home" (OuterVolumeSpecName: "home") pod "f16836a9-2f8c-43d6-834e-1f0c9dd93ec3" (UID: "f16836a9-2f8c-43d6-834e-1f0c9dd93ec3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:33:42.998268 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:42.998253    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f16836a9-2f8c-43d6-834e-1f0c9dd93ec3-model-cache\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:33:42.998268 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:42.998275    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f16836a9-2f8c-43d6-834e-1f0c9dd93ec3-tmp-dir\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:33:42.998369 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:42.998288    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f16836a9-2f8c-43d6-834e-1f0c9dd93ec3-home\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:33:42.999785 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:42.999761    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f16836a9-2f8c-43d6-834e-1f0c9dd93ec3-kube-api-access-tjmb8" (OuterVolumeSpecName: "kube-api-access-tjmb8") pod "f16836a9-2f8c-43d6-834e-1f0c9dd93ec3" (UID: "f16836a9-2f8c-43d6-834e-1f0c9dd93ec3"). InnerVolumeSpecName "kube-api-access-tjmb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:33:43.000170 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:43.000144    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f16836a9-2f8c-43d6-834e-1f0c9dd93ec3-dshm" (OuterVolumeSpecName: "dshm") pod "f16836a9-2f8c-43d6-834e-1f0c9dd93ec3" (UID: "f16836a9-2f8c-43d6-834e-1f0c9dd93ec3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:33:43.000170 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:43.000153    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f16836a9-2f8c-43d6-834e-1f0c9dd93ec3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f16836a9-2f8c-43d6-834e-1f0c9dd93ec3" (UID: "f16836a9-2f8c-43d6-834e-1f0c9dd93ec3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:33:43.099306 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:43.099271    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f16836a9-2f8c-43d6-834e-1f0c9dd93ec3-dshm\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:33:43.099306 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:43.099300    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f16836a9-2f8c-43d6-834e-1f0c9dd93ec3-tls-certs\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:33:43.099306 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:43.099312    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjmb8\" (UniqueName: \"kubernetes.io/projected/f16836a9-2f8c-43d6-834e-1f0c9dd93ec3-kube-api-access-tjmb8\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:33:43.452108 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:43.452006    2581 generic.go:356] "Generic (PLEG): container finished" podID="f16836a9-2f8c-43d6-834e-1f0c9dd93ec3" containerID="d99964b89e07257312fac69beea29a13d1f371ed519bc0f351590c60d3d1df29" exitCode=0
Sep 11 14:33:43.452108 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:43.452054    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-84d3a8/autores-model-84d3a8-kserve-6494496fc9-xvg8t"
Sep 11 14:33:43.452640 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:43.452095    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-84d3a8/autores-model-84d3a8-kserve-6494496fc9-xvg8t" event={"ID":"f16836a9-2f8c-43d6-834e-1f0c9dd93ec3","Type":"ContainerDied","Data":"d99964b89e07257312fac69beea29a13d1f371ed519bc0f351590c60d3d1df29"}
Sep 11 14:33:43.452640 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:43.452140    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-84d3a8/autores-model-84d3a8-kserve-6494496fc9-xvg8t" event={"ID":"f16836a9-2f8c-43d6-834e-1f0c9dd93ec3","Type":"ContainerDied","Data":"540fe7eafa5ae5f721b46cc77151ccd648137049eaa12e5cf54b6ad4963715e3"}
Sep 11 14:33:43.452640 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:43.452154    2581 scope.go:117] "RemoveContainer" containerID="d99964b89e07257312fac69beea29a13d1f371ed519bc0f351590c60d3d1df29"
Sep 11 14:33:43.463080 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:43.461749    2581 scope.go:117] "RemoveContainer" containerID="d99964b89e07257312fac69beea29a13d1f371ed519bc0f351590c60d3d1df29"
Sep 11 14:33:43.481648 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:33:43.481611    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d99964b89e07257312fac69beea29a13d1f371ed519bc0f351590c60d3d1df29\": container with ID starting with d99964b89e07257312fac69beea29a13d1f371ed519bc0f351590c60d3d1df29 not found: ID does not exist" containerID="d99964b89e07257312fac69beea29a13d1f371ed519bc0f351590c60d3d1df29"
Sep 11 14:33:43.481812 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:43.481656    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d99964b89e07257312fac69beea29a13d1f371ed519bc0f351590c60d3d1df29"} err="failed to get container status \"d99964b89e07257312fac69beea29a13d1f371ed519bc0f351590c60d3d1df29\": rpc error: code = NotFound desc = could not find container \"d99964b89e07257312fac69beea29a13d1f371ed519bc0f351590c60d3d1df29\": container with ID starting with d99964b89e07257312fac69beea29a13d1f371ed519bc0f351590c60d3d1df29 not found: ID does not exist"
Sep 11 14:33:43.499985 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:43.499942    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26a3b068-3f7c-4b11-97df-1494cb551008" path="/var/lib/kubelet/pods/26a3b068-3f7c-4b11-97df-1494cb551008/volumes"
Sep 11 14:33:43.515029 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:43.514995    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-84d3a8/autores-model-84d3a8-kserve-6494496fc9-xvg8t"]
Sep 11 14:33:43.520623 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:43.520601    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-84d3a8/autores-model-84d3a8-kserve-6494496fc9-xvg8t"]
Sep 11 14:33:44.388407 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:44.388356    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-ca8d752a-kserve-5f75475f98-5bmm9" podUID="eb0cb030-aed3-4445-b7b0-6ad3326effb6" containerName="main" containerID="cri-o://9b1afb9049690b44c60bb68576a335fa2d48d4320d2ec067adb66288813b3549" gracePeriod=45
Sep 11 14:33:44.644140 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:44.644045    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-ca8d752a-kserve-5f75475f98-5bmm9"
Sep 11 14:33:44.715844 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:44.715805    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb0cb030-aed3-4445-b7b0-6ad3326effb6-tls-certs\") pod \"eb0cb030-aed3-4445-b7b0-6ad3326effb6\" (UID: \"eb0cb030-aed3-4445-b7b0-6ad3326effb6\") "
Sep 11 14:33:44.716050 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:44.715875    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb0cb030-aed3-4445-b7b0-6ad3326effb6-model-cache\") pod \"eb0cb030-aed3-4445-b7b0-6ad3326effb6\" (UID: \"eb0cb030-aed3-4445-b7b0-6ad3326effb6\") "
Sep 11 14:33:44.716050 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:44.715998    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb0cb030-aed3-4445-b7b0-6ad3326effb6-tmp-dir\") pod \"eb0cb030-aed3-4445-b7b0-6ad3326effb6\" (UID: \"eb0cb030-aed3-4445-b7b0-6ad3326effb6\") "
Sep 11 14:33:44.716190 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:44.716107    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb0cb030-aed3-4445-b7b0-6ad3326effb6-dshm\") pod \"eb0cb030-aed3-4445-b7b0-6ad3326effb6\" (UID: \"eb0cb030-aed3-4445-b7b0-6ad3326effb6\") "
Sep 11 14:33:44.716190 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:44.716144    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb0cb030-aed3-4445-b7b0-6ad3326effb6-home\") pod \"eb0cb030-aed3-4445-b7b0-6ad3326effb6\" (UID: \"eb0cb030-aed3-4445-b7b0-6ad3326effb6\") "
Sep 11 14:33:44.716190 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:44.716174    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb0cb030-aed3-4445-b7b0-6ad3326effb6-model-cache" (OuterVolumeSpecName: "model-cache") pod "eb0cb030-aed3-4445-b7b0-6ad3326effb6" (UID: "eb0cb030-aed3-4445-b7b0-6ad3326effb6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:33:44.716334 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:44.716189    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb0cb030-aed3-4445-b7b0-6ad3326effb6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "eb0cb030-aed3-4445-b7b0-6ad3326effb6" (UID: "eb0cb030-aed3-4445-b7b0-6ad3326effb6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:33:44.716334 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:44.716181    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2jztg\" (UniqueName: \"kubernetes.io/projected/eb0cb030-aed3-4445-b7b0-6ad3326effb6-kube-api-access-2jztg\") pod \"eb0cb030-aed3-4445-b7b0-6ad3326effb6\" (UID: \"eb0cb030-aed3-4445-b7b0-6ad3326effb6\") "
Sep 11 14:33:44.716415 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:44.716387    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb0cb030-aed3-4445-b7b0-6ad3326effb6-home" (OuterVolumeSpecName: "home") pod "eb0cb030-aed3-4445-b7b0-6ad3326effb6" (UID: "eb0cb030-aed3-4445-b7b0-6ad3326effb6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:33:44.716651 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:44.716625    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb0cb030-aed3-4445-b7b0-6ad3326effb6-home\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:33:44.716772 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:44.716651    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb0cb030-aed3-4445-b7b0-6ad3326effb6-model-cache\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:33:44.716772 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:44.716666    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb0cb030-aed3-4445-b7b0-6ad3326effb6-tmp-dir\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:33:44.718186 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:44.718159    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb0cb030-aed3-4445-b7b0-6ad3326effb6-dshm" (OuterVolumeSpecName: "dshm") pod "eb0cb030-aed3-4445-b7b0-6ad3326effb6" (UID: "eb0cb030-aed3-4445-b7b0-6ad3326effb6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:33:44.718316 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:44.718189    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb0cb030-aed3-4445-b7b0-6ad3326effb6-kube-api-access-2jztg" (OuterVolumeSpecName: "kube-api-access-2jztg") pod "eb0cb030-aed3-4445-b7b0-6ad3326effb6" (UID: "eb0cb030-aed3-4445-b7b0-6ad3326effb6"). InnerVolumeSpecName "kube-api-access-2jztg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:33:44.718316 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:44.718244    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb0cb030-aed3-4445-b7b0-6ad3326effb6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eb0cb030-aed3-4445-b7b0-6ad3326effb6" (UID: "eb0cb030-aed3-4445-b7b0-6ad3326effb6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:33:44.818155 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:44.818123    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb0cb030-aed3-4445-b7b0-6ad3326effb6-dshm\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:33:44.818155 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:44.818153    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jztg\" (UniqueName: \"kubernetes.io/projected/eb0cb030-aed3-4445-b7b0-6ad3326effb6-kube-api-access-2jztg\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:33:44.818155 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:44.818163    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb0cb030-aed3-4445-b7b0-6ad3326effb6-tls-certs\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:33:45.468422 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:45.468390    2581 generic.go:356] "Generic (PLEG): container finished" podID="eb0cb030-aed3-4445-b7b0-6ad3326effb6" containerID="9b1afb9049690b44c60bb68576a335fa2d48d4320d2ec067adb66288813b3549" exitCode=0
Sep 11 14:33:45.468605 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:45.468435    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-ca8d752a-kserve-5f75475f98-5bmm9"
Sep 11 14:33:45.468605 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:45.468469    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-ca8d752a-kserve-5f75475f98-5bmm9" event={"ID":"eb0cb030-aed3-4445-b7b0-6ad3326effb6","Type":"ContainerDied","Data":"9b1afb9049690b44c60bb68576a335fa2d48d4320d2ec067adb66288813b3549"}
Sep 11 14:33:45.468605 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:45.468514    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-ca8d752a-kserve-5f75475f98-5bmm9" event={"ID":"eb0cb030-aed3-4445-b7b0-6ad3326effb6","Type":"ContainerDied","Data":"7662c605159fa174b560fa46ecf61ab506bed19c1e97744650a34fc035e04e73"}
Sep 11 14:33:45.468605 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:45.468538    2581 scope.go:117] "RemoveContainer" containerID="9b1afb9049690b44c60bb68576a335fa2d48d4320d2ec067adb66288813b3549"
Sep 11 14:33:45.478291 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:45.478267    2581 scope.go:117] "RemoveContainer" containerID="9b1afb9049690b44c60bb68576a335fa2d48d4320d2ec067adb66288813b3549"
Sep 11 14:33:45.478635 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:33:45.478616    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b1afb9049690b44c60bb68576a335fa2d48d4320d2ec067adb66288813b3549\": container with ID starting with 9b1afb9049690b44c60bb68576a335fa2d48d4320d2ec067adb66288813b3549 not found: ID does not exist" containerID="9b1afb9049690b44c60bb68576a335fa2d48d4320d2ec067adb66288813b3549"
Sep 11 14:33:45.478694 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:45.478644    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b1afb9049690b44c60bb68576a335fa2d48d4320d2ec067adb66288813b3549"} err="failed to get container status \"9b1afb9049690b44c60bb68576a335fa2d48d4320d2ec067adb66288813b3549\": rpc error: code = NotFound desc = could not find container \"9b1afb9049690b44c60bb68576a335fa2d48d4320d2ec067adb66288813b3549\": container with ID starting with 9b1afb9049690b44c60bb68576a335fa2d48d4320d2ec067adb66288813b3549 not found: ID does not exist"
Sep 11 14:33:45.502779 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:45.502735    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f16836a9-2f8c-43d6-834e-1f0c9dd93ec3" path="/var/lib/kubelet/pods/f16836a9-2f8c-43d6-834e-1f0c9dd93ec3/volumes"
Sep 11 14:33:45.532902 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:45.532870    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-ca8d752a-kserve-5f75475f98-5bmm9"]
Sep 11 14:33:45.542814 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:45.542783    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-ca8d752a-kserve-5f75475f98-5bmm9"]
Sep 11 14:33:47.501941 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:47.501895    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb0cb030-aed3-4445-b7b0-6ad3326effb6" path="/var/lib/kubelet/pods/eb0cb030-aed3-4445-b7b0-6ad3326effb6/volumes"
Sep 11 14:33:47.921864 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:47.921773    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93c38c4c-cllbh"]
Sep 11 14:33:47.935654 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:47.935621    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93c38c4c-cllbh"]
Sep 11 14:33:49.499542 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:49.499505    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fa06311-1a87-4419-952d-71be0a1151e6" path="/var/lib/kubelet/pods/0fa06311-1a87-4419-952d-71be0a1151e6/volumes"
Sep 11 14:33:51.230569 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:51.230533    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-5a7035/test-model-5a7035-kserve-5f44ddb7-9s2b6"]
Sep 11 14:33:51.230946 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:51.230882    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26a3b068-3f7c-4b11-97df-1494cb551008" containerName="main"
Sep 11 14:33:51.230946 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:51.230893    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="26a3b068-3f7c-4b11-97df-1494cb551008" containerName="main"
Sep 11 14:33:51.230946 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:51.230911    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb0cb030-aed3-4445-b7b0-6ad3326effb6" containerName="main"
Sep 11 14:33:51.230946 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:51.230917    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb0cb030-aed3-4445-b7b0-6ad3326effb6" containerName="main"
Sep 11 14:33:51.230946 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:51.230927    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fa06311-1a87-4419-952d-71be0a1151e6" containerName="revoke-keys"
Sep 11 14:33:51.230946 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:51.230933    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fa06311-1a87-4419-952d-71be0a1151e6" containerName="revoke-keys"
Sep 11 14:33:51.230946 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:51.230943    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f16836a9-2f8c-43d6-834e-1f0c9dd93ec3" containerName="main"
Sep 11 14:33:51.230946 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:51.230948    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f16836a9-2f8c-43d6-834e-1f0c9dd93ec3" containerName="main"
Sep 11 14:33:51.231207 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:51.231012    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f16836a9-2f8c-43d6-834e-1f0c9dd93ec3" containerName="main"
Sep 11 14:33:51.231207 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:51.231018    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fa06311-1a87-4419-952d-71be0a1151e6" containerName="revoke-keys"
Sep 11 14:33:51.231207 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:51.231025    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb0cb030-aed3-4445-b7b0-6ad3326effb6" containerName="main"
Sep 11 14:33:51.231207 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:51.231033    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="26a3b068-3f7c-4b11-97df-1494cb551008" containerName="main"
Sep 11 14:33:51.235355 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:51.235336    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5a7035/test-model-5a7035-kserve-5f44ddb7-9s2b6"
Sep 11 14:33:51.239510 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:51.239489    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5a7035\"/\"default-dockercfg-4xvql\""
Sep 11 14:33:51.239635 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:51.239536    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5a7035\"/\"test-model-5a7035-kserve-self-signed-certs\""
Sep 11 14:33:51.239635 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:51.239489    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5a7035\"/\"openshift-service-ca.crt\""
Sep 11 14:33:51.239635 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:51.239541    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5a7035\"/\"kube-root-ca.crt\""
Sep 11 14:33:51.267094 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:51.267062    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5a7035/test-model-5a7035-kserve-5f44ddb7-9s2b6"]
Sep 11 14:33:51.277905 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:51.277876    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44f5281c-8d94-4459-b151-cdfe7565ae31-tls-certs\") pod \"test-model-5a7035-kserve-5f44ddb7-9s2b6\" (UID: \"44f5281c-8d94-4459-b151-cdfe7565ae31\") " pod="ai-tenant-e2e-inf-b-5a7035/test-model-5a7035-kserve-5f44ddb7-9s2b6"
Sep 11 14:33:51.278063 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:51.277930    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd9p2\" (UniqueName: \"kubernetes.io/projected/44f5281c-8d94-4459-b151-cdfe7565ae31-kube-api-access-vd9p2\") pod \"test-model-5a7035-kserve-5f44ddb7-9s2b6\" (UID: \"44f5281c-8d94-4459-b151-cdfe7565ae31\") " pod="ai-tenant-e2e-inf-b-5a7035/test-model-5a7035-kserve-5f44ddb7-9s2b6"
Sep 11 14:33:51.278144 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:51.278054    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44f5281c-8d94-4459-b151-cdfe7565ae31-tmp-dir\") pod \"test-model-5a7035-kserve-5f44ddb7-9s2b6\" (UID: \"44f5281c-8d94-4459-b151-cdfe7565ae31\") " pod="ai-tenant-e2e-inf-b-5a7035/test-model-5a7035-kserve-5f44ddb7-9s2b6"
Sep 11 14:33:51.278144 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:51.278119    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44f5281c-8d94-4459-b151-cdfe7565ae31-model-cache\") pod \"test-model-5a7035-kserve-5f44ddb7-9s2b6\" (UID: \"44f5281c-8d94-4459-b151-cdfe7565ae31\") " pod="ai-tenant-e2e-inf-b-5a7035/test-model-5a7035-kserve-5f44ddb7-9s2b6"
Sep 11 14:33:51.278258 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:51.278203    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44f5281c-8d94-4459-b151-cdfe7565ae31-dshm\") pod \"test-model-5a7035-kserve-5f44ddb7-9s2b6\" (UID: \"44f5281c-8d94-4459-b151-cdfe7565ae31\") " pod="ai-tenant-e2e-inf-b-5a7035/test-model-5a7035-kserve-5f44ddb7-9s2b6"
Sep 11 14:33:51.278325 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:51.278304    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44f5281c-8d94-4459-b151-cdfe7565ae31-home\") pod \"test-model-5a7035-kserve-5f44ddb7-9s2b6\" (UID: \"44f5281c-8d94-4459-b151-cdfe7565ae31\") " pod="ai-tenant-e2e-inf-b-5a7035/test-model-5a7035-kserve-5f44ddb7-9s2b6"
Sep 11 14:33:51.379293 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:51.379257    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44f5281c-8d94-4459-b151-cdfe7565ae31-home\") pod \"test-model-5a7035-kserve-5f44ddb7-9s2b6\" (UID: \"44f5281c-8d94-4459-b151-cdfe7565ae31\") " pod="ai-tenant-e2e-inf-b-5a7035/test-model-5a7035-kserve-5f44ddb7-9s2b6"
Sep 11 14:33:51.379487 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:51.379300    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44f5281c-8d94-4459-b151-cdfe7565ae31-tls-certs\") pod \"test-model-5a7035-kserve-5f44ddb7-9s2b6\" (UID: \"44f5281c-8d94-4459-b151-cdfe7565ae31\") " pod="ai-tenant-e2e-inf-b-5a7035/test-model-5a7035-kserve-5f44ddb7-9s2b6"
Sep 11 14:33:51.379487 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:51.379333    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd9p2\" (UniqueName: \"kubernetes.io/projected/44f5281c-8d94-4459-b151-cdfe7565ae31-kube-api-access-vd9p2\") pod \"test-model-5a7035-kserve-5f44ddb7-9s2b6\" (UID: \"44f5281c-8d94-4459-b151-cdfe7565ae31\") " pod="ai-tenant-e2e-inf-b-5a7035/test-model-5a7035-kserve-5f44ddb7-9s2b6"
Sep 11 14:33:51.379487 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:51.379358    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44f5281c-8d94-4459-b151-cdfe7565ae31-tmp-dir\") pod \"test-model-5a7035-kserve-5f44ddb7-9s2b6\" (UID: \"44f5281c-8d94-4459-b151-cdfe7565ae31\") " pod="ai-tenant-e2e-inf-b-5a7035/test-model-5a7035-kserve-5f44ddb7-9s2b6"
Sep 11 14:33:51.379487 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:51.379393    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44f5281c-8d94-4459-b151-cdfe7565ae31-model-cache\") pod \"test-model-5a7035-kserve-5f44ddb7-9s2b6\" (UID: \"44f5281c-8d94-4459-b151-cdfe7565ae31\") " pod="ai-tenant-e2e-inf-b-5a7035/test-model-5a7035-kserve-5f44ddb7-9s2b6"
Sep 11 14:33:51.379487 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:51.379444    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44f5281c-8d94-4459-b151-cdfe7565ae31-dshm\") pod \"test-model-5a7035-kserve-5f44ddb7-9s2b6\" (UID: \"44f5281c-8d94-4459-b151-cdfe7565ae31\") " pod="ai-tenant-e2e-inf-b-5a7035/test-model-5a7035-kserve-5f44ddb7-9s2b6"
Sep 11 14:33:51.379742 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:51.379714    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44f5281c-8d94-4459-b151-cdfe7565ae31-home\") pod \"test-model-5a7035-kserve-5f44ddb7-9s2b6\" (UID: \"44f5281c-8d94-4459-b151-cdfe7565ae31\") " pod="ai-tenant-e2e-inf-b-5a7035/test-model-5a7035-kserve-5f44ddb7-9s2b6"
Sep 11 14:33:51.379802 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:51.379737    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44f5281c-8d94-4459-b151-cdfe7565ae31-tmp-dir\") pod \"test-model-5a7035-kserve-5f44ddb7-9s2b6\" (UID: \"44f5281c-8d94-4459-b151-cdfe7565ae31\") " pod="ai-tenant-e2e-inf-b-5a7035/test-model-5a7035-kserve-5f44ddb7-9s2b6"
Sep 11 14:33:51.379802 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:51.379792    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44f5281c-8d94-4459-b151-cdfe7565ae31-model-cache\") pod \"test-model-5a7035-kserve-5f44ddb7-9s2b6\" (UID: \"44f5281c-8d94-4459-b151-cdfe7565ae31\") " pod="ai-tenant-e2e-inf-b-5a7035/test-model-5a7035-kserve-5f44ddb7-9s2b6"
Sep 11 14:33:51.381595 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:51.381573    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44f5281c-8d94-4459-b151-cdfe7565ae31-dshm\") pod \"test-model-5a7035-kserve-5f44ddb7-9s2b6\" (UID: \"44f5281c-8d94-4459-b151-cdfe7565ae31\") " pod="ai-tenant-e2e-inf-b-5a7035/test-model-5a7035-kserve-5f44ddb7-9s2b6"
Sep 11 14:33:51.381859 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:51.381843    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44f5281c-8d94-4459-b151-cdfe7565ae31-tls-certs\") pod \"test-model-5a7035-kserve-5f44ddb7-9s2b6\" (UID: \"44f5281c-8d94-4459-b151-cdfe7565ae31\") " pod="ai-tenant-e2e-inf-b-5a7035/test-model-5a7035-kserve-5f44ddb7-9s2b6"
Sep 11 14:33:51.392928 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:51.392905    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd9p2\" (UniqueName: \"kubernetes.io/projected/44f5281c-8d94-4459-b151-cdfe7565ae31-kube-api-access-vd9p2\") pod \"test-model-5a7035-kserve-5f44ddb7-9s2b6\" (UID: \"44f5281c-8d94-4459-b151-cdfe7565ae31\") " pod="ai-tenant-e2e-inf-b-5a7035/test-model-5a7035-kserve-5f44ddb7-9s2b6"
Sep 11 14:33:51.544523 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:51.544491    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5a7035/test-model-5a7035-kserve-5f44ddb7-9s2b6"
Sep 11 14:33:51.826415 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:51.826255    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5a7035/test-model-5a7035-kserve-5f44ddb7-9s2b6"]
Sep 11 14:33:51.829193 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:33:51.829163    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44f5281c_8d94_4459_b151_cdfe7565ae31.slice/crio-fc2f91f4475e26a732d35c2815e8eaaca6e0a429b42c7326ba17b7368e650d99 WatchSource:0}: Error finding container fc2f91f4475e26a732d35c2815e8eaaca6e0a429b42c7326ba17b7368e650d99: Status 404 returned error can't find the container with id fc2f91f4475e26a732d35c2815e8eaaca6e0a429b42c7326ba17b7368e650d99
Sep 11 14:33:52.506654 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:52.506620    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5a7035/test-model-5a7035-kserve-5f44ddb7-9s2b6" event={"ID":"44f5281c-8d94-4459-b151-cdfe7565ae31","Type":"ContainerStarted","Data":"8fa2f2be23b05bfc7827185fa4b12850693e938846a5133f484c71ad06c90d17"}
Sep 11 14:33:52.507119 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:52.506665    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5a7035/test-model-5a7035-kserve-5f44ddb7-9s2b6" event={"ID":"44f5281c-8d94-4459-b151-cdfe7565ae31","Type":"ContainerStarted","Data":"fc2f91f4475e26a732d35c2815e8eaaca6e0a429b42c7326ba17b7368e650d99"}
Sep 11 14:33:52.527061 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:33:52.527008    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-5a7035/test-model-5a7035-kserve-5f44ddb7-9s2b6" podStartSLOduration=1.526962025 podStartE2EDuration="1.526962025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:33:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:33:52.526752483 +0000 UTC m=+1135.677302548" watchObservedRunningTime="2026-09-11 14:33:52.526962025 +0000 UTC m=+1135.677512088"
Sep 11 14:34:01.545485 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:34:01.545440    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-5a7035/test-model-5a7035-kserve-5f44ddb7-9s2b6"
Sep 11 14:34:01.545934 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:34:01.545504    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-5a7035/test-model-5a7035-kserve-5f44ddb7-9s2b6"
Sep 11 14:34:01.558849 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:34:01.558821    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-5a7035/test-model-5a7035-kserve-5f44ddb7-9s2b6"
Sep 11 14:34:02.567506 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:34:02.567476    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-5a7035/test-model-5a7035-kserve-5f44ddb7-9s2b6"
Sep 11 14:34:48.430441 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:34:48.430405    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-259c79/explicit-model-259c79-kserve-5dcdc74c6c-dwd4r"]
Sep 11 14:34:48.433755 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:34:48.433736    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-259c79/explicit-model-259c79-kserve-5dcdc74c6c-dwd4r"
Sep 11 14:34:48.437824 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:34:48.437798    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-259c79\"/\"kube-root-ca.crt\""
Sep 11 14:34:48.437959 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:34:48.437805    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-259c79\"/\"default-dockercfg-525hg\""
Sep 11 14:34:48.437959 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:34:48.437808    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-259c79\"/\"openshift-service-ca.crt\""
Sep 11 14:34:48.437959 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:34:48.437865    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-259c79\"/\"explicit-model-259c79-kserve-self-signed-certs\""
Sep 11 14:34:48.447532 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:34:48.447509    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33064ede-db68-4b7c-9d56-14e670c42a2f-tls-certs\") pod \"explicit-model-259c79-kserve-5dcdc74c6c-dwd4r\" (UID: \"33064ede-db68-4b7c-9d56-14e670c42a2f\") " pod="ai-tenant-e2e-explicit-259c79/explicit-model-259c79-kserve-5dcdc74c6c-dwd4r"
Sep 11 14:34:48.447636 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:34:48.447539    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33064ede-db68-4b7c-9d56-14e670c42a2f-tmp-dir\") pod \"explicit-model-259c79-kserve-5dcdc74c6c-dwd4r\" (UID: \"33064ede-db68-4b7c-9d56-14e670c42a2f\") " pod="ai-tenant-e2e-explicit-259c79/explicit-model-259c79-kserve-5dcdc74c6c-dwd4r"
Sep 11 14:34:48.447636 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:34:48.447564    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33064ede-db68-4b7c-9d56-14e670c42a2f-home\") pod \"explicit-model-259c79-kserve-5dcdc74c6c-dwd4r\" (UID: \"33064ede-db68-4b7c-9d56-14e670c42a2f\") " pod="ai-tenant-e2e-explicit-259c79/explicit-model-259c79-kserve-5dcdc74c6c-dwd4r"
Sep 11 14:34:48.447636 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:34:48.447587    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcpbm\" (UniqueName: \"kubernetes.io/projected/33064ede-db68-4b7c-9d56-14e670c42a2f-kube-api-access-wcpbm\") pod \"explicit-model-259c79-kserve-5dcdc74c6c-dwd4r\" (UID: \"33064ede-db68-4b7c-9d56-14e670c42a2f\") " pod="ai-tenant-e2e-explicit-259c79/explicit-model-259c79-kserve-5dcdc74c6c-dwd4r"
Sep 11 14:34:48.447750 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:34:48.447637    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33064ede-db68-4b7c-9d56-14e670c42a2f-model-cache\") pod \"explicit-model-259c79-kserve-5dcdc74c6c-dwd4r\" (UID: \"33064ede-db68-4b7c-9d56-14e670c42a2f\") " pod="ai-tenant-e2e-explicit-259c79/explicit-model-259c79-kserve-5dcdc74c6c-dwd4r"
Sep 11 14:34:48.447750 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:34:48.447654    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33064ede-db68-4b7c-9d56-14e670c42a2f-dshm\") pod \"explicit-model-259c79-kserve-5dcdc74c6c-dwd4r\" (UID: \"33064ede-db68-4b7c-9d56-14e670c42a2f\") " pod="ai-tenant-e2e-explicit-259c79/explicit-model-259c79-kserve-5dcdc74c6c-dwd4r"
Sep 11 14:34:48.458282 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:34:48.458258    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-259c79/explicit-model-259c79-kserve-5dcdc74c6c-dwd4r"]
Sep 11 14:34:48.548833 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:34:48.548795    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33064ede-db68-4b7c-9d56-14e670c42a2f-home\") pod \"explicit-model-259c79-kserve-5dcdc74c6c-dwd4r\" (UID: \"33064ede-db68-4b7c-9d56-14e670c42a2f\") " pod="ai-tenant-e2e-explicit-259c79/explicit-model-259c79-kserve-5dcdc74c6c-dwd4r"
Sep 11 14:34:48.549053 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:34:48.548843    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcpbm\" (UniqueName: \"kubernetes.io/projected/33064ede-db68-4b7c-9d56-14e670c42a2f-kube-api-access-wcpbm\") pod \"explicit-model-259c79-kserve-5dcdc74c6c-dwd4r\" (UID: \"33064ede-db68-4b7c-9d56-14e670c42a2f\") " pod="ai-tenant-e2e-explicit-259c79/explicit-model-259c79-kserve-5dcdc74c6c-dwd4r"
Sep 11 14:34:48.549053 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:34:48.548900    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33064ede-db68-4b7c-9d56-14e670c42a2f-model-cache\") pod \"explicit-model-259c79-kserve-5dcdc74c6c-dwd4r\" (UID: \"33064ede-db68-4b7c-9d56-14e670c42a2f\") " pod="ai-tenant-e2e-explicit-259c79/explicit-model-259c79-kserve-5dcdc74c6c-dwd4r"
Sep 11 14:34:48.549053 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:34:48.548919    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33064ede-db68-4b7c-9d56-14e670c42a2f-dshm\") pod \"explicit-model-259c79-kserve-5dcdc74c6c-dwd4r\" (UID: \"33064ede-db68-4b7c-9d56-14e670c42a2f\") " pod="ai-tenant-e2e-explicit-259c79/explicit-model-259c79-kserve-5dcdc74c6c-dwd4r"
Sep 11 14:34:48.549053 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:34:48.549013    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33064ede-db68-4b7c-9d56-14e670c42a2f-tls-certs\") pod \"explicit-model-259c79-kserve-5dcdc74c6c-dwd4r\" (UID: \"33064ede-db68-4b7c-9d56-14e670c42a2f\") " pod="ai-tenant-e2e-explicit-259c79/explicit-model-259c79-kserve-5dcdc74c6c-dwd4r"
Sep 11 14:34:48.549053 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:34:48.549044    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33064ede-db68-4b7c-9d56-14e670c42a2f-tmp-dir\") pod \"explicit-model-259c79-kserve-5dcdc74c6c-dwd4r\" (UID: \"33064ede-db68-4b7c-9d56-14e670c42a2f\") " pod="ai-tenant-e2e-explicit-259c79/explicit-model-259c79-kserve-5dcdc74c6c-dwd4r"
Sep 11 14:34:48.549329 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:34:48.549273    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33064ede-db68-4b7c-9d56-14e670c42a2f-home\") pod \"explicit-model-259c79-kserve-5dcdc74c6c-dwd4r\" (UID: \"33064ede-db68-4b7c-9d56-14e670c42a2f\") " pod="ai-tenant-e2e-explicit-259c79/explicit-model-259c79-kserve-5dcdc74c6c-dwd4r"
Sep 11 14:34:48.549389 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:34:48.549353    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33064ede-db68-4b7c-9d56-14e670c42a2f-model-cache\") pod \"explicit-model-259c79-kserve-5dcdc74c6c-dwd4r\" (UID: \"33064ede-db68-4b7c-9d56-14e670c42a2f\") " pod="ai-tenant-e2e-explicit-259c79/explicit-model-259c79-kserve-5dcdc74c6c-dwd4r"
Sep 11 14:34:48.549389 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:34:48.549373    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33064ede-db68-4b7c-9d56-14e670c42a2f-tmp-dir\") pod \"explicit-model-259c79-kserve-5dcdc74c6c-dwd4r\" (UID: \"33064ede-db68-4b7c-9d56-14e670c42a2f\") " pod="ai-tenant-e2e-explicit-259c79/explicit-model-259c79-kserve-5dcdc74c6c-dwd4r"
Sep 11 14:34:48.551364 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:34:48.551331    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33064ede-db68-4b7c-9d56-14e670c42a2f-dshm\") pod \"explicit-model-259c79-kserve-5dcdc74c6c-dwd4r\" (UID: \"33064ede-db68-4b7c-9d56-14e670c42a2f\") " pod="ai-tenant-e2e-explicit-259c79/explicit-model-259c79-kserve-5dcdc74c6c-dwd4r"
Sep 11 14:34:48.551569 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:34:48.551554    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33064ede-db68-4b7c-9d56-14e670c42a2f-tls-certs\") pod \"explicit-model-259c79-kserve-5dcdc74c6c-dwd4r\" (UID: \"33064ede-db68-4b7c-9d56-14e670c42a2f\") " pod="ai-tenant-e2e-explicit-259c79/explicit-model-259c79-kserve-5dcdc74c6c-dwd4r"
Sep 11 14:34:48.560380 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:34:48.560360    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcpbm\" (UniqueName: \"kubernetes.io/projected/33064ede-db68-4b7c-9d56-14e670c42a2f-kube-api-access-wcpbm\") pod \"explicit-model-259c79-kserve-5dcdc74c6c-dwd4r\" (UID: \"33064ede-db68-4b7c-9d56-14e670c42a2f\") " pod="ai-tenant-e2e-explicit-259c79/explicit-model-259c79-kserve-5dcdc74c6c-dwd4r"
Sep 11 14:34:48.743458 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:34:48.743423    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-259c79/explicit-model-259c79-kserve-5dcdc74c6c-dwd4r"
Sep 11 14:34:49.030922 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:34:49.030754    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-259c79/explicit-model-259c79-kserve-5dcdc74c6c-dwd4r"]
Sep 11 14:34:49.033221 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:34:49.033187    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod33064ede_db68_4b7c_9d56_14e670c42a2f.slice/crio-b1e81d7e2bddc0484c17637ca503d3ec41aa0d39ee2fad4971bf9be7f46495cc WatchSource:0}: Error finding container b1e81d7e2bddc0484c17637ca503d3ec41aa0d39ee2fad4971bf9be7f46495cc: Status 404 returned error can't find the container with id b1e81d7e2bddc0484c17637ca503d3ec41aa0d39ee2fad4971bf9be7f46495cc
Sep 11 14:34:49.801621 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:34:49.801585    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-259c79/explicit-model-259c79-kserve-5dcdc74c6c-dwd4r" event={"ID":"33064ede-db68-4b7c-9d56-14e670c42a2f","Type":"ContainerStarted","Data":"45d686a9f5fc4a56f57e9107794099b0f389272ef9c83f98e5cfa9a9362099e4"}
Sep 11 14:34:49.802290 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:34:49.801630    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-259c79/explicit-model-259c79-kserve-5dcdc74c6c-dwd4r" event={"ID":"33064ede-db68-4b7c-9d56-14e670c42a2f","Type":"ContainerStarted","Data":"b1e81d7e2bddc0484c17637ca503d3ec41aa0d39ee2fad4971bf9be7f46495cc"}
Sep 11 14:34:49.831551 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:34:49.831492    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-259c79/explicit-model-259c79-kserve-5dcdc74c6c-dwd4r" podStartSLOduration=1.831473744 podStartE2EDuration="1.831473744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:34:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:34:49.830561733 +0000 UTC m=+1192.981111810" watchObservedRunningTime="2026-09-11 14:34:49.831473744 +0000 UTC m=+1192.982023807"
Sep 11 14:34:58.744192 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:34:58.744152    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-259c79/explicit-model-259c79-kserve-5dcdc74c6c-dwd4r"
Sep 11 14:34:58.744601 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:34:58.744210    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-259c79/explicit-model-259c79-kserve-5dcdc74c6c-dwd4r"
Sep 11 14:34:58.756827 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:34:58.756803    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-259c79/explicit-model-259c79-kserve-5dcdc74c6c-dwd4r"
Sep 11 14:34:58.858109 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:34:58.858078    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-259c79/explicit-model-259c79-kserve-5dcdc74c6c-dwd4r"
Sep 11 14:34:59.211254 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:34:59.211178    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-259c79/explicit-model-259c79-kserve-5dcdc74c6c-dwd4r"]
Sep 11 14:34:59.349506 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:34:59.349474    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-259c79-ssrj9"]
Sep 11 14:34:59.354768 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:34:59.354745    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-259c79-ssrj9"
Sep 11 14:34:59.357673 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:34:59.357657    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 14:34:59.357768 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:34:59.357665    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hcm4g\""
Sep 11 14:34:59.369702 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:34:59.367178    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-259c79-ssrj9"]
Sep 11 14:34:59.446846 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:34:59.446810    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f24e5138-11cb-43cf-a0b4-250fa8f99e92-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-259c79-ssrj9\" (UID: \"f24e5138-11cb-43cf-a0b4-250fa8f99e92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-259c79-ssrj9"
Sep 11 14:34:59.547542 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:34:59.547507    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f24e5138-11cb-43cf-a0b4-250fa8f99e92-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-259c79-ssrj9\" (UID: \"f24e5138-11cb-43cf-a0b4-250fa8f99e92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-259c79-ssrj9"
Sep 11 14:34:59.548216 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:34:59.548192    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f24e5138-11cb-43cf-a0b4-250fa8f99e92-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-259c79-ssrj9\" (UID: \"f24e5138-11cb-43cf-a0b4-250fa8f99e92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-259c79-ssrj9"
Sep 11 14:34:59.671353 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:34:59.671322    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-259c79-ssrj9"
Sep 11 14:34:59.952465 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:34:59.952364    2581 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-259c79/explicit-model-259c79-kserve-self-signed-certs: secret "explicit-model-259c79-kserve-self-signed-certs" not found
Sep 11 14:34:59.952465 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:34:59.952462    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33064ede-db68-4b7c-9d56-14e670c42a2f-tls-certs podName:33064ede-db68-4b7c-9d56-14e670c42a2f nodeName:}" failed. No retries permitted until 2026-09-11 14:35:00.452440769 +0000 UTC m=+1203.602990816 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/33064ede-db68-4b7c-9d56-14e670c42a2f-tls-certs") pod "explicit-model-259c79-kserve-5dcdc74c6c-dwd4r" (UID: "33064ede-db68-4b7c-9d56-14e670c42a2f") : secret "explicit-model-259c79-kserve-self-signed-certs" not found
Sep 11 14:34:59.954190 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:34:59.954168    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-259c79-ssrj9"]
Sep 11 14:34:59.956727 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:34:59.956687    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf24e5138_11cb_43cf_a0b4_250fa8f99e92.slice/crio-bbcda89f7c9bd80a9846d7f0f8a9bf0585370936f057f3c7f7d7e08a3323d1fc WatchSource:0}: Error finding container bbcda89f7c9bd80a9846d7f0f8a9bf0585370936f057f3c7f7d7e08a3323d1fc: Status 404 returned error can't find the container with id bbcda89f7c9bd80a9846d7f0f8a9bf0585370936f057f3c7f7d7e08a3323d1fc
Sep 11 14:35:00.458356 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:35:00.458316    2581 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-259c79/explicit-model-259c79-kserve-self-signed-certs: secret "explicit-model-259c79-kserve-self-signed-certs" not found
Sep 11 14:35:00.458541 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:35:00.458388    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33064ede-db68-4b7c-9d56-14e670c42a2f-tls-certs podName:33064ede-db68-4b7c-9d56-14e670c42a2f nodeName:}" failed. No retries permitted until 2026-09-11 14:35:01.458373605 +0000 UTC m=+1204.608923648 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/33064ede-db68-4b7c-9d56-14e670c42a2f-tls-certs") pod "explicit-model-259c79-kserve-5dcdc74c6c-dwd4r" (UID: "33064ede-db68-4b7c-9d56-14e670c42a2f") : secret "explicit-model-259c79-kserve-self-signed-certs" not found
Sep 11 14:35:00.857664 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:00.857627    2581 generic.go:356] "Generic (PLEG): container finished" podID="f24e5138-11cb-43cf-a0b4-250fa8f99e92" containerID="7f82ce0f4228f338a94de1b19ed44d664f39319a1ac6d719d0a265725e071dff" exitCode=0
Sep 11 14:35:00.857851 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:00.857705    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-259c79-ssrj9" event={"ID":"f24e5138-11cb-43cf-a0b4-250fa8f99e92","Type":"ContainerDied","Data":"7f82ce0f4228f338a94de1b19ed44d664f39319a1ac6d719d0a265725e071dff"}
Sep 11 14:35:00.857851 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:00.857755    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-259c79-ssrj9" event={"ID":"f24e5138-11cb-43cf-a0b4-250fa8f99e92","Type":"ContainerStarted","Data":"bbcda89f7c9bd80a9846d7f0f8a9bf0585370936f057f3c7f7d7e08a3323d1fc"}
Sep 11 14:35:01.468603 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:35:01.468565    2581 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-259c79/explicit-model-259c79-kserve-self-signed-certs: secret "explicit-model-259c79-kserve-self-signed-certs" not found
Sep 11 14:35:01.469052 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:35:01.468640    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33064ede-db68-4b7c-9d56-14e670c42a2f-tls-certs podName:33064ede-db68-4b7c-9d56-14e670c42a2f nodeName:}" failed. No retries permitted until 2026-09-11 14:35:03.468624636 +0000 UTC m=+1206.619174678 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/33064ede-db68-4b7c-9d56-14e670c42a2f-tls-certs") pod "explicit-model-259c79-kserve-5dcdc74c6c-dwd4r" (UID: "33064ede-db68-4b7c-9d56-14e670c42a2f") : secret "explicit-model-259c79-kserve-self-signed-certs" not found
Sep 11 14:35:01.995390 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:01.995366    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-259c79-ssrj9"
Sep 11 14:35:02.074608 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:02.074576    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f24e5138-11cb-43cf-a0b4-250fa8f99e92-maas-api-service-ca\") pod \"f24e5138-11cb-43cf-a0b4-250fa8f99e92\" (UID: \"f24e5138-11cb-43cf-a0b4-250fa8f99e92\") "
Sep 11 14:35:02.074937 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:02.074913    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f24e5138-11cb-43cf-a0b4-250fa8f99e92-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f24e5138-11cb-43cf-a0b4-250fa8f99e92" (UID: "f24e5138-11cb-43cf-a0b4-250fa8f99e92"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:35:02.176091 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:02.176003    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f24e5138-11cb-43cf-a0b4-250fa8f99e92-maas-api-service-ca\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:35:02.869508 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:02.869470    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-259c79-ssrj9" event={"ID":"f24e5138-11cb-43cf-a0b4-250fa8f99e92","Type":"ContainerDied","Data":"bbcda89f7c9bd80a9846d7f0f8a9bf0585370936f057f3c7f7d7e08a3323d1fc"}
Sep 11 14:35:02.869508 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:02.869507    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-259c79-ssrj9"
Sep 11 14:35:02.869508 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:02.869520    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbcda89f7c9bd80a9846d7f0f8a9bf0585370936f057f3c7f7d7e08a3323d1fc"
Sep 11 14:35:03.490382 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:35:03.490346    2581 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-259c79/explicit-model-259c79-kserve-self-signed-certs: secret "explicit-model-259c79-kserve-self-signed-certs" not found
Sep 11 14:35:03.490570 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:35:03.490423    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33064ede-db68-4b7c-9d56-14e670c42a2f-tls-certs podName:33064ede-db68-4b7c-9d56-14e670c42a2f nodeName:}" failed. No retries permitted until 2026-09-11 14:35:07.490403329 +0000 UTC m=+1210.640953371 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/33064ede-db68-4b7c-9d56-14e670c42a2f-tls-certs") pod "explicit-model-259c79-kserve-5dcdc74c6c-dwd4r" (UID: "33064ede-db68-4b7c-9d56-14e670c42a2f") : secret "explicit-model-259c79-kserve-self-signed-certs" not found
Sep 11 14:35:06.901257 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:06.901219    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e85961/test-model-e85961-kserve-544759b8b5-xqh2b"]
Sep 11 14:35:07.531689 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:35:07.531652    2581 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-259c79/explicit-model-259c79-kserve-self-signed-certs: secret "explicit-model-259c79-kserve-self-signed-certs" not found
Sep 11 14:35:07.531855 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:35:07.531723    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33064ede-db68-4b7c-9d56-14e670c42a2f-tls-certs podName:33064ede-db68-4b7c-9d56-14e670c42a2f nodeName:}" failed. No retries permitted until 2026-09-11 14:35:15.531705625 +0000 UTC m=+1218.682255684 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/33064ede-db68-4b7c-9d56-14e670c42a2f-tls-certs") pod "explicit-model-259c79-kserve-5dcdc74c6c-dwd4r" (UID: "33064ede-db68-4b7c-9d56-14e670c42a2f") : secret "explicit-model-259c79-kserve-self-signed-certs" not found
Sep 11 14:35:07.965598 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:07.965515    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5a7035/test-model-5a7035-kserve-5f44ddb7-9s2b6"]
Sep 11 14:35:08.077763 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:08.077728    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e85961-4cn45"]
Sep 11 14:35:08.078114 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:08.078100    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f24e5138-11cb-43cf-a0b4-250fa8f99e92" containerName="revoke-keys"
Sep 11 14:35:08.078178 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:08.078115    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f24e5138-11cb-43cf-a0b4-250fa8f99e92" containerName="revoke-keys"
Sep 11 14:35:08.078216 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:08.078182    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f24e5138-11cb-43cf-a0b4-250fa8f99e92" containerName="revoke-keys"
Sep 11 14:35:08.081258 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:08.081242    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e85961-4cn45"
Sep 11 14:35:08.084218 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:08.084198    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hcm4g\""
Sep 11 14:35:08.084343 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:08.084199    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 14:35:08.094498 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:08.094467    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e85961-4cn45"]
Sep 11 14:35:08.137342 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:08.137303    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77d56620-4d9d-44c1-8bd4-bc5f97e0f2ee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e85961-4cn45\" (UID: \"77d56620-4d9d-44c1-8bd4-bc5f97e0f2ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e85961-4cn45"
Sep 11 14:35:08.238353 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:08.238259    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77d56620-4d9d-44c1-8bd4-bc5f97e0f2ee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e85961-4cn45\" (UID: \"77d56620-4d9d-44c1-8bd4-bc5f97e0f2ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e85961-4cn45"
Sep 11 14:35:08.238902 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:08.238879    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77d56620-4d9d-44c1-8bd4-bc5f97e0f2ee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e85961-4cn45\" (UID: \"77d56620-4d9d-44c1-8bd4-bc5f97e0f2ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e85961-4cn45"
Sep 11 14:35:08.390881 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:08.390840    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e85961-4cn45"
Sep 11 14:35:08.663558 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:08.663532    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e85961-4cn45"]
Sep 11 14:35:08.665028 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:35:08.664995    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77d56620_4d9d_44c1_8bd4_bc5f97e0f2ee.slice/crio-88ef0ef1868692e06b31cdb5a2b773ba9a802914c245d597f0434cd355319209 WatchSource:0}: Error finding container 88ef0ef1868692e06b31cdb5a2b773ba9a802914c245d597f0434cd355319209: Status 404 returned error can't find the container with id 88ef0ef1868692e06b31cdb5a2b773ba9a802914c245d597f0434cd355319209
Sep 11 14:35:08.899317 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:08.899282    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e85961-4cn45" event={"ID":"77d56620-4d9d-44c1-8bd4-bc5f97e0f2ee","Type":"ContainerStarted","Data":"88ef0ef1868692e06b31cdb5a2b773ba9a802914c245d597f0434cd355319209"}
Sep 11 14:35:09.911753 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:09.911715    2581 generic.go:356] "Generic (PLEG): container finished" podID="77d56620-4d9d-44c1-8bd4-bc5f97e0f2ee" containerID="dad5922a64c1308d360a55c2718ab7791d3cacfe85348de1f57dcbf1145af2c7" exitCode=0
Sep 11 14:35:09.912162 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:09.911803    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e85961-4cn45" event={"ID":"77d56620-4d9d-44c1-8bd4-bc5f97e0f2ee","Type":"ContainerDied","Data":"dad5922a64c1308d360a55c2718ab7791d3cacfe85348de1f57dcbf1145af2c7"}
Sep 11 14:35:11.049083 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:11.049055    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e85961-4cn45"
Sep 11 14:35:11.168029 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:11.167908    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77d56620-4d9d-44c1-8bd4-bc5f97e0f2ee-maas-api-service-ca\") pod \"77d56620-4d9d-44c1-8bd4-bc5f97e0f2ee\" (UID: \"77d56620-4d9d-44c1-8bd4-bc5f97e0f2ee\") "
Sep 11 14:35:11.168262 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:11.168230    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77d56620-4d9d-44c1-8bd4-bc5f97e0f2ee-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "77d56620-4d9d-44c1-8bd4-bc5f97e0f2ee" (UID: "77d56620-4d9d-44c1-8bd4-bc5f97e0f2ee"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:35:11.268869 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:11.268835    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77d56620-4d9d-44c1-8bd4-bc5f97e0f2ee-maas-api-service-ca\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:35:11.924258 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:11.924163    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e85961-4cn45"
Sep 11 14:35:11.924258 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:11.924180    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e85961-4cn45" event={"ID":"77d56620-4d9d-44c1-8bd4-bc5f97e0f2ee","Type":"ContainerDied","Data":"88ef0ef1868692e06b31cdb5a2b773ba9a802914c245d597f0434cd355319209"}
Sep 11 14:35:11.924258 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:11.924245    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88ef0ef1868692e06b31cdb5a2b773ba9a802914c245d597f0434cd355319209"
Sep 11 14:35:13.826880 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:13.826847    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-259c79-ssrj9"]
Sep 11 14:35:13.834119 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:13.834082    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-259c79-ssrj9"]
Sep 11 14:35:15.499251 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:15.499215    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f24e5138-11cb-43cf-a0b4-250fa8f99e92" path="/var/lib/kubelet/pods/f24e5138-11cb-43cf-a0b4-250fa8f99e92/volumes"
Sep 11 14:35:15.612639 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:35:15.612603    2581 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-259c79/explicit-model-259c79-kserve-self-signed-certs: secret "explicit-model-259c79-kserve-self-signed-certs" not found
Sep 11 14:35:15.612822 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:35:15.612686    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33064ede-db68-4b7c-9d56-14e670c42a2f-tls-certs podName:33064ede-db68-4b7c-9d56-14e670c42a2f nodeName:}" failed. No retries permitted until 2026-09-11 14:35:31.612668286 +0000 UTC m=+1234.763218328 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/33064ede-db68-4b7c-9d56-14e670c42a2f-tls-certs") pod "explicit-model-259c79-kserve-5dcdc74c6c-dwd4r" (UID: "33064ede-db68-4b7c-9d56-14e670c42a2f") : secret "explicit-model-259c79-kserve-self-signed-certs" not found
Sep 11 14:35:15.878066 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:15.877950    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-259c79/explicit-model-259c79-kserve-5dcdc74c6c-dwd4r" podUID="33064ede-db68-4b7c-9d56-14e670c42a2f" containerName="main" containerID="cri-o://45d686a9f5fc4a56f57e9107794099b0f389272ef9c83f98e5cfa9a9362099e4" gracePeriod=45
Sep 11 14:35:16.132143 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:16.132078    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-259c79/explicit-model-259c79-kserve-5dcdc74c6c-dwd4r"
Sep 11 14:35:16.219213 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:16.219177    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wcpbm\" (UniqueName: \"kubernetes.io/projected/33064ede-db68-4b7c-9d56-14e670c42a2f-kube-api-access-wcpbm\") pod \"33064ede-db68-4b7c-9d56-14e670c42a2f\" (UID: \"33064ede-db68-4b7c-9d56-14e670c42a2f\") "
Sep 11 14:35:16.219410 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:16.219250    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33064ede-db68-4b7c-9d56-14e670c42a2f-tls-certs\") pod \"33064ede-db68-4b7c-9d56-14e670c42a2f\" (UID: \"33064ede-db68-4b7c-9d56-14e670c42a2f\") "
Sep 11 14:35:16.219410 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:16.219295    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33064ede-db68-4b7c-9d56-14e670c42a2f-model-cache\") pod \"33064ede-db68-4b7c-9d56-14e670c42a2f\" (UID: \"33064ede-db68-4b7c-9d56-14e670c42a2f\") "
Sep 11 14:35:16.219410 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:16.219327    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33064ede-db68-4b7c-9d56-14e670c42a2f-dshm\") pod \"33064ede-db68-4b7c-9d56-14e670c42a2f\" (UID: \"33064ede-db68-4b7c-9d56-14e670c42a2f\") "
Sep 11 14:35:16.219410 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:16.219364    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33064ede-db68-4b7c-9d56-14e670c42a2f-tmp-dir\") pod \"33064ede-db68-4b7c-9d56-14e670c42a2f\" (UID: \"33064ede-db68-4b7c-9d56-14e670c42a2f\") "
Sep 11 14:35:16.219410 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:16.219395    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33064ede-db68-4b7c-9d56-14e670c42a2f-home\") pod \"33064ede-db68-4b7c-9d56-14e670c42a2f\" (UID: \"33064ede-db68-4b7c-9d56-14e670c42a2f\") "
Sep 11 14:35:16.219699 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:16.219570    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33064ede-db68-4b7c-9d56-14e670c42a2f-model-cache" (OuterVolumeSpecName: "model-cache") pod "33064ede-db68-4b7c-9d56-14e670c42a2f" (UID: "33064ede-db68-4b7c-9d56-14e670c42a2f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:35:16.219699 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:16.219667    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33064ede-db68-4b7c-9d56-14e670c42a2f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "33064ede-db68-4b7c-9d56-14e670c42a2f" (UID: "33064ede-db68-4b7c-9d56-14e670c42a2f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:35:16.219821 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:16.219771    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33064ede-db68-4b7c-9d56-14e670c42a2f-home" (OuterVolumeSpecName: "home") pod "33064ede-db68-4b7c-9d56-14e670c42a2f" (UID: "33064ede-db68-4b7c-9d56-14e670c42a2f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:35:16.219872 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:16.219833    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33064ede-db68-4b7c-9d56-14e670c42a2f-model-cache\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:35:16.219872 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:16.219852    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33064ede-db68-4b7c-9d56-14e670c42a2f-tmp-dir\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:35:16.221545 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:16.221524    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33064ede-db68-4b7c-9d56-14e670c42a2f-dshm" (OuterVolumeSpecName: "dshm") pod "33064ede-db68-4b7c-9d56-14e670c42a2f" (UID: "33064ede-db68-4b7c-9d56-14e670c42a2f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:35:16.221640 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:16.221545    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33064ede-db68-4b7c-9d56-14e670c42a2f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "33064ede-db68-4b7c-9d56-14e670c42a2f" (UID: "33064ede-db68-4b7c-9d56-14e670c42a2f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:35:16.221696 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:16.221643    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33064ede-db68-4b7c-9d56-14e670c42a2f-kube-api-access-wcpbm" (OuterVolumeSpecName: "kube-api-access-wcpbm") pod "33064ede-db68-4b7c-9d56-14e670c42a2f" (UID: "33064ede-db68-4b7c-9d56-14e670c42a2f"). InnerVolumeSpecName "kube-api-access-wcpbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:35:16.321336 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:16.321303    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33064ede-db68-4b7c-9d56-14e670c42a2f-dshm\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:35:16.321336 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:16.321328    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33064ede-db68-4b7c-9d56-14e670c42a2f-home\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:35:16.321336 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:16.321339    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcpbm\" (UniqueName: \"kubernetes.io/projected/33064ede-db68-4b7c-9d56-14e670c42a2f-kube-api-access-wcpbm\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:35:16.321586 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:16.321349    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33064ede-db68-4b7c-9d56-14e670c42a2f-tls-certs\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:35:16.961175 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:16.961137    2581 generic.go:356] "Generic (PLEG): container finished" podID="33064ede-db68-4b7c-9d56-14e670c42a2f" containerID="45d686a9f5fc4a56f57e9107794099b0f389272ef9c83f98e5cfa9a9362099e4" exitCode=0
Sep 11 14:35:16.961621 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:16.961187    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-259c79/explicit-model-259c79-kserve-5dcdc74c6c-dwd4r"
Sep 11 14:35:16.961621 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:16.961201    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-259c79/explicit-model-259c79-kserve-5dcdc74c6c-dwd4r" event={"ID":"33064ede-db68-4b7c-9d56-14e670c42a2f","Type":"ContainerDied","Data":"45d686a9f5fc4a56f57e9107794099b0f389272ef9c83f98e5cfa9a9362099e4"}
Sep 11 14:35:16.961621 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:16.961232    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-259c79/explicit-model-259c79-kserve-5dcdc74c6c-dwd4r" event={"ID":"33064ede-db68-4b7c-9d56-14e670c42a2f","Type":"ContainerDied","Data":"b1e81d7e2bddc0484c17637ca503d3ec41aa0d39ee2fad4971bf9be7f46495cc"}
Sep 11 14:35:16.961621 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:16.961249    2581 scope.go:117] "RemoveContainer" containerID="45d686a9f5fc4a56f57e9107794099b0f389272ef9c83f98e5cfa9a9362099e4"
Sep 11 14:35:16.973801 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:16.973781    2581 scope.go:117] "RemoveContainer" containerID="45d686a9f5fc4a56f57e9107794099b0f389272ef9c83f98e5cfa9a9362099e4"
Sep 11 14:35:16.974127 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:35:16.974107    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45d686a9f5fc4a56f57e9107794099b0f389272ef9c83f98e5cfa9a9362099e4\": container with ID starting with 45d686a9f5fc4a56f57e9107794099b0f389272ef9c83f98e5cfa9a9362099e4 not found: ID does not exist" containerID="45d686a9f5fc4a56f57e9107794099b0f389272ef9c83f98e5cfa9a9362099e4"
Sep 11 14:35:16.974195 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:16.974136    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45d686a9f5fc4a56f57e9107794099b0f389272ef9c83f98e5cfa9a9362099e4"} err="failed to get container status \"45d686a9f5fc4a56f57e9107794099b0f389272ef9c83f98e5cfa9a9362099e4\": rpc error: code = NotFound desc = could not find container \"45d686a9f5fc4a56f57e9107794099b0f389272ef9c83f98e5cfa9a9362099e4\": container with ID starting with 45d686a9f5fc4a56f57e9107794099b0f389272ef9c83f98e5cfa9a9362099e4 not found: ID does not exist"
Sep 11 14:35:16.993820 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:16.993787    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-259c79/explicit-model-259c79-kserve-5dcdc74c6c-dwd4r"]
Sep 11 14:35:17.025186 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:17.025156    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-259c79/explicit-model-259c79-kserve-5dcdc74c6c-dwd4r"]
Sep 11 14:35:17.501072 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:17.501034    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33064ede-db68-4b7c-9d56-14e670c42a2f" path="/var/lib/kubelet/pods/33064ede-db68-4b7c-9d56-14e670c42a2f/volumes"
Sep 11 14:35:18.627112 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:18.627079    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e85961-4cn45"]
Sep 11 14:35:18.631612 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:18.631584    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e85961-4cn45"]
Sep 11 14:35:19.499412 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:19.499377    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77d56620-4d9d-44c1-8bd4-bc5f97e0f2ee" path="/var/lib/kubelet/pods/77d56620-4d9d-44c1-8bd4-bc5f97e0f2ee/volumes"
Sep 11 14:35:21.920156 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:21.920093    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-e85961/test-model-e85961-kserve-544759b8b5-xqh2b" podUID="f5685f06-073d-48b6-8d7b-b332528573c1" containerName="main" containerID="cri-o://65baeffb2ea960736fd9a375823267d068b958c4a16d337e174807bd7ce58bbc" gracePeriod=45
Sep 11 14:35:22.168885 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:22.168857    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e85961/test-model-e85961-kserve-544759b8b5-xqh2b"
Sep 11 14:35:22.279480 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:22.279445    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nq8b9\" (UniqueName: \"kubernetes.io/projected/f5685f06-073d-48b6-8d7b-b332528573c1-kube-api-access-nq8b9\") pod \"f5685f06-073d-48b6-8d7b-b332528573c1\" (UID: \"f5685f06-073d-48b6-8d7b-b332528573c1\") "
Sep 11 14:35:22.279654 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:22.279510    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5685f06-073d-48b6-8d7b-b332528573c1-tmp-dir\") pod \"f5685f06-073d-48b6-8d7b-b332528573c1\" (UID: \"f5685f06-073d-48b6-8d7b-b332528573c1\") "
Sep 11 14:35:22.279654 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:22.279562    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5685f06-073d-48b6-8d7b-b332528573c1-home\") pod \"f5685f06-073d-48b6-8d7b-b332528573c1\" (UID: \"f5685f06-073d-48b6-8d7b-b332528573c1\") "
Sep 11 14:35:22.279654 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:22.279618    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5685f06-073d-48b6-8d7b-b332528573c1-tls-certs\") pod \"f5685f06-073d-48b6-8d7b-b332528573c1\" (UID: \"f5685f06-073d-48b6-8d7b-b332528573c1\") "
Sep 11 14:35:22.279815 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:22.279682    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5685f06-073d-48b6-8d7b-b332528573c1-model-cache\") pod \"f5685f06-073d-48b6-8d7b-b332528573c1\" (UID: \"f5685f06-073d-48b6-8d7b-b332528573c1\") "
Sep 11 14:35:22.279815 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:22.279734    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5685f06-073d-48b6-8d7b-b332528573c1-dshm\") pod \"f5685f06-073d-48b6-8d7b-b332528573c1\" (UID: \"f5685f06-073d-48b6-8d7b-b332528573c1\") "
Sep 11 14:35:22.279924 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:22.279849    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5685f06-073d-48b6-8d7b-b332528573c1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f5685f06-073d-48b6-8d7b-b332528573c1" (UID: "f5685f06-073d-48b6-8d7b-b332528573c1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:35:22.280011 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:22.279924    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5685f06-073d-48b6-8d7b-b332528573c1-home" (OuterVolumeSpecName: "home") pod "f5685f06-073d-48b6-8d7b-b332528573c1" (UID: "f5685f06-073d-48b6-8d7b-b332528573c1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:35:22.280011 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:22.279953    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5685f06-073d-48b6-8d7b-b332528573c1-model-cache" (OuterVolumeSpecName: "model-cache") pod "f5685f06-073d-48b6-8d7b-b332528573c1" (UID: "f5685f06-073d-48b6-8d7b-b332528573c1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:35:22.280146 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:22.280098    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5685f06-073d-48b6-8d7b-b332528573c1-tmp-dir\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:35:22.280146 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:22.280116    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5685f06-073d-48b6-8d7b-b332528573c1-home\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:35:22.280146 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:22.280124    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5685f06-073d-48b6-8d7b-b332528573c1-model-cache\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:35:22.281798 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:22.281768    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5685f06-073d-48b6-8d7b-b332528573c1-kube-api-access-nq8b9" (OuterVolumeSpecName: "kube-api-access-nq8b9") pod "f5685f06-073d-48b6-8d7b-b332528573c1" (UID: "f5685f06-073d-48b6-8d7b-b332528573c1"). InnerVolumeSpecName "kube-api-access-nq8b9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:35:22.281798 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:22.281783    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5685f06-073d-48b6-8d7b-b332528573c1-dshm" (OuterVolumeSpecName: "dshm") pod "f5685f06-073d-48b6-8d7b-b332528573c1" (UID: "f5685f06-073d-48b6-8d7b-b332528573c1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:35:22.281798 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:22.281777    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5685f06-073d-48b6-8d7b-b332528573c1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f5685f06-073d-48b6-8d7b-b332528573c1" (UID: "f5685f06-073d-48b6-8d7b-b332528573c1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:35:22.381326 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:22.381295    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5685f06-073d-48b6-8d7b-b332528573c1-dshm\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:35:22.381326 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:22.381320    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nq8b9\" (UniqueName: \"kubernetes.io/projected/f5685f06-073d-48b6-8d7b-b332528573c1-kube-api-access-nq8b9\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:35:22.381326 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:22.381332    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5685f06-073d-48b6-8d7b-b332528573c1-tls-certs\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:35:22.984337 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:22.984288    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-5a7035/test-model-5a7035-kserve-5f44ddb7-9s2b6" podUID="44f5281c-8d94-4459-b151-cdfe7565ae31" containerName="main" containerID="cri-o://8fa2f2be23b05bfc7827185fa4b12850693e938846a5133f484c71ad06c90d17" gracePeriod=45
Sep 11 14:35:22.994766 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:22.994737    2581 generic.go:356] "Generic (PLEG): container finished" podID="f5685f06-073d-48b6-8d7b-b332528573c1" containerID="65baeffb2ea960736fd9a375823267d068b958c4a16d337e174807bd7ce58bbc" exitCode=0
Sep 11 14:35:22.994900 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:22.994796    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e85961/test-model-e85961-kserve-544759b8b5-xqh2b"
Sep 11 14:35:22.994900 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:22.994816    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e85961/test-model-e85961-kserve-544759b8b5-xqh2b" event={"ID":"f5685f06-073d-48b6-8d7b-b332528573c1","Type":"ContainerDied","Data":"65baeffb2ea960736fd9a375823267d068b958c4a16d337e174807bd7ce58bbc"}
Sep 11 14:35:22.994900 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:22.994859    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e85961/test-model-e85961-kserve-544759b8b5-xqh2b" event={"ID":"f5685f06-073d-48b6-8d7b-b332528573c1","Type":"ContainerDied","Data":"ffa363b4a8648aa26f2469238ad186864a2838f6eb02ae4940e8d000bc5f8d26"}
Sep 11 14:35:22.994900 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:22.994876    2581 scope.go:117] "RemoveContainer" containerID="65baeffb2ea960736fd9a375823267d068b958c4a16d337e174807bd7ce58bbc"
Sep 11 14:35:23.076879 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:23.076840    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e85961/test-model-e85961-kserve-544759b8b5-xqh2b"]
Sep 11 14:35:23.081831 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:23.081804    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-e85961/test-model-e85961-kserve-544759b8b5-xqh2b"]
Sep 11 14:35:23.095954 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:23.095935    2581 scope.go:117] "RemoveContainer" containerID="65baeffb2ea960736fd9a375823267d068b958c4a16d337e174807bd7ce58bbc"
Sep 11 14:35:23.096364 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:35:23.096338    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65baeffb2ea960736fd9a375823267d068b958c4a16d337e174807bd7ce58bbc\": container with ID starting with 65baeffb2ea960736fd9a375823267d068b958c4a16d337e174807bd7ce58bbc not found: ID does not exist" containerID="65baeffb2ea960736fd9a375823267d068b958c4a16d337e174807bd7ce58bbc"
Sep 11 14:35:23.096479 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:23.096369    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65baeffb2ea960736fd9a375823267d068b958c4a16d337e174807bd7ce58bbc"} err="failed to get container status \"65baeffb2ea960736fd9a375823267d068b958c4a16d337e174807bd7ce58bbc\": rpc error: code = NotFound desc = could not find container \"65baeffb2ea960736fd9a375823267d068b958c4a16d337e174807bd7ce58bbc\": container with ID starting with 65baeffb2ea960736fd9a375823267d068b958c4a16d337e174807bd7ce58bbc not found: ID does not exist"
Sep 11 14:35:23.240164 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:23.240141    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5a7035/test-model-5a7035-kserve-5f44ddb7-9s2b6"
Sep 11 14:35:23.289641 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:23.289612    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vd9p2\" (UniqueName: \"kubernetes.io/projected/44f5281c-8d94-4459-b151-cdfe7565ae31-kube-api-access-vd9p2\") pod \"44f5281c-8d94-4459-b151-cdfe7565ae31\" (UID: \"44f5281c-8d94-4459-b151-cdfe7565ae31\") "
Sep 11 14:35:23.289788 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:23.289674    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44f5281c-8d94-4459-b151-cdfe7565ae31-tls-certs\") pod \"44f5281c-8d94-4459-b151-cdfe7565ae31\" (UID: \"44f5281c-8d94-4459-b151-cdfe7565ae31\") "
Sep 11 14:35:23.289788 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:23.289702    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44f5281c-8d94-4459-b151-cdfe7565ae31-model-cache\") pod \"44f5281c-8d94-4459-b151-cdfe7565ae31\" (UID: \"44f5281c-8d94-4459-b151-cdfe7565ae31\") "
Sep 11 14:35:23.289788 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:23.289742    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44f5281c-8d94-4459-b151-cdfe7565ae31-tmp-dir\") pod \"44f5281c-8d94-4459-b151-cdfe7565ae31\" (UID: \"44f5281c-8d94-4459-b151-cdfe7565ae31\") "
Sep 11 14:35:23.289788 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:23.289760    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44f5281c-8d94-4459-b151-cdfe7565ae31-home\") pod \"44f5281c-8d94-4459-b151-cdfe7565ae31\" (UID: \"44f5281c-8d94-4459-b151-cdfe7565ae31\") "
Sep 11 14:35:23.290042 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:23.289799    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44f5281c-8d94-4459-b151-cdfe7565ae31-dshm\") pod \"44f5281c-8d94-4459-b151-cdfe7565ae31\" (UID: \"44f5281c-8d94-4459-b151-cdfe7565ae31\") "
Sep 11 14:35:23.290117 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:23.290056    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44f5281c-8d94-4459-b151-cdfe7565ae31-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "44f5281c-8d94-4459-b151-cdfe7565ae31" (UID: "44f5281c-8d94-4459-b151-cdfe7565ae31"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:35:23.290117 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:23.290102    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44f5281c-8d94-4459-b151-cdfe7565ae31-model-cache" (OuterVolumeSpecName: "model-cache") pod "44f5281c-8d94-4459-b151-cdfe7565ae31" (UID: "44f5281c-8d94-4459-b151-cdfe7565ae31"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:35:23.290218 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:23.290111    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44f5281c-8d94-4459-b151-cdfe7565ae31-home" (OuterVolumeSpecName: "home") pod "44f5281c-8d94-4459-b151-cdfe7565ae31" (UID: "44f5281c-8d94-4459-b151-cdfe7565ae31"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:35:23.292292 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:23.292265    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44f5281c-8d94-4459-b151-cdfe7565ae31-dshm" (OuterVolumeSpecName: "dshm") pod "44f5281c-8d94-4459-b151-cdfe7565ae31" (UID: "44f5281c-8d94-4459-b151-cdfe7565ae31"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:35:23.292391 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:23.292332    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44f5281c-8d94-4459-b151-cdfe7565ae31-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "44f5281c-8d94-4459-b151-cdfe7565ae31" (UID: "44f5281c-8d94-4459-b151-cdfe7565ae31"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:35:23.292391 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:23.292348    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44f5281c-8d94-4459-b151-cdfe7565ae31-kube-api-access-vd9p2" (OuterVolumeSpecName: "kube-api-access-vd9p2") pod "44f5281c-8d94-4459-b151-cdfe7565ae31" (UID: "44f5281c-8d94-4459-b151-cdfe7565ae31"). InnerVolumeSpecName "kube-api-access-vd9p2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:35:23.390863 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:23.390819    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44f5281c-8d94-4459-b151-cdfe7565ae31-tls-certs\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:35:23.390863 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:23.390855    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44f5281c-8d94-4459-b151-cdfe7565ae31-model-cache\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:35:23.390863 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:23.390864    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44f5281c-8d94-4459-b151-cdfe7565ae31-tmp-dir\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:35:23.390863 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:23.390871    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44f5281c-8d94-4459-b151-cdfe7565ae31-home\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:35:23.391179 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:23.390879    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44f5281c-8d94-4459-b151-cdfe7565ae31-dshm\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:35:23.391179 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:23.390888    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd9p2\" (UniqueName: \"kubernetes.io/projected/44f5281c-8d94-4459-b151-cdfe7565ae31-kube-api-access-vd9p2\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:35:23.499080 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:23.498997    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5685f06-073d-48b6-8d7b-b332528573c1" path="/var/lib/kubelet/pods/f5685f06-073d-48b6-8d7b-b332528573c1/volumes"
Sep 11 14:35:24.001146 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:24.001117    2581 generic.go:356] "Generic (PLEG): container finished" podID="44f5281c-8d94-4459-b151-cdfe7565ae31" containerID="8fa2f2be23b05bfc7827185fa4b12850693e938846a5133f484c71ad06c90d17" exitCode=0
Sep 11 14:35:24.001602 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:24.001165    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5a7035/test-model-5a7035-kserve-5f44ddb7-9s2b6"
Sep 11 14:35:24.001602 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:24.001198    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5a7035/test-model-5a7035-kserve-5f44ddb7-9s2b6" event={"ID":"44f5281c-8d94-4459-b151-cdfe7565ae31","Type":"ContainerDied","Data":"8fa2f2be23b05bfc7827185fa4b12850693e938846a5133f484c71ad06c90d17"}
Sep 11 14:35:24.001602 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:24.001246    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5a7035/test-model-5a7035-kserve-5f44ddb7-9s2b6" event={"ID":"44f5281c-8d94-4459-b151-cdfe7565ae31","Type":"ContainerDied","Data":"fc2f91f4475e26a732d35c2815e8eaaca6e0a429b42c7326ba17b7368e650d99"}
Sep 11 14:35:24.001602 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:24.001269    2581 scope.go:117] "RemoveContainer" containerID="8fa2f2be23b05bfc7827185fa4b12850693e938846a5133f484c71ad06c90d17"
Sep 11 14:35:24.009650 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:24.009629    2581 scope.go:117] "RemoveContainer" containerID="8fa2f2be23b05bfc7827185fa4b12850693e938846a5133f484c71ad06c90d17"
Sep 11 14:35:24.009887 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:35:24.009869    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fa2f2be23b05bfc7827185fa4b12850693e938846a5133f484c71ad06c90d17\": container with ID starting with 8fa2f2be23b05bfc7827185fa4b12850693e938846a5133f484c71ad06c90d17 not found: ID does not exist" containerID="8fa2f2be23b05bfc7827185fa4b12850693e938846a5133f484c71ad06c90d17"
Sep 11 14:35:24.009940 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:24.009898    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fa2f2be23b05bfc7827185fa4b12850693e938846a5133f484c71ad06c90d17"} err="failed to get container status \"8fa2f2be23b05bfc7827185fa4b12850693e938846a5133f484c71ad06c90d17\": rpc error: code = NotFound desc = could not find container \"8fa2f2be23b05bfc7827185fa4b12850693e938846a5133f484c71ad06c90d17\": container with ID starting with 8fa2f2be23b05bfc7827185fa4b12850693e938846a5133f484c71ad06c90d17 not found: ID does not exist"
Sep 11 14:35:24.030465 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:24.030433    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5a7035/test-model-5a7035-kserve-5f44ddb7-9s2b6"]
Sep 11 14:35:24.036742 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:24.036720    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-5a7035/test-model-5a7035-kserve-5f44ddb7-9s2b6"]
Sep 11 14:35:25.215086 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:25.215054    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4"]
Sep 11 14:35:25.215464 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:25.215388    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5685f06-073d-48b6-8d7b-b332528573c1" containerName="main"
Sep 11 14:35:25.215464 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:25.215400    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5685f06-073d-48b6-8d7b-b332528573c1" containerName="main"
Sep 11 14:35:25.215464 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:25.215412    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77d56620-4d9d-44c1-8bd4-bc5f97e0f2ee" containerName="revoke-keys"
Sep 11 14:35:25.215464 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:25.215421    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="77d56620-4d9d-44c1-8bd4-bc5f97e0f2ee" containerName="revoke-keys"
Sep 11 14:35:25.215464 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:25.215427    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44f5281c-8d94-4459-b151-cdfe7565ae31" containerName="main"
Sep 11 14:35:25.215464 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:25.215432    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="44f5281c-8d94-4459-b151-cdfe7565ae31" containerName="main"
Sep 11 14:35:25.215464 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:25.215438    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33064ede-db68-4b7c-9d56-14e670c42a2f" containerName="main"
Sep 11 14:35:25.215464 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:25.215444    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="33064ede-db68-4b7c-9d56-14e670c42a2f" containerName="main"
Sep 11 14:35:25.215718 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:25.215491    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="44f5281c-8d94-4459-b151-cdfe7565ae31" containerName="main"
Sep 11 14:35:25.215718 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:25.215499    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="33064ede-db68-4b7c-9d56-14e670c42a2f" containerName="main"
Sep 11 14:35:25.215718 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:25.215507    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="77d56620-4d9d-44c1-8bd4-bc5f97e0f2ee" containerName="revoke-keys"
Sep 11 14:35:25.215718 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:25.215514    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5685f06-073d-48b6-8d7b-b332528573c1" containerName="main"
Sep 11 14:35:25.219826 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:25.219807    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4"
Sep 11 14:35:25.233306 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:25.233284    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-98be89-openshift-default-dockercfg-g6sr8\""
Sep 11 14:35:25.250410 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:25.250381    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4"]
Sep 11 14:35:25.306041 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:25.306006    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/849d7667-fb78-4454-aac1-662b2c962367-workload-certs\") pod \"e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4\" (UID: \"849d7667-fb78-4454-aac1-662b2c962367\") " pod="openshift-ingress/e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4"
Sep 11 14:35:25.306232 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:25.306061    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/849d7667-fb78-4454-aac1-662b2c962367-workload-socket\") pod \"e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4\" (UID: \"849d7667-fb78-4454-aac1-662b2c962367\") " pod="openshift-ingress/e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4"
Sep 11 14:35:25.306232 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:25.306106    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/849d7667-fb78-4454-aac1-662b2c962367-istio-podinfo\") pod \"e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4\" (UID: \"849d7667-fb78-4454-aac1-662b2c962367\") " pod="openshift-ingress/e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4"
Sep 11 14:35:25.306232 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:25.306197    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/849d7667-fb78-4454-aac1-662b2c962367-istiod-ca-cert\") pod \"e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4\" (UID: \"849d7667-fb78-4454-aac1-662b2c962367\") " pod="openshift-ingress/e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4"
Sep 11 14:35:25.306424 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:25.306270    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/849d7667-fb78-4454-aac1-662b2c962367-istio-data\") pod \"e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4\" (UID: \"849d7667-fb78-4454-aac1-662b2c962367\") " pod="openshift-ingress/e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4"
Sep 11 14:35:25.306424 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:25.306301    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/849d7667-fb78-4454-aac1-662b2c962367-istio-token\") pod \"e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4\" (UID: \"849d7667-fb78-4454-aac1-662b2c962367\") " pod="openshift-ingress/e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4"
Sep 11 14:35:25.306424 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:25.306355    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/849d7667-fb78-4454-aac1-662b2c962367-istio-envoy\") pod \"e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4\" (UID: \"849d7667-fb78-4454-aac1-662b2c962367\") " pod="openshift-ingress/e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4"
Sep 11 14:35:25.306586 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:25.306447    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/849d7667-fb78-4454-aac1-662b2c962367-credential-socket\") pod \"e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4\" (UID: \"849d7667-fb78-4454-aac1-662b2c962367\") " pod="openshift-ingress/e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4"
Sep 11 14:35:25.306586 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:25.306499    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95554\" (UniqueName: \"kubernetes.io/projected/849d7667-fb78-4454-aac1-662b2c962367-kube-api-access-95554\") pod \"e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4\" (UID: \"849d7667-fb78-4454-aac1-662b2c962367\") " pod="openshift-ingress/e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4"
Sep 11 14:35:25.407703 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:25.407665    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/849d7667-fb78-4454-aac1-662b2c962367-credential-socket\") pod \"e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4\" (UID: \"849d7667-fb78-4454-aac1-662b2c962367\") " pod="openshift-ingress/e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4"
Sep 11 14:35:25.407703 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:25.407711    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95554\" (UniqueName: \"kubernetes.io/projected/849d7667-fb78-4454-aac1-662b2c962367-kube-api-access-95554\") pod \"e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4\" (UID: \"849d7667-fb78-4454-aac1-662b2c962367\") " pod="openshift-ingress/e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4"
Sep 11 14:35:25.407960 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:25.407771    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/849d7667-fb78-4454-aac1-662b2c962367-workload-certs\") pod \"e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4\" (UID: \"849d7667-fb78-4454-aac1-662b2c962367\") " pod="openshift-ingress/e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4"
Sep 11 14:35:25.407960 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:25.407796    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/849d7667-fb78-4454-aac1-662b2c962367-workload-socket\") pod \"e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4\" (UID: \"849d7667-fb78-4454-aac1-662b2c962367\") " pod="openshift-ingress/e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4"
Sep 11 14:35:25.407960 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:25.407817    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/849d7667-fb78-4454-aac1-662b2c962367-istio-podinfo\") pod \"e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4\" (UID: \"849d7667-fb78-4454-aac1-662b2c962367\") " pod="openshift-ingress/e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4"
Sep 11 14:35:25.407960 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:25.407836    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/849d7667-fb78-4454-aac1-662b2c962367-istiod-ca-cert\") pod \"e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4\" (UID: \"849d7667-fb78-4454-aac1-662b2c962367\") " pod="openshift-ingress/e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4"
Sep 11 14:35:25.407960 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:25.407857    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/849d7667-fb78-4454-aac1-662b2c962367-istio-data\") pod \"e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4\" (UID: \"849d7667-fb78-4454-aac1-662b2c962367\") " pod="openshift-ingress/e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4"
Sep 11 14:35:25.407960 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:25.407884    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/849d7667-fb78-4454-aac1-662b2c962367-istio-token\") pod \"e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4\" (UID: \"849d7667-fb78-4454-aac1-662b2c962367\") " pod="openshift-ingress/e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4"
Sep 11 14:35:25.407960 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:25.407921    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/849d7667-fb78-4454-aac1-662b2c962367-istio-envoy\") pod \"e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4\" (UID: \"849d7667-fb78-4454-aac1-662b2c962367\") " pod="openshift-ingress/e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4"
Sep 11 14:35:25.408383 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:25.408103    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/849d7667-fb78-4454-aac1-662b2c962367-credential-socket\") pod \"e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4\" (UID: \"849d7667-fb78-4454-aac1-662b2c962367\") " pod="openshift-ingress/e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4"
Sep 11 14:35:25.408383 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:25.408162    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/849d7667-fb78-4454-aac1-662b2c962367-workload-certs\") pod \"e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4\" (UID: \"849d7667-fb78-4454-aac1-662b2c962367\") " pod="openshift-ingress/e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4"
Sep 11 14:35:25.408383 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:25.408331    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/849d7667-fb78-4454-aac1-662b2c962367-workload-socket\") pod \"e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4\" (UID: \"849d7667-fb78-4454-aac1-662b2c962367\") " pod="openshift-ingress/e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4"
Sep 11 14:35:25.408546 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:25.408471    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/849d7667-fb78-4454-aac1-662b2c962367-istio-data\") pod \"e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4\" (UID: \"849d7667-fb78-4454-aac1-662b2c962367\") " pod="openshift-ingress/e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4"
Sep 11 14:35:25.408703 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:25.408679    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/849d7667-fb78-4454-aac1-662b2c962367-istiod-ca-cert\") pod \"e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4\" (UID: \"849d7667-fb78-4454-aac1-662b2c962367\") " pod="openshift-ingress/e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4"
Sep 11 14:35:25.410514 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:25.410492    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/849d7667-fb78-4454-aac1-662b2c962367-istio-podinfo\") pod \"e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4\" (UID: \"849d7667-fb78-4454-aac1-662b2c962367\") " pod="openshift-ingress/e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4"
Sep 11 14:35:25.410621 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:25.410490    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/849d7667-fb78-4454-aac1-662b2c962367-istio-envoy\") pod \"e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4\" (UID: \"849d7667-fb78-4454-aac1-662b2c962367\") " pod="openshift-ingress/e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4"
Sep 11 14:35:25.424689 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:25.424661    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/849d7667-fb78-4454-aac1-662b2c962367-istio-token\") pod \"e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4\" (UID: \"849d7667-fb78-4454-aac1-662b2c962367\") " pod="openshift-ingress/e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4"
Sep 11 14:35:25.424797 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:25.424749    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95554\" (UniqueName: \"kubernetes.io/projected/849d7667-fb78-4454-aac1-662b2c962367-kube-api-access-95554\") pod \"e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4\" (UID: \"849d7667-fb78-4454-aac1-662b2c962367\") " pod="openshift-ingress/e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4"
Sep 11 14:35:25.500386 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:25.500351    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44f5281c-8d94-4459-b151-cdfe7565ae31" path="/var/lib/kubelet/pods/44f5281c-8d94-4459-b151-cdfe7565ae31/volumes"
Sep 11 14:35:25.530420 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:25.530387    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4"
Sep 11 14:35:25.838421 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:25.838394    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4"]
Sep 11 14:35:25.840944 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:35:25.840902    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod849d7667_fb78_4454_aac1_662b2c962367.slice/crio-7316e1028fc7d42b0d2b9d3228da39cb970ff1e9e6c0fb820ea3630c25f4846e WatchSource:0}: Error finding container 7316e1028fc7d42b0d2b9d3228da39cb970ff1e9e6c0fb820ea3630c25f4846e: Status 404 returned error can't find the container with id 7316e1028fc7d42b0d2b9d3228da39cb970ff1e9e6c0fb820ea3630c25f4846e
Sep 11 14:35:25.843222 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:25.843184    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:35:25.843308 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:25.843255    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:35:25.843308 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:25.843281    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:35:26.017804 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:26.017771    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4" event={"ID":"849d7667-fb78-4454-aac1-662b2c962367","Type":"ContainerStarted","Data":"8dbecd4c45b5c7d363582bb0f10f1a4406357efe2044f57b268e0e8618519ba2"}
Sep 11 14:35:26.017995 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:26.017846    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4" event={"ID":"849d7667-fb78-4454-aac1-662b2c962367","Type":"ContainerStarted","Data":"7316e1028fc7d42b0d2b9d3228da39cb970ff1e9e6c0fb820ea3630c25f4846e"}
Sep 11 14:35:26.055901 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:26.055837    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4" podStartSLOduration=1.05581411 podStartE2EDuration="1.05581411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:35:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:35:26.053267009 +0000 UTC m=+1229.203817073" watchObservedRunningTime="2026-09-11 14:35:26.05581411 +0000 UTC m=+1229.206364174"
Sep 11 14:35:26.530992 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:26.530947    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4"
Sep 11 14:35:26.535676 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:26.535653    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4"
Sep 11 14:35:27.023755 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:27.023722    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4"
Sep 11 14:35:27.024699 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:27.024675    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4"
Sep 11 14:35:30.898117 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:30.898081    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg"]
Sep 11 14:35:30.898634 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:30.898313    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg" podUID="b0984e71-aec3-4372-b101-ecf16c20488a" containerName="istio-proxy" containerID="cri-o://9c6c6dc399f3c3953584bd1a943ab1c011c5dcf7bc50dba4c8b29ed7884c1f14" gracePeriod=30
Sep 11 14:35:31.750387 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:31.750354    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-98be89/noten-model-98be89-kserve-5fcf5695ff-88j24"]
Sep 11 14:35:31.753678 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:31.753653    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-98be89/noten-model-98be89-kserve-5fcf5695ff-88j24"
Sep 11 14:35:31.759431 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:31.759405    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-98be89\"/\"default-dockercfg-wbt46\""
Sep 11 14:35:31.759556 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:31.759433    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-98be89\"/\"noten-model-98be89-kserve-self-signed-certs\""
Sep 11 14:35:31.759556 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:31.759461    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-98be89\"/\"kube-root-ca.crt\""
Sep 11 14:35:31.759556 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:31.759536    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-98be89\"/\"openshift-service-ca.crt\""
Sep 11 14:35:31.782241 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:31.782211    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-98be89/noten-model-98be89-kserve-5fcf5695ff-88j24"]
Sep 11 14:35:31.867087 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:31.867051    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwqjq\" (UniqueName: \"kubernetes.io/projected/e1ee2609-c0e7-4161-812c-820087413336-kube-api-access-cwqjq\") pod \"noten-model-98be89-kserve-5fcf5695ff-88j24\" (UID: \"e1ee2609-c0e7-4161-812c-820087413336\") " pod="ai-tenant-e2e-noten-98be89/noten-model-98be89-kserve-5fcf5695ff-88j24"
Sep 11 14:35:31.867265 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:31.867113    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1ee2609-c0e7-4161-812c-820087413336-home\") pod \"noten-model-98be89-kserve-5fcf5695ff-88j24\" (UID: \"e1ee2609-c0e7-4161-812c-820087413336\") " pod="ai-tenant-e2e-noten-98be89/noten-model-98be89-kserve-5fcf5695ff-88j24"
Sep 11 14:35:31.867265 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:31.867176    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1ee2609-c0e7-4161-812c-820087413336-model-cache\") pod \"noten-model-98be89-kserve-5fcf5695ff-88j24\" (UID: \"e1ee2609-c0e7-4161-812c-820087413336\") " pod="ai-tenant-e2e-noten-98be89/noten-model-98be89-kserve-5fcf5695ff-88j24"
Sep 11 14:35:31.867265 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:31.867244    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1ee2609-c0e7-4161-812c-820087413336-tmp-dir\") pod \"noten-model-98be89-kserve-5fcf5695ff-88j24\" (UID: \"e1ee2609-c0e7-4161-812c-820087413336\") " pod="ai-tenant-e2e-noten-98be89/noten-model-98be89-kserve-5fcf5695ff-88j24"
Sep 11 14:35:31.867399 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:31.867280    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1ee2609-c0e7-4161-812c-820087413336-tls-certs\") pod \"noten-model-98be89-kserve-5fcf5695ff-88j24\" (UID: \"e1ee2609-c0e7-4161-812c-820087413336\") " pod="ai-tenant-e2e-noten-98be89/noten-model-98be89-kserve-5fcf5695ff-88j24"
Sep 11 14:35:31.867399 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:31.867313    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1ee2609-c0e7-4161-812c-820087413336-dshm\") pod \"noten-model-98be89-kserve-5fcf5695ff-88j24\" (UID: \"e1ee2609-c0e7-4161-812c-820087413336\") " pod="ai-tenant-e2e-noten-98be89/noten-model-98be89-kserve-5fcf5695ff-88j24"
Sep 11 14:35:31.968373 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:31.968333    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1ee2609-c0e7-4161-812c-820087413336-tmp-dir\") pod \"noten-model-98be89-kserve-5fcf5695ff-88j24\" (UID: \"e1ee2609-c0e7-4161-812c-820087413336\") " pod="ai-tenant-e2e-noten-98be89/noten-model-98be89-kserve-5fcf5695ff-88j24"
Sep 11 14:35:31.968858 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:31.968396    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1ee2609-c0e7-4161-812c-820087413336-tls-certs\") pod \"noten-model-98be89-kserve-5fcf5695ff-88j24\" (UID: \"e1ee2609-c0e7-4161-812c-820087413336\") " pod="ai-tenant-e2e-noten-98be89/noten-model-98be89-kserve-5fcf5695ff-88j24"
Sep 11 14:35:31.968858 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:31.968443    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1ee2609-c0e7-4161-812c-820087413336-dshm\") pod \"noten-model-98be89-kserve-5fcf5695ff-88j24\" (UID: \"e1ee2609-c0e7-4161-812c-820087413336\") " pod="ai-tenant-e2e-noten-98be89/noten-model-98be89-kserve-5fcf5695ff-88j24"
Sep 11 14:35:31.968858 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:31.968510    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwqjq\" (UniqueName: \"kubernetes.io/projected/e1ee2609-c0e7-4161-812c-820087413336-kube-api-access-cwqjq\") pod \"noten-model-98be89-kserve-5fcf5695ff-88j24\" (UID: \"e1ee2609-c0e7-4161-812c-820087413336\") " pod="ai-tenant-e2e-noten-98be89/noten-model-98be89-kserve-5fcf5695ff-88j24"
Sep 11 14:35:31.968858 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:31.968572    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1ee2609-c0e7-4161-812c-820087413336-home\") pod \"noten-model-98be89-kserve-5fcf5695ff-88j24\" (UID: \"e1ee2609-c0e7-4161-812c-820087413336\") " pod="ai-tenant-e2e-noten-98be89/noten-model-98be89-kserve-5fcf5695ff-88j24"
Sep 11 14:35:31.968858 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:31.968599    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1ee2609-c0e7-4161-812c-820087413336-model-cache\") pod \"noten-model-98be89-kserve-5fcf5695ff-88j24\" (UID: \"e1ee2609-c0e7-4161-812c-820087413336\") " pod="ai-tenant-e2e-noten-98be89/noten-model-98be89-kserve-5fcf5695ff-88j24"
Sep 11 14:35:31.968858 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:31.968821    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1ee2609-c0e7-4161-812c-820087413336-tmp-dir\") pod \"noten-model-98be89-kserve-5fcf5695ff-88j24\" (UID: \"e1ee2609-c0e7-4161-812c-820087413336\") " pod="ai-tenant-e2e-noten-98be89/noten-model-98be89-kserve-5fcf5695ff-88j24"
Sep 11 14:35:31.969217 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:31.968914    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1ee2609-c0e7-4161-812c-820087413336-home\") pod \"noten-model-98be89-kserve-5fcf5695ff-88j24\" (UID: \"e1ee2609-c0e7-4161-812c-820087413336\") " pod="ai-tenant-e2e-noten-98be89/noten-model-98be89-kserve-5fcf5695ff-88j24"
Sep 11 14:35:31.969217 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:31.969004    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1ee2609-c0e7-4161-812c-820087413336-model-cache\") pod \"noten-model-98be89-kserve-5fcf5695ff-88j24\" (UID: \"e1ee2609-c0e7-4161-812c-820087413336\") " pod="ai-tenant-e2e-noten-98be89/noten-model-98be89-kserve-5fcf5695ff-88j24"
Sep 11 14:35:31.970605 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:31.970586    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1ee2609-c0e7-4161-812c-820087413336-dshm\") pod \"noten-model-98be89-kserve-5fcf5695ff-88j24\" (UID: \"e1ee2609-c0e7-4161-812c-820087413336\") " pod="ai-tenant-e2e-noten-98be89/noten-model-98be89-kserve-5fcf5695ff-88j24"
Sep 11 14:35:31.970925 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:31.970907    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1ee2609-c0e7-4161-812c-820087413336-tls-certs\") pod \"noten-model-98be89-kserve-5fcf5695ff-88j24\" (UID: \"e1ee2609-c0e7-4161-812c-820087413336\") " pod="ai-tenant-e2e-noten-98be89/noten-model-98be89-kserve-5fcf5695ff-88j24"
Sep 11 14:35:31.979944 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:31.979924    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwqjq\" (UniqueName: \"kubernetes.io/projected/e1ee2609-c0e7-4161-812c-820087413336-kube-api-access-cwqjq\") pod \"noten-model-98be89-kserve-5fcf5695ff-88j24\" (UID: \"e1ee2609-c0e7-4161-812c-820087413336\") " pod="ai-tenant-e2e-noten-98be89/noten-model-98be89-kserve-5fcf5695ff-88j24"
Sep 11 14:35:32.063206 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:32.063114    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-98be89/noten-model-98be89-kserve-5fcf5695ff-88j24"
Sep 11 14:35:32.356260 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:32.356220    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-98be89/noten-model-98be89-kserve-5fcf5695ff-88j24"]
Sep 11 14:35:32.357262 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:35:32.357240    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1ee2609_c0e7_4161_812c_820087413336.slice/crio-f37b7c42b1725be118ad1079489ec3d3ba4f6b25a029d80f982cabe3c1d125cc WatchSource:0}: Error finding container f37b7c42b1725be118ad1079489ec3d3ba4f6b25a029d80f982cabe3c1d125cc: Status 404 returned error can't find the container with id f37b7c42b1725be118ad1079489ec3d3ba4f6b25a029d80f982cabe3c1d125cc
Sep 11 14:35:33.056546 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:33.056509    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-98be89/noten-model-98be89-kserve-5fcf5695ff-88j24" event={"ID":"e1ee2609-c0e7-4161-812c-820087413336","Type":"ContainerStarted","Data":"dd1f43bf07b522ec95e327b238bba3592840ed55a04e1527fd299aa28cb908a8"}
Sep 11 14:35:33.056546 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:33.056554    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-98be89/noten-model-98be89-kserve-5fcf5695ff-88j24" event={"ID":"e1ee2609-c0e7-4161-812c-820087413336","Type":"ContainerStarted","Data":"f37b7c42b1725be118ad1079489ec3d3ba4f6b25a029d80f982cabe3c1d125cc"}
Sep 11 14:35:33.089744 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:33.089693    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-98be89/noten-model-98be89-kserve-5fcf5695ff-88j24" podStartSLOduration=2.089677506 podStartE2EDuration="2.089677506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:35:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:35:33.089091955 +0000 UTC m=+1236.239642019" watchObservedRunningTime="2026-09-11 14:35:33.089677506 +0000 UTC m=+1236.240227570"
Sep 11 14:35:33.604104 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:33.604073    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-98be89/noten-model-98be89-kserve-5fcf5695ff-88j24"]
Sep 11 14:35:33.772401 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:33.772360    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4"]
Sep 11 14:35:33.772582 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:33.772541    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4" podUID="849d7667-fb78-4454-aac1-662b2c962367" containerName="istio-proxy" containerID="cri-o://8dbecd4c45b5c7d363582bb0f10f1a4406357efe2044f57b268e0e8618519ba2" gracePeriod=30
Sep 11 14:35:34.088158 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:35:34.087793    2581 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-98be89/noten-model-98be89-kserve-self-signed-certs: secret "noten-model-98be89-kserve-self-signed-certs" not found
Sep 11 14:35:34.088158 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:35:34.087889    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1ee2609-c0e7-4161-812c-820087413336-tls-certs podName:e1ee2609-c0e7-4161-812c-820087413336 nodeName:}" failed. No retries permitted until 2026-09-11 14:35:34.587867633 +0000 UTC m=+1237.738417675 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/e1ee2609-c0e7-4161-812c-820087413336-tls-certs") pod "noten-model-98be89-kserve-5fcf5695ff-88j24" (UID: "e1ee2609-c0e7-4161-812c-820087413336") : secret "noten-model-98be89-kserve-self-signed-certs" not found
Sep 11 14:35:34.594385 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:35:34.594352    2581 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-98be89/noten-model-98be89-kserve-self-signed-certs: secret "noten-model-98be89-kserve-self-signed-certs" not found
Sep 11 14:35:34.594576 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:35:34.594436    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1ee2609-c0e7-4161-812c-820087413336-tls-certs podName:e1ee2609-c0e7-4161-812c-820087413336 nodeName:}" failed. No retries permitted until 2026-09-11 14:35:35.594418691 +0000 UTC m=+1238.744968736 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/e1ee2609-c0e7-4161-812c-820087413336-tls-certs") pod "noten-model-98be89-kserve-5fcf5695ff-88j24" (UID: "e1ee2609-c0e7-4161-812c-820087413336") : secret "noten-model-98be89-kserve-self-signed-certs" not found
Sep 11 14:35:35.603578 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:35:35.603536    2581 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-98be89/noten-model-98be89-kserve-self-signed-certs: secret "noten-model-98be89-kserve-self-signed-certs" not found
Sep 11 14:35:35.604040 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:35:35.603617    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1ee2609-c0e7-4161-812c-820087413336-tls-certs podName:e1ee2609-c0e7-4161-812c-820087413336 nodeName:}" failed. No retries permitted until 2026-09-11 14:35:37.603603401 +0000 UTC m=+1240.754153443 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/e1ee2609-c0e7-4161-812c-820087413336-tls-certs") pod "noten-model-98be89-kserve-5fcf5695ff-88j24" (UID: "e1ee2609-c0e7-4161-812c-820087413336") : secret "noten-model-98be89-kserve-self-signed-certs" not found
Sep 11 14:35:36.080088 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:36.080051    2581 generic.go:356] "Generic (PLEG): container finished" podID="b0984e71-aec3-4372-b101-ecf16c20488a" containerID="9c6c6dc399f3c3953584bd1a943ab1c011c5dcf7bc50dba4c8b29ed7884c1f14" exitCode=0
Sep 11 14:35:36.080234 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:36.080116    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg" event={"ID":"b0984e71-aec3-4372-b101-ecf16c20488a","Type":"ContainerDied","Data":"9c6c6dc399f3c3953584bd1a943ab1c011c5dcf7bc50dba4c8b29ed7884c1f14"}
Sep 11 14:35:36.080234 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:36.080185    2581 scope.go:117] "RemoveContainer" containerID="b21e822f719dcfcd695ac024cd3876796f8f18297affd36803174cd42777ee4f"
Sep 11 14:35:36.144423 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:36.144398    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg"
Sep 11 14:35:36.209322 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:36.209239    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0984e71-aec3-4372-b101-ecf16c20488a-istio-token\") pod \"b0984e71-aec3-4372-b101-ecf16c20488a\" (UID: \"b0984e71-aec3-4372-b101-ecf16c20488a\") "
Sep 11 14:35:36.209322 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:36.209286    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0984e71-aec3-4372-b101-ecf16c20488a-istiod-ca-cert\") pod \"b0984e71-aec3-4372-b101-ecf16c20488a\" (UID: \"b0984e71-aec3-4372-b101-ecf16c20488a\") "
Sep 11 14:35:36.209491 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:36.209327    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wmkpn\" (UniqueName: \"kubernetes.io/projected/b0984e71-aec3-4372-b101-ecf16c20488a-kube-api-access-wmkpn\") pod \"b0984e71-aec3-4372-b101-ecf16c20488a\" (UID: \"b0984e71-aec3-4372-b101-ecf16c20488a\") "
Sep 11 14:35:36.209491 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:36.209432    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0984e71-aec3-4372-b101-ecf16c20488a-istio-podinfo\") pod \"b0984e71-aec3-4372-b101-ecf16c20488a\" (UID: \"b0984e71-aec3-4372-b101-ecf16c20488a\") "
Sep 11 14:35:36.209617 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:36.209491    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0984e71-aec3-4372-b101-ecf16c20488a-istio-envoy\") pod \"b0984e71-aec3-4372-b101-ecf16c20488a\" (UID: \"b0984e71-aec3-4372-b101-ecf16c20488a\") "
Sep 11 14:35:36.209617 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:36.209519    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0984e71-aec3-4372-b101-ecf16c20488a-credential-socket\") pod \"b0984e71-aec3-4372-b101-ecf16c20488a\" (UID: \"b0984e71-aec3-4372-b101-ecf16c20488a\") "
Sep 11 14:35:36.209617 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:36.209544    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0984e71-aec3-4372-b101-ecf16c20488a-workload-socket\") pod \"b0984e71-aec3-4372-b101-ecf16c20488a\" (UID: \"b0984e71-aec3-4372-b101-ecf16c20488a\") "
Sep 11 14:35:36.209617 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:36.209600    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0984e71-aec3-4372-b101-ecf16c20488a-workload-certs\") pod \"b0984e71-aec3-4372-b101-ecf16c20488a\" (UID: \"b0984e71-aec3-4372-b101-ecf16c20488a\") "
Sep 11 14:35:36.209827 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:36.209645    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0984e71-aec3-4372-b101-ecf16c20488a-istio-data\") pod \"b0984e71-aec3-4372-b101-ecf16c20488a\" (UID: \"b0984e71-aec3-4372-b101-ecf16c20488a\") "
Sep 11 14:35:36.209827 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:36.209725    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0984e71-aec3-4372-b101-ecf16c20488a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b0984e71-aec3-4372-b101-ecf16c20488a" (UID: "b0984e71-aec3-4372-b101-ecf16c20488a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:35:36.210095 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:36.209953    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0984e71-aec3-4372-b101-ecf16c20488a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b0984e71-aec3-4372-b101-ecf16c20488a" (UID: "b0984e71-aec3-4372-b101-ecf16c20488a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:35:36.210095 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:36.210064    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0984e71-aec3-4372-b101-ecf16c20488a-istio-data" (OuterVolumeSpecName: "istio-data") pod "b0984e71-aec3-4372-b101-ecf16c20488a" (UID: "b0984e71-aec3-4372-b101-ecf16c20488a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:35:36.210095 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:36.210083    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0984e71-aec3-4372-b101-ecf16c20488a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b0984e71-aec3-4372-b101-ecf16c20488a" (UID: "b0984e71-aec3-4372-b101-ecf16c20488a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:35:36.210369 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:36.210101    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0984e71-aec3-4372-b101-ecf16c20488a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b0984e71-aec3-4372-b101-ecf16c20488a" (UID: "b0984e71-aec3-4372-b101-ecf16c20488a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:35:36.210369 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:36.210173    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0984e71-aec3-4372-b101-ecf16c20488a-istio-data\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:35:36.210369 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:36.210192    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0984e71-aec3-4372-b101-ecf16c20488a-istiod-ca-cert\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:35:36.210369 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:36.210206    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0984e71-aec3-4372-b101-ecf16c20488a-workload-socket\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:35:36.210369 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:36.210221    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0984e71-aec3-4372-b101-ecf16c20488a-workload-certs\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:35:36.211570 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:36.211547    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0984e71-aec3-4372-b101-ecf16c20488a-istio-token" (OuterVolumeSpecName: "istio-token") pod "b0984e71-aec3-4372-b101-ecf16c20488a" (UID: "b0984e71-aec3-4372-b101-ecf16c20488a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:35:36.211570 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:36.211557    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0984e71-aec3-4372-b101-ecf16c20488a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b0984e71-aec3-4372-b101-ecf16c20488a" (UID: "b0984e71-aec3-4372-b101-ecf16c20488a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 14:35:36.211719 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:36.211686    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0984e71-aec3-4372-b101-ecf16c20488a-kube-api-access-wmkpn" (OuterVolumeSpecName: "kube-api-access-wmkpn") pod "b0984e71-aec3-4372-b101-ecf16c20488a" (UID: "b0984e71-aec3-4372-b101-ecf16c20488a"). InnerVolumeSpecName "kube-api-access-wmkpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:35:36.211759 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:36.211723    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0984e71-aec3-4372-b101-ecf16c20488a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b0984e71-aec3-4372-b101-ecf16c20488a" (UID: "b0984e71-aec3-4372-b101-ecf16c20488a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:35:36.311690 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:36.311649    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmkpn\" (UniqueName: \"kubernetes.io/projected/b0984e71-aec3-4372-b101-ecf16c20488a-kube-api-access-wmkpn\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:35:36.311690 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:36.311684    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0984e71-aec3-4372-b101-ecf16c20488a-istio-podinfo\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:35:36.311690 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:36.311693    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0984e71-aec3-4372-b101-ecf16c20488a-istio-envoy\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:35:36.311925 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:36.311702    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0984e71-aec3-4372-b101-ecf16c20488a-credential-socket\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:35:36.311925 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:36.311711    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0984e71-aec3-4372-b101-ecf16c20488a-istio-token\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:35:37.087620 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:37.087589    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg"
Sep 11 14:35:37.088119 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:37.087627    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg" event={"ID":"b0984e71-aec3-4372-b101-ecf16c20488a","Type":"ContainerDied","Data":"ed1916cfbd0f370d630ea29997019a1722f63c509b86ad6a15ff749219e52be2"}
Sep 11 14:35:37.088119 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:37.087683    2581 scope.go:117] "RemoveContainer" containerID="9c6c6dc399f3c3953584bd1a943ab1c011c5dcf7bc50dba4c8b29ed7884c1f14"
Sep 11 14:35:37.115117 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:37.115087    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg"]
Sep 11 14:35:37.129088 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:37.129059    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e85961-openshift-default-7b55cbd9b6-5c4dg"]
Sep 11 14:35:37.500538 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:37.500507    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0984e71-aec3-4372-b101-ecf16c20488a" path="/var/lib/kubelet/pods/b0984e71-aec3-4372-b101-ecf16c20488a/volumes"
Sep 11 14:35:37.625492 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:35:37.625454    2581 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-98be89/noten-model-98be89-kserve-self-signed-certs: secret "noten-model-98be89-kserve-self-signed-certs" not found
Sep 11 14:35:37.625678 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:35:37.625532    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1ee2609-c0e7-4161-812c-820087413336-tls-certs podName:e1ee2609-c0e7-4161-812c-820087413336 nodeName:}" failed. No retries permitted until 2026-09-11 14:35:41.625514605 +0000 UTC m=+1244.776064667 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/e1ee2609-c0e7-4161-812c-820087413336-tls-certs") pod "noten-model-98be89-kserve-5fcf5695ff-88j24" (UID: "e1ee2609-c0e7-4161-812c-820087413336") : secret "noten-model-98be89-kserve-self-signed-certs" not found
Sep 11 14:35:39.025939 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:39.025914    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4"
Sep 11 14:35:39.106828 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:39.106732    2581 generic.go:356] "Generic (PLEG): container finished" podID="849d7667-fb78-4454-aac1-662b2c962367" containerID="8dbecd4c45b5c7d363582bb0f10f1a4406357efe2044f57b268e0e8618519ba2" exitCode=0
Sep 11 14:35:39.106828 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:39.106779    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4" event={"ID":"849d7667-fb78-4454-aac1-662b2c962367","Type":"ContainerDied","Data":"8dbecd4c45b5c7d363582bb0f10f1a4406357efe2044f57b268e0e8618519ba2"}
Sep 11 14:35:39.106828 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:39.106803    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4"
Sep 11 14:35:39.107144 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:39.106856    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4" event={"ID":"849d7667-fb78-4454-aac1-662b2c962367","Type":"ContainerDied","Data":"7316e1028fc7d42b0d2b9d3228da39cb970ff1e9e6c0fb820ea3630c25f4846e"}
Sep 11 14:35:39.107144 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:39.106881    2581 scope.go:117] "RemoveContainer" containerID="8dbecd4c45b5c7d363582bb0f10f1a4406357efe2044f57b268e0e8618519ba2"
Sep 11 14:35:39.115539 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:39.115524    2581 scope.go:117] "RemoveContainer" containerID="8dbecd4c45b5c7d363582bb0f10f1a4406357efe2044f57b268e0e8618519ba2"
Sep 11 14:35:39.115793 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:35:39.115775    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dbecd4c45b5c7d363582bb0f10f1a4406357efe2044f57b268e0e8618519ba2\": container with ID starting with 8dbecd4c45b5c7d363582bb0f10f1a4406357efe2044f57b268e0e8618519ba2 not found: ID does not exist" containerID="8dbecd4c45b5c7d363582bb0f10f1a4406357efe2044f57b268e0e8618519ba2"
Sep 11 14:35:39.115842 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:39.115801    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dbecd4c45b5c7d363582bb0f10f1a4406357efe2044f57b268e0e8618519ba2"} err="failed to get container status \"8dbecd4c45b5c7d363582bb0f10f1a4406357efe2044f57b268e0e8618519ba2\": rpc error: code = NotFound desc = could not find container \"8dbecd4c45b5c7d363582bb0f10f1a4406357efe2044f57b268e0e8618519ba2\": container with ID starting with 8dbecd4c45b5c7d363582bb0f10f1a4406357efe2044f57b268e0e8618519ba2 not found: ID does not exist"
Sep 11 14:35:39.141275 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:39.141251    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/849d7667-fb78-4454-aac1-662b2c962367-istio-envoy\") pod \"849d7667-fb78-4454-aac1-662b2c962367\" (UID: \"849d7667-fb78-4454-aac1-662b2c962367\") "
Sep 11 14:35:39.141376 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:39.141282    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-95554\" (UniqueName: \"kubernetes.io/projected/849d7667-fb78-4454-aac1-662b2c962367-kube-api-access-95554\") pod \"849d7667-fb78-4454-aac1-662b2c962367\" (UID: \"849d7667-fb78-4454-aac1-662b2c962367\") "
Sep 11 14:35:39.141376 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:39.141304    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/849d7667-fb78-4454-aac1-662b2c962367-workload-socket\") pod \"849d7667-fb78-4454-aac1-662b2c962367\" (UID: \"849d7667-fb78-4454-aac1-662b2c962367\") "
Sep 11 14:35:39.141376 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:39.141354    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/849d7667-fb78-4454-aac1-662b2c962367-credential-socket\") pod \"849d7667-fb78-4454-aac1-662b2c962367\" (UID: \"849d7667-fb78-4454-aac1-662b2c962367\") "
Sep 11 14:35:39.141513 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:39.141402    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/849d7667-fb78-4454-aac1-662b2c962367-istio-podinfo\") pod \"849d7667-fb78-4454-aac1-662b2c962367\" (UID: \"849d7667-fb78-4454-aac1-662b2c962367\") "
Sep 11 14:35:39.141513 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:39.141432    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/849d7667-fb78-4454-aac1-662b2c962367-istiod-ca-cert\") pod \"849d7667-fb78-4454-aac1-662b2c962367\" (UID: \"849d7667-fb78-4454-aac1-662b2c962367\") "
Sep 11 14:35:39.141513 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:39.141460    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/849d7667-fb78-4454-aac1-662b2c962367-istio-token\") pod \"849d7667-fb78-4454-aac1-662b2c962367\" (UID: \"849d7667-fb78-4454-aac1-662b2c962367\") "
Sep 11 14:35:39.141513 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:39.141508    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/849d7667-fb78-4454-aac1-662b2c962367-workload-certs\") pod \"849d7667-fb78-4454-aac1-662b2c962367\" (UID: \"849d7667-fb78-4454-aac1-662b2c962367\") "
Sep 11 14:35:39.141711 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:39.141567    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/849d7667-fb78-4454-aac1-662b2c962367-istio-data\") pod \"849d7667-fb78-4454-aac1-662b2c962367\" (UID: \"849d7667-fb78-4454-aac1-662b2c962367\") "
Sep 11 14:35:39.141711 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:39.141605    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/849d7667-fb78-4454-aac1-662b2c962367-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "849d7667-fb78-4454-aac1-662b2c962367" (UID: "849d7667-fb78-4454-aac1-662b2c962367"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:35:39.141711 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:39.141656    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/849d7667-fb78-4454-aac1-662b2c962367-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "849d7667-fb78-4454-aac1-662b2c962367" (UID: "849d7667-fb78-4454-aac1-662b2c962367"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:35:39.141881 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:39.141822    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/849d7667-fb78-4454-aac1-662b2c962367-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "849d7667-fb78-4454-aac1-662b2c962367" (UID: "849d7667-fb78-4454-aac1-662b2c962367"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:35:39.141881 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:39.141848    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/849d7667-fb78-4454-aac1-662b2c962367-istio-data" (OuterVolumeSpecName: "istio-data") pod "849d7667-fb78-4454-aac1-662b2c962367" (UID: "849d7667-fb78-4454-aac1-662b2c962367"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:35:39.142014 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:39.141865    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/849d7667-fb78-4454-aac1-662b2c962367-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "849d7667-fb78-4454-aac1-662b2c962367" (UID: "849d7667-fb78-4454-aac1-662b2c962367"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:35:39.142083 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:39.142053    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/849d7667-fb78-4454-aac1-662b2c962367-workload-certs\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:35:39.142083 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:39.142073    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/849d7667-fb78-4454-aac1-662b2c962367-istio-data\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:35:39.142186 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:39.142085    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/849d7667-fb78-4454-aac1-662b2c962367-workload-socket\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:35:39.142186 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:39.142098    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/849d7667-fb78-4454-aac1-662b2c962367-credential-socket\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:35:39.142186 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:39.142113    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/849d7667-fb78-4454-aac1-662b2c962367-istiod-ca-cert\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:35:39.143575 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:39.143551    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/849d7667-fb78-4454-aac1-662b2c962367-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "849d7667-fb78-4454-aac1-662b2c962367" (UID: "849d7667-fb78-4454-aac1-662b2c962367"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 14:35:39.143682 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:39.143577    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/849d7667-fb78-4454-aac1-662b2c962367-kube-api-access-95554" (OuterVolumeSpecName: "kube-api-access-95554") pod "849d7667-fb78-4454-aac1-662b2c962367" (UID: "849d7667-fb78-4454-aac1-662b2c962367"). InnerVolumeSpecName "kube-api-access-95554". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:35:39.143682 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:39.143578    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/849d7667-fb78-4454-aac1-662b2c962367-istio-token" (OuterVolumeSpecName: "istio-token") pod "849d7667-fb78-4454-aac1-662b2c962367" (UID: "849d7667-fb78-4454-aac1-662b2c962367"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:35:39.143682 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:39.143657    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/849d7667-fb78-4454-aac1-662b2c962367-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "849d7667-fb78-4454-aac1-662b2c962367" (UID: "849d7667-fb78-4454-aac1-662b2c962367"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:35:39.243388 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:39.243356    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/849d7667-fb78-4454-aac1-662b2c962367-istio-podinfo\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:35:39.243388 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:39.243384    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/849d7667-fb78-4454-aac1-662b2c962367-istio-token\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:35:39.243388 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:39.243393    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/849d7667-fb78-4454-aac1-662b2c962367-istio-envoy\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:35:39.243626 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:39.243401    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95554\" (UniqueName: \"kubernetes.io/projected/849d7667-fb78-4454-aac1-662b2c962367-kube-api-access-95554\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:35:39.433246 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:39.433217    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4"]
Sep 11 14:35:39.438183 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:39.438157    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-98be89-openshift-default-6cfdb65d46-cz5f4"]
Sep 11 14:35:39.501006 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:39.500963    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="849d7667-fb78-4454-aac1-662b2c962367" path="/var/lib/kubelet/pods/849d7667-fb78-4454-aac1-662b2c962367/volumes"
Sep 11 14:35:41.668920 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:35:41.668879    2581 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-98be89/noten-model-98be89-kserve-self-signed-certs: secret "noten-model-98be89-kserve-self-signed-certs" not found
Sep 11 14:35:41.669364 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:35:41.668995    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1ee2609-c0e7-4161-812c-820087413336-tls-certs podName:e1ee2609-c0e7-4161-812c-820087413336 nodeName:}" failed. No retries permitted until 2026-09-11 14:35:49.668955117 +0000 UTC m=+1252.819505159 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/e1ee2609-c0e7-4161-812c-820087413336-tls-certs") pod "noten-model-98be89-kserve-5fcf5695ff-88j24" (UID: "e1ee2609-c0e7-4161-812c-820087413336") : secret "noten-model-98be89-kserve-self-signed-certs" not found
Sep 11 14:35:42.063399 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:42.063365    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-98be89/noten-model-98be89-kserve-5fcf5695ff-88j24"
Sep 11 14:35:49.756128 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:35:49.756041    2581 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-98be89/noten-model-98be89-kserve-self-signed-certs: secret "noten-model-98be89-kserve-self-signed-certs" not found
Sep 11 14:35:49.756487 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:35:49.756140    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1ee2609-c0e7-4161-812c-820087413336-tls-certs podName:e1ee2609-c0e7-4161-812c-820087413336 nodeName:}" failed. No retries permitted until 2026-09-11 14:36:05.756101404 +0000 UTC m=+1268.906651460 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/e1ee2609-c0e7-4161-812c-820087413336-tls-certs") pod "noten-model-98be89-kserve-5fcf5695ff-88j24" (UID: "e1ee2609-c0e7-4161-812c-820087413336") : secret "noten-model-98be89-kserve-self-signed-certs" not found
Sep 11 14:35:50.087859 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:50.087770    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-98be89/noten-model-98be89-kserve-5fcf5695ff-88j24" podUID="e1ee2609-c0e7-4161-812c-820087413336" containerName="main" containerID="cri-o://dd1f43bf07b522ec95e327b238bba3592840ed55a04e1527fd299aa28cb908a8" gracePeriod=45
Sep 11 14:35:50.334864 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:50.334840    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-98be89/noten-model-98be89-kserve-5fcf5695ff-88j24"
Sep 11 14:35:50.463157 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:50.463071    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1ee2609-c0e7-4161-812c-820087413336-model-cache\") pod \"e1ee2609-c0e7-4161-812c-820087413336\" (UID: \"e1ee2609-c0e7-4161-812c-820087413336\") "
Sep 11 14:35:50.463157 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:50.463138    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1ee2609-c0e7-4161-812c-820087413336-tmp-dir\") pod \"e1ee2609-c0e7-4161-812c-820087413336\" (UID: \"e1ee2609-c0e7-4161-812c-820087413336\") "
Sep 11 14:35:50.463410 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:50.463195    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cwqjq\" (UniqueName: \"kubernetes.io/projected/e1ee2609-c0e7-4161-812c-820087413336-kube-api-access-cwqjq\") pod \"e1ee2609-c0e7-4161-812c-820087413336\" (UID: \"e1ee2609-c0e7-4161-812c-820087413336\") "
Sep 11 14:35:50.463410 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:50.463250    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1ee2609-c0e7-4161-812c-820087413336-dshm\") pod \"e1ee2609-c0e7-4161-812c-820087413336\" (UID: \"e1ee2609-c0e7-4161-812c-820087413336\") "
Sep 11 14:35:50.463410 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:50.463304    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1ee2609-c0e7-4161-812c-820087413336-tls-certs\") pod \"e1ee2609-c0e7-4161-812c-820087413336\" (UID: \"e1ee2609-c0e7-4161-812c-820087413336\") "
Sep 11 14:35:50.463410 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:50.463392    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1ee2609-c0e7-4161-812c-820087413336-home\") pod \"e1ee2609-c0e7-4161-812c-820087413336\" (UID: \"e1ee2609-c0e7-4161-812c-820087413336\") "
Sep 11 14:35:50.463410 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:50.463380    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1ee2609-c0e7-4161-812c-820087413336-model-cache" (OuterVolumeSpecName: "model-cache") pod "e1ee2609-c0e7-4161-812c-820087413336" (UID: "e1ee2609-c0e7-4161-812c-820087413336"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:35:50.463665 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:50.463519    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1ee2609-c0e7-4161-812c-820087413336-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e1ee2609-c0e7-4161-812c-820087413336" (UID: "e1ee2609-c0e7-4161-812c-820087413336"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:35:50.463753 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:50.463727    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1ee2609-c0e7-4161-812c-820087413336-home" (OuterVolumeSpecName: "home") pod "e1ee2609-c0e7-4161-812c-820087413336" (UID: "e1ee2609-c0e7-4161-812c-820087413336"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:35:50.463815 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:50.463754    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1ee2609-c0e7-4161-812c-820087413336-model-cache\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:35:50.463815 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:50.463775    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1ee2609-c0e7-4161-812c-820087413336-tmp-dir\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:35:50.465899 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:50.465870    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1ee2609-c0e7-4161-812c-820087413336-dshm" (OuterVolumeSpecName: "dshm") pod "e1ee2609-c0e7-4161-812c-820087413336" (UID: "e1ee2609-c0e7-4161-812c-820087413336"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:35:50.466039 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:50.465930    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1ee2609-c0e7-4161-812c-820087413336-kube-api-access-cwqjq" (OuterVolumeSpecName: "kube-api-access-cwqjq") pod "e1ee2609-c0e7-4161-812c-820087413336" (UID: "e1ee2609-c0e7-4161-812c-820087413336"). InnerVolumeSpecName "kube-api-access-cwqjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:35:50.466039 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:50.465985    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1ee2609-c0e7-4161-812c-820087413336-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e1ee2609-c0e7-4161-812c-820087413336" (UID: "e1ee2609-c0e7-4161-812c-820087413336"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:35:50.564901 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:50.564848    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1ee2609-c0e7-4161-812c-820087413336-tls-certs\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:35:50.564901 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:50.564894    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1ee2609-c0e7-4161-812c-820087413336-home\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:35:50.564901 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:50.564907    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwqjq\" (UniqueName: \"kubernetes.io/projected/e1ee2609-c0e7-4161-812c-820087413336-kube-api-access-cwqjq\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:35:50.565260 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:50.564920    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1ee2609-c0e7-4161-812c-820087413336-dshm\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:35:51.104141 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:51.104108    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt"]
Sep 11 14:35:51.104595 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:51.104322    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt" podUID="8bde0890-f752-4b4b-976f-804512c66e7c" containerName="istio-proxy" containerID="cri-o://730dd42bf8c9fe1a78019eab32498867f0d998cf1635563316dc9c74c9e01f0a" gracePeriod=30
Sep 11 14:35:51.173439 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:51.173400    2581 generic.go:356] "Generic (PLEG): container finished" podID="e1ee2609-c0e7-4161-812c-820087413336" containerID="dd1f43bf07b522ec95e327b238bba3592840ed55a04e1527fd299aa28cb908a8" exitCode=0
Sep 11 14:35:51.173625 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:51.173442    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-98be89/noten-model-98be89-kserve-5fcf5695ff-88j24"
Sep 11 14:35:51.173625 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:51.173451    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-98be89/noten-model-98be89-kserve-5fcf5695ff-88j24" event={"ID":"e1ee2609-c0e7-4161-812c-820087413336","Type":"ContainerDied","Data":"dd1f43bf07b522ec95e327b238bba3592840ed55a04e1527fd299aa28cb908a8"}
Sep 11 14:35:51.173625 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:51.173487    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-98be89/noten-model-98be89-kserve-5fcf5695ff-88j24" event={"ID":"e1ee2609-c0e7-4161-812c-820087413336","Type":"ContainerDied","Data":"f37b7c42b1725be118ad1079489ec3d3ba4f6b25a029d80f982cabe3c1d125cc"}
Sep 11 14:35:51.173625 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:51.173506    2581 scope.go:117] "RemoveContainer" containerID="dd1f43bf07b522ec95e327b238bba3592840ed55a04e1527fd299aa28cb908a8"
Sep 11 14:35:51.183938 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:51.183796    2581 scope.go:117] "RemoveContainer" containerID="dd1f43bf07b522ec95e327b238bba3592840ed55a04e1527fd299aa28cb908a8"
Sep 11 14:35:51.184306 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:35:51.184267    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd1f43bf07b522ec95e327b238bba3592840ed55a04e1527fd299aa28cb908a8\": container with ID starting with dd1f43bf07b522ec95e327b238bba3592840ed55a04e1527fd299aa28cb908a8 not found: ID does not exist" containerID="dd1f43bf07b522ec95e327b238bba3592840ed55a04e1527fd299aa28cb908a8"
Sep 11 14:35:51.184416 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:51.184311    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd1f43bf07b522ec95e327b238bba3592840ed55a04e1527fd299aa28cb908a8"} err="failed to get container status \"dd1f43bf07b522ec95e327b238bba3592840ed55a04e1527fd299aa28cb908a8\": rpc error: code = NotFound desc = could not find container \"dd1f43bf07b522ec95e327b238bba3592840ed55a04e1527fd299aa28cb908a8\": container with ID starting with dd1f43bf07b522ec95e327b238bba3592840ed55a04e1527fd299aa28cb908a8 not found: ID does not exist"
Sep 11 14:35:51.208544 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:51.208500    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-98be89/noten-model-98be89-kserve-5fcf5695ff-88j24"]
Sep 11 14:35:51.214508 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:51.214478    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-98be89/noten-model-98be89-kserve-5fcf5695ff-88j24"]
Sep 11 14:35:51.499413 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:51.499382    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1ee2609-c0e7-4161-812c-820087413336" path="/var/lib/kubelet/pods/e1ee2609-c0e7-4161-812c-820087413336/volumes"
Sep 11 14:35:52.167463 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:52.167429    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh"]
Sep 11 14:35:52.167996 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:52.167754    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="849d7667-fb78-4454-aac1-662b2c962367" containerName="istio-proxy"
Sep 11 14:35:52.167996 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:52.167765    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="849d7667-fb78-4454-aac1-662b2c962367" containerName="istio-proxy"
Sep 11 14:35:52.167996 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:52.167780    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0984e71-aec3-4372-b101-ecf16c20488a" containerName="istio-proxy"
Sep 11 14:35:52.167996 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:52.167786    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0984e71-aec3-4372-b101-ecf16c20488a" containerName="istio-proxy"
Sep 11 14:35:52.167996 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:52.167797    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1ee2609-c0e7-4161-812c-820087413336" containerName="main"
Sep 11 14:35:52.167996 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:52.167802    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1ee2609-c0e7-4161-812c-820087413336" containerName="main"
Sep 11 14:35:52.167996 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:52.167809    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0984e71-aec3-4372-b101-ecf16c20488a" containerName="istio-proxy"
Sep 11 14:35:52.167996 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:52.167814    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0984e71-aec3-4372-b101-ecf16c20488a" containerName="istio-proxy"
Sep 11 14:35:52.167996 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:52.167868    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="849d7667-fb78-4454-aac1-662b2c962367" containerName="istio-proxy"
Sep 11 14:35:52.167996 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:52.167875    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0984e71-aec3-4372-b101-ecf16c20488a" containerName="istio-proxy"
Sep 11 14:35:52.167996 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:52.167883    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1ee2609-c0e7-4161-812c-820087413336" containerName="main"
Sep 11 14:35:52.172224 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:52.172206    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh"
Sep 11 14:35:52.175045 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:52.175023    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-09b16c-openshift-default-dockercfg-v8wnx\""
Sep 11 14:35:52.194149 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:52.194113    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh"]
Sep 11 14:35:52.281052 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:52.281012    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88197d9f-d033-4403-aef8-dd65332427a9-istio-podinfo\") pod \"e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh\" (UID: \"88197d9f-d033-4403-aef8-dd65332427a9\") " pod="openshift-ingress/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh"
Sep 11 14:35:52.281266 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:52.281100    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88197d9f-d033-4403-aef8-dd65332427a9-istio-data\") pod \"e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh\" (UID: \"88197d9f-d033-4403-aef8-dd65332427a9\") " pod="openshift-ingress/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh"
Sep 11 14:35:52.281266 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:52.281132    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z5tg\" (UniqueName: \"kubernetes.io/projected/88197d9f-d033-4403-aef8-dd65332427a9-kube-api-access-6z5tg\") pod \"e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh\" (UID: \"88197d9f-d033-4403-aef8-dd65332427a9\") " pod="openshift-ingress/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh"
Sep 11 14:35:52.281266 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:52.281177    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88197d9f-d033-4403-aef8-dd65332427a9-istio-token\") pod \"e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh\" (UID: \"88197d9f-d033-4403-aef8-dd65332427a9\") " pod="openshift-ingress/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh"
Sep 11 14:35:52.281266 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:52.281252    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88197d9f-d033-4403-aef8-dd65332427a9-workload-socket\") pod \"e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh\" (UID: \"88197d9f-d033-4403-aef8-dd65332427a9\") " pod="openshift-ingress/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh"
Sep 11 14:35:52.281523 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:52.281312    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88197d9f-d033-4403-aef8-dd65332427a9-credential-socket\") pod \"e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh\" (UID: \"88197d9f-d033-4403-aef8-dd65332427a9\") " pod="openshift-ingress/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh"
Sep 11 14:35:52.281523 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:52.281339    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88197d9f-d033-4403-aef8-dd65332427a9-istiod-ca-cert\") pod \"e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh\" (UID: \"88197d9f-d033-4403-aef8-dd65332427a9\") " pod="openshift-ingress/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh"
Sep 11 14:35:52.281523 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:52.281386    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88197d9f-d033-4403-aef8-dd65332427a9-istio-envoy\") pod \"e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh\" (UID: \"88197d9f-d033-4403-aef8-dd65332427a9\") " pod="openshift-ingress/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh"
Sep 11 14:35:52.281523 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:52.281421    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88197d9f-d033-4403-aef8-dd65332427a9-workload-certs\") pod \"e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh\" (UID: \"88197d9f-d033-4403-aef8-dd65332427a9\") " pod="openshift-ingress/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh"
Sep 11 14:35:52.382728 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:52.382693    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88197d9f-d033-4403-aef8-dd65332427a9-istio-envoy\") pod \"e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh\" (UID: \"88197d9f-d033-4403-aef8-dd65332427a9\") " pod="openshift-ingress/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh"
Sep 11 14:35:52.382920 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:52.382745    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88197d9f-d033-4403-aef8-dd65332427a9-workload-certs\") pod \"e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh\" (UID: \"88197d9f-d033-4403-aef8-dd65332427a9\") " pod="openshift-ingress/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh"
Sep 11 14:35:52.382920 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:52.382789    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88197d9f-d033-4403-aef8-dd65332427a9-istio-podinfo\") pod \"e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh\" (UID: \"88197d9f-d033-4403-aef8-dd65332427a9\") " pod="openshift-ingress/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh"
Sep 11 14:35:52.383097 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:52.383074    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88197d9f-d033-4403-aef8-dd65332427a9-istio-data\") pod \"e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh\" (UID: \"88197d9f-d033-4403-aef8-dd65332427a9\") " pod="openshift-ingress/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh"
Sep 11 14:35:52.383148 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:52.383130    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z5tg\" (UniqueName: \"kubernetes.io/projected/88197d9f-d033-4403-aef8-dd65332427a9-kube-api-access-6z5tg\") pod \"e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh\" (UID: \"88197d9f-d033-4403-aef8-dd65332427a9\") " pod="openshift-ingress/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh"
Sep 11 14:35:52.383197 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:52.383184    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88197d9f-d033-4403-aef8-dd65332427a9-workload-certs\") pod \"e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh\" (UID: \"88197d9f-d033-4403-aef8-dd65332427a9\") " pod="openshift-ingress/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh"
Sep 11 14:35:52.383256 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:52.383183    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88197d9f-d033-4403-aef8-dd65332427a9-istio-token\") pod \"e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh\" (UID: \"88197d9f-d033-4403-aef8-dd65332427a9\") " pod="openshift-ingress/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh"
Sep 11 14:35:52.383256 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:52.383245    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88197d9f-d033-4403-aef8-dd65332427a9-workload-socket\") pod \"e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh\" (UID: \"88197d9f-d033-4403-aef8-dd65332427a9\") " pod="openshift-ingress/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh"
Sep 11 14:35:52.383355 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:52.383323    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88197d9f-d033-4403-aef8-dd65332427a9-credential-socket\") pod \"e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh\" (UID: \"88197d9f-d033-4403-aef8-dd65332427a9\") " pod="openshift-ingress/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh"
Sep 11 14:35:52.383355 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:52.383351    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88197d9f-d033-4403-aef8-dd65332427a9-istiod-ca-cert\") pod \"e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh\" (UID: \"88197d9f-d033-4403-aef8-dd65332427a9\") " pod="openshift-ingress/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh"
Sep 11 14:35:52.383461 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:52.383376    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88197d9f-d033-4403-aef8-dd65332427a9-istio-data\") pod \"e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh\" (UID: \"88197d9f-d033-4403-aef8-dd65332427a9\") " pod="openshift-ingress/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh"
Sep 11 14:35:52.383660 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:52.383631    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88197d9f-d033-4403-aef8-dd65332427a9-workload-socket\") pod \"e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh\" (UID: \"88197d9f-d033-4403-aef8-dd65332427a9\") " pod="openshift-ingress/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh"
Sep 11 14:35:52.383821 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:52.383801    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88197d9f-d033-4403-aef8-dd65332427a9-credential-socket\") pod \"e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh\" (UID: \"88197d9f-d033-4403-aef8-dd65332427a9\") " pod="openshift-ingress/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh"
Sep 11 14:35:52.384039 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:52.384018    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88197d9f-d033-4403-aef8-dd65332427a9-istiod-ca-cert\") pod \"e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh\" (UID: \"88197d9f-d033-4403-aef8-dd65332427a9\") " pod="openshift-ingress/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh"
Sep 11 14:35:52.385083 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:52.385064    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88197d9f-d033-4403-aef8-dd65332427a9-istio-envoy\") pod \"e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh\" (UID: \"88197d9f-d033-4403-aef8-dd65332427a9\") " pod="openshift-ingress/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh"
Sep 11 14:35:52.385611 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:52.385593    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88197d9f-d033-4403-aef8-dd65332427a9-istio-podinfo\") pod \"e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh\" (UID: \"88197d9f-d033-4403-aef8-dd65332427a9\") " pod="openshift-ingress/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh"
Sep 11 14:35:52.395238 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:52.395211    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88197d9f-d033-4403-aef8-dd65332427a9-istio-token\") pod \"e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh\" (UID: \"88197d9f-d033-4403-aef8-dd65332427a9\") " pod="openshift-ingress/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh"
Sep 11 14:35:52.395337 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:52.395306    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z5tg\" (UniqueName: \"kubernetes.io/projected/88197d9f-d033-4403-aef8-dd65332427a9-kube-api-access-6z5tg\") pod \"e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh\" (UID: \"88197d9f-d033-4403-aef8-dd65332427a9\") " pod="openshift-ingress/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh"
Sep 11 14:35:52.483950 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:52.483853    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh"
Sep 11 14:35:52.778073 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:52.778047    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh"]
Sep 11 14:35:52.780094 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:35:52.780071    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88197d9f_d033_4403_aef8_dd65332427a9.slice/crio-fedb7e7769d81d9928802051994f0c0e3f9b3fcc4dcd7e576aa229cf3f29780b WatchSource:0}: Error finding container fedb7e7769d81d9928802051994f0c0e3f9b3fcc4dcd7e576aa229cf3f29780b: Status 404 returned error can't find the container with id fedb7e7769d81d9928802051994f0c0e3f9b3fcc4dcd7e576aa229cf3f29780b
Sep 11 14:35:52.782186 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:52.782154    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:35:52.782287 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:52.782230    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:35:52.782287 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:52.782276    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:35:53.187830 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:53.187729    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh" event={"ID":"88197d9f-d033-4403-aef8-dd65332427a9","Type":"ContainerStarted","Data":"f8e71261cd31bb95cee7abb65ad46a5893bb12f629044fc8971098b24a223e42"}
Sep 11 14:35:53.187830 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:53.187788    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh" event={"ID":"88197d9f-d033-4403-aef8-dd65332427a9","Type":"ContainerStarted","Data":"fedb7e7769d81d9928802051994f0c0e3f9b3fcc4dcd7e576aa229cf3f29780b"}
Sep 11 14:35:53.211466 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:53.211416    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh" podStartSLOduration=1.211399304 podStartE2EDuration="1.211399304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:35:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:35:53.210613997 +0000 UTC m=+1256.361164062" watchObservedRunningTime="2026-09-11 14:35:53.211399304 +0000 UTC m=+1256.361949368"
Sep 11 14:35:53.484714 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:53.484625    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh"
Sep 11 14:35:53.489188 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:53.489162    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh"
Sep 11 14:35:54.193253 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:54.193222    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh"
Sep 11 14:35:54.194057 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:54.194037    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh"
Sep 11 14:35:56.348956 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:56.348933    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt"
Sep 11 14:35:56.416615 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:56.416521    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bde0890-f752-4b4b-976f-804512c66e7c-istio-data\") pod \"8bde0890-f752-4b4b-976f-804512c66e7c\" (UID: \"8bde0890-f752-4b4b-976f-804512c66e7c\") "
Sep 11 14:35:56.416615 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:56.416555    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bde0890-f752-4b4b-976f-804512c66e7c-workload-certs\") pod \"8bde0890-f752-4b4b-976f-804512c66e7c\" (UID: \"8bde0890-f752-4b4b-976f-804512c66e7c\") "
Sep 11 14:35:56.416615 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:56.416582    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bde0890-f752-4b4b-976f-804512c66e7c-istiod-ca-cert\") pod \"8bde0890-f752-4b4b-976f-804512c66e7c\" (UID: \"8bde0890-f752-4b4b-976f-804512c66e7c\") "
Sep 11 14:35:56.416615 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:56.416612    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bde0890-f752-4b4b-976f-804512c66e7c-credential-socket\") pod \"8bde0890-f752-4b4b-976f-804512c66e7c\" (UID: \"8bde0890-f752-4b4b-976f-804512c66e7c\") "
Sep 11 14:35:56.416956 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:56.416636    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bde0890-f752-4b4b-976f-804512c66e7c-istio-token\") pod \"8bde0890-f752-4b4b-976f-804512c66e7c\" (UID: \"8bde0890-f752-4b4b-976f-804512c66e7c\") "
Sep 11 14:35:56.416956 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:56.416655    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bde0890-f752-4b4b-976f-804512c66e7c-istio-podinfo\") pod \"8bde0890-f752-4b4b-976f-804512c66e7c\" (UID: \"8bde0890-f752-4b4b-976f-804512c66e7c\") "
Sep 11 14:35:56.416956 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:56.416678    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bde0890-f752-4b4b-976f-804512c66e7c-istio-envoy\") pod \"8bde0890-f752-4b4b-976f-804512c66e7c\" (UID: \"8bde0890-f752-4b4b-976f-804512c66e7c\") "
Sep 11 14:35:56.416956 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:56.416710    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bde0890-f752-4b4b-976f-804512c66e7c-workload-socket\") pod \"8bde0890-f752-4b4b-976f-804512c66e7c\" (UID: \"8bde0890-f752-4b4b-976f-804512c66e7c\") "
Sep 11 14:35:56.416956 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:56.416743    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8sjhh\" (UniqueName: \"kubernetes.io/projected/8bde0890-f752-4b4b-976f-804512c66e7c-kube-api-access-8sjhh\") pod \"8bde0890-f752-4b4b-976f-804512c66e7c\" (UID: \"8bde0890-f752-4b4b-976f-804512c66e7c\") "
Sep 11 14:35:56.416956 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:56.416817    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bde0890-f752-4b4b-976f-804512c66e7c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8bde0890-f752-4b4b-976f-804512c66e7c" (UID: "8bde0890-f752-4b4b-976f-804512c66e7c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:35:56.416956 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:56.416952    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bde0890-f752-4b4b-976f-804512c66e7c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8bde0890-f752-4b4b-976f-804512c66e7c" (UID: "8bde0890-f752-4b4b-976f-804512c66e7c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:35:56.417411 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:56.417108    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bde0890-f752-4b4b-976f-804512c66e7c-credential-socket\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:35:56.417411 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:56.417127    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bde0890-f752-4b4b-976f-804512c66e7c-workload-certs\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:35:56.417411 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:56.417055    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bde0890-f752-4b4b-976f-804512c66e7c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8bde0890-f752-4b4b-976f-804512c66e7c" (UID: "8bde0890-f752-4b4b-976f-804512c66e7c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:35:56.417411 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:56.417165    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bde0890-f752-4b4b-976f-804512c66e7c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8bde0890-f752-4b4b-976f-804512c66e7c" (UID: "8bde0890-f752-4b4b-976f-804512c66e7c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:35:56.417411 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:56.417231    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bde0890-f752-4b4b-976f-804512c66e7c-istio-data" (OuterVolumeSpecName: "istio-data") pod "8bde0890-f752-4b4b-976f-804512c66e7c" (UID: "8bde0890-f752-4b4b-976f-804512c66e7c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:35:56.419170 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:56.419138    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8bde0890-f752-4b4b-976f-804512c66e7c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8bde0890-f752-4b4b-976f-804512c66e7c" (UID: "8bde0890-f752-4b4b-976f-804512c66e7c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 14:35:56.419280 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:56.419162    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bde0890-f752-4b4b-976f-804512c66e7c-kube-api-access-8sjhh" (OuterVolumeSpecName: "kube-api-access-8sjhh") pod "8bde0890-f752-4b4b-976f-804512c66e7c" (UID: "8bde0890-f752-4b4b-976f-804512c66e7c"). InnerVolumeSpecName "kube-api-access-8sjhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:35:56.419408 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:56.419388    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bde0890-f752-4b4b-976f-804512c66e7c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8bde0890-f752-4b4b-976f-804512c66e7c" (UID: "8bde0890-f752-4b4b-976f-804512c66e7c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:35:56.419482 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:56.419463    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bde0890-f752-4b4b-976f-804512c66e7c-istio-token" (OuterVolumeSpecName: "istio-token") pod "8bde0890-f752-4b4b-976f-804512c66e7c" (UID: "8bde0890-f752-4b4b-976f-804512c66e7c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:35:56.517849 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:56.517817    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sjhh\" (UniqueName: \"kubernetes.io/projected/8bde0890-f752-4b4b-976f-804512c66e7c-kube-api-access-8sjhh\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:35:56.517849 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:56.517842    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bde0890-f752-4b4b-976f-804512c66e7c-istio-data\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:35:56.517849 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:56.517851    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bde0890-f752-4b4b-976f-804512c66e7c-istiod-ca-cert\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:35:56.517849 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:56.517858    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bde0890-f752-4b4b-976f-804512c66e7c-istio-token\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:35:56.518128 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:56.517868    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bde0890-f752-4b4b-976f-804512c66e7c-istio-podinfo\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:35:56.518128 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:56.517876    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bde0890-f752-4b4b-976f-804512c66e7c-istio-envoy\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:35:56.518128 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:56.517884    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bde0890-f752-4b4b-976f-804512c66e7c-workload-socket\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:35:57.212268 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:57.212225    2581 generic.go:356] "Generic (PLEG): container finished" podID="8bde0890-f752-4b4b-976f-804512c66e7c" containerID="730dd42bf8c9fe1a78019eab32498867f0d998cf1635563316dc9c74c9e01f0a" exitCode=0
Sep 11 14:35:57.212443 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:57.212281    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt"
Sep 11 14:35:57.212443 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:57.212308    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt" event={"ID":"8bde0890-f752-4b4b-976f-804512c66e7c","Type":"ContainerDied","Data":"730dd42bf8c9fe1a78019eab32498867f0d998cf1635563316dc9c74c9e01f0a"}
Sep 11 14:35:57.212443 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:57.212358    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt" event={"ID":"8bde0890-f752-4b4b-976f-804512c66e7c","Type":"ContainerDied","Data":"11c4aa3868cb8b334b6e06215c9f78a4f45d8be45370900c9918ed3d29550e99"}
Sep 11 14:35:57.212443 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:57.212379    2581 scope.go:117] "RemoveContainer" containerID="730dd42bf8c9fe1a78019eab32498867f0d998cf1635563316dc9c74c9e01f0a"
Sep 11 14:35:57.221859 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:57.221676    2581 scope.go:117] "RemoveContainer" containerID="05b9327f3bdb2bd741dc4dcd8a48cd091862b275dabf80dfeec4ec0a0c2bed59"
Sep 11 14:35:57.230248 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:57.230229    2581 scope.go:117] "RemoveContainer" containerID="730dd42bf8c9fe1a78019eab32498867f0d998cf1635563316dc9c74c9e01f0a"
Sep 11 14:35:57.230502 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:35:57.230486    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"730dd42bf8c9fe1a78019eab32498867f0d998cf1635563316dc9c74c9e01f0a\": container with ID starting with 730dd42bf8c9fe1a78019eab32498867f0d998cf1635563316dc9c74c9e01f0a not found: ID does not exist" containerID="730dd42bf8c9fe1a78019eab32498867f0d998cf1635563316dc9c74c9e01f0a"
Sep 11 14:35:57.230552 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:57.230510    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"730dd42bf8c9fe1a78019eab32498867f0d998cf1635563316dc9c74c9e01f0a"} err="failed to get container status \"730dd42bf8c9fe1a78019eab32498867f0d998cf1635563316dc9c74c9e01f0a\": rpc error: code = NotFound desc = could not find container \"730dd42bf8c9fe1a78019eab32498867f0d998cf1635563316dc9c74c9e01f0a\": container with ID starting with 730dd42bf8c9fe1a78019eab32498867f0d998cf1635563316dc9c74c9e01f0a not found: ID does not exist"
Sep 11 14:35:57.230552 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:57.230526    2581 scope.go:117] "RemoveContainer" containerID="05b9327f3bdb2bd741dc4dcd8a48cd091862b275dabf80dfeec4ec0a0c2bed59"
Sep 11 14:35:57.230750 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:35:57.230732    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05b9327f3bdb2bd741dc4dcd8a48cd091862b275dabf80dfeec4ec0a0c2bed59\": container with ID starting with 05b9327f3bdb2bd741dc4dcd8a48cd091862b275dabf80dfeec4ec0a0c2bed59 not found: ID does not exist" containerID="05b9327f3bdb2bd741dc4dcd8a48cd091862b275dabf80dfeec4ec0a0c2bed59"
Sep 11 14:35:57.230798 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:57.230757    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05b9327f3bdb2bd741dc4dcd8a48cd091862b275dabf80dfeec4ec0a0c2bed59"} err="failed to get container status \"05b9327f3bdb2bd741dc4dcd8a48cd091862b275dabf80dfeec4ec0a0c2bed59\": rpc error: code = NotFound desc = could not find container \"05b9327f3bdb2bd741dc4dcd8a48cd091862b275dabf80dfeec4ec0a0c2bed59\": container with ID starting with 05b9327f3bdb2bd741dc4dcd8a48cd091862b275dabf80dfeec4ec0a0c2bed59 not found: ID does not exist"
Sep 11 14:35:57.239151 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:57.239129    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt"]
Sep 11 14:35:57.247695 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:57.247673    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-5a7035-openshift-default-6459b84b5d-7csxt"]
Sep 11 14:35:57.499210 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:35:57.499181    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bde0890-f752-4b4b-976f-804512c66e7c" path="/var/lib/kubelet/pods/8bde0890-f752-4b4b-976f-804512c66e7c/volumes"
Sep 11 14:36:00.056941 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:00.056901    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-09b16c-57c884f8fc-8pm84"]
Sep 11 14:36:00.057447 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:00.057426    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bde0890-f752-4b4b-976f-804512c66e7c" containerName="istio-proxy"
Sep 11 14:36:00.057492 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:00.057451    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bde0890-f752-4b4b-976f-804512c66e7c" containerName="istio-proxy"
Sep 11 14:36:00.057492 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:00.057486    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bde0890-f752-4b4b-976f-804512c66e7c" containerName="istio-proxy"
Sep 11 14:36:00.057555 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:00.057494    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bde0890-f752-4b4b-976f-804512c66e7c" containerName="istio-proxy"
Sep 11 14:36:00.057589 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:00.057571    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0984e71-aec3-4372-b101-ecf16c20488a" containerName="istio-proxy"
Sep 11 14:36:00.057589 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:00.057585    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bde0890-f752-4b4b-976f-804512c66e7c" containerName="istio-proxy"
Sep 11 14:36:00.057652 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:00.057592    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bde0890-f752-4b4b-976f-804512c66e7c" containerName="istio-proxy"
Sep 11 14:36:00.062128 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:00.062112    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-09b16c-57c884f8fc-8pm84"
Sep 11 14:36:00.066848 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:00.066827    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-09b16c-dockercfg-nswlm\""
Sep 11 14:36:00.067539 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:00.067522    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-09b16c\""
Sep 11 14:36:00.086569 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:00.086537    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-09b16c-57c884f8fc-8pm84"]
Sep 11 14:36:00.149158 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:00.149123    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwmx2\" (UniqueName: \"kubernetes.io/projected/984b5347-e783-4113-96c3-49552ac4bdcc-kube-api-access-rwmx2\") pod \"payload-processing-e2e-shared-a-w3-09b16c-57c884f8fc-8pm84\" (UID: \"984b5347-e783-4113-96c3-49552ac4bdcc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-09b16c-57c884f8fc-8pm84"
Sep 11 14:36:00.149364 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:00.149165    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/984b5347-e783-4113-96c3-49552ac4bdcc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-09b16c-57c884f8fc-8pm84\" (UID: \"984b5347-e783-4113-96c3-49552ac4bdcc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-09b16c-57c884f8fc-8pm84"
Sep 11 14:36:00.250686 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:00.250649    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwmx2\" (UniqueName: \"kubernetes.io/projected/984b5347-e783-4113-96c3-49552ac4bdcc-kube-api-access-rwmx2\") pod \"payload-processing-e2e-shared-a-w3-09b16c-57c884f8fc-8pm84\" (UID: \"984b5347-e783-4113-96c3-49552ac4bdcc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-09b16c-57c884f8fc-8pm84"
Sep 11 14:36:00.250686 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:00.250688    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/984b5347-e783-4113-96c3-49552ac4bdcc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-09b16c-57c884f8fc-8pm84\" (UID: \"984b5347-e783-4113-96c3-49552ac4bdcc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-09b16c-57c884f8fc-8pm84"
Sep 11 14:36:00.251398 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:00.251379    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/984b5347-e783-4113-96c3-49552ac4bdcc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-09b16c-57c884f8fc-8pm84\" (UID: \"984b5347-e783-4113-96c3-49552ac4bdcc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-09b16c-57c884f8fc-8pm84"
Sep 11 14:36:00.272170 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:00.272137    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwmx2\" (UniqueName: \"kubernetes.io/projected/984b5347-e783-4113-96c3-49552ac4bdcc-kube-api-access-rwmx2\") pod \"payload-processing-e2e-shared-a-w3-09b16c-57c884f8fc-8pm84\" (UID: \"984b5347-e783-4113-96c3-49552ac4bdcc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-09b16c-57c884f8fc-8pm84"
Sep 11 14:36:00.371743 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:00.371651    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-09b16c-57c884f8fc-8pm84"
Sep 11 14:36:00.708102 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:00.708076    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-09b16c-57c884f8fc-8pm84"]
Sep 11 14:36:00.709938 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:36:00.709860    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod984b5347_e783_4113_96c3_49552ac4bdcc.slice/crio-7ca4cbe5a5d1f5f3154eb56374d65febc05e21b1a13285758f0e6fdb59863196 WatchSource:0}: Error finding container 7ca4cbe5a5d1f5f3154eb56374d65febc05e21b1a13285758f0e6fdb59863196: Status 404 returned error can't find the container with id 7ca4cbe5a5d1f5f3154eb56374d65febc05e21b1a13285758f0e6fdb59863196
Sep 11 14:36:00.712035 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:00.712017    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 14:36:01.233770 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:01.233737    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-09b16c-57c884f8fc-8pm84" event={"ID":"984b5347-e783-4113-96c3-49552ac4bdcc","Type":"ContainerStarted","Data":"7ca4cbe5a5d1f5f3154eb56374d65febc05e21b1a13285758f0e6fdb59863196"}
Sep 11 14:36:03.246273 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:03.246237    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-09b16c-57c884f8fc-8pm84" event={"ID":"984b5347-e783-4113-96c3-49552ac4bdcc","Type":"ContainerStarted","Data":"ff94a58c1cbd23839f2e247ffa4fa156a3eaf70d7edfd10c3b869f5bf354a920"}
Sep 11 14:36:03.246699 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:03.246370    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-09b16c-57c884f8fc-8pm84"
Sep 11 14:36:03.285373 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:03.285319    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-09b16c-57c884f8fc-8pm84" podStartSLOduration=1.050045464 podStartE2EDuration="3.285298522s" totalImagesPullingTime="2.235253058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:36:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:36:03.283232349 +0000 UTC m=+1266.433782412" watchObservedRunningTime="2026-09-11 14:36:03.285298522 +0000 UTC m=+1266.435848582"
Sep 11 14:36:14.253648 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:14.253616    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-09b16c-57c884f8fc-8pm84"
Sep 11 14:36:14.304776 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:14.304744    2581 generic.go:356] "Generic (PLEG): container finished" podID="88197d9f-d033-4403-aef8-dd65332427a9" containerID="f8e71261cd31bb95cee7abb65ad46a5893bb12f629044fc8971098b24a223e42" exitCode=0
Sep 11 14:36:14.304942 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:14.304826    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh" event={"ID":"88197d9f-d033-4403-aef8-dd65332427a9","Type":"ContainerDied","Data":"f8e71261cd31bb95cee7abb65ad46a5893bb12f629044fc8971098b24a223e42"}
Sep 11 14:36:14.305254 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:14.305240    2581 scope.go:117] "RemoveContainer" containerID="f8e71261cd31bb95cee7abb65ad46a5893bb12f629044fc8971098b24a223e42"
Sep 11 14:36:14.306320 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:14.306293    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:36:14.306395 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:14.306357    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:36:14.306435 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:14.306410    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:36:15.315555 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:15.315515    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh" event={"ID":"88197d9f-d033-4403-aef8-dd65332427a9","Type":"ContainerStarted","Data":"a59c51c6a043bc3b5b9752833b854ffa4da631470ce2a213163173e114111445"}
Sep 11 14:36:15.484725 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:15.484692    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh"
Sep 11 14:36:15.485007 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:15.484985    2581 patch_prober.go:28] interesting pod/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Sep 11 14:36:15.485068 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:15.485032    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh" podUID="88197d9f-d033-4403-aef8-dd65332427a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Sep 11 14:36:16.484595 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:16.484553    2581 patch_prober.go:28] interesting pod/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Sep 11 14:36:16.485038 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:16.484615    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh" podUID="88197d9f-d033-4403-aef8-dd65332427a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Sep 11 14:36:17.485362 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:17.485262    2581 patch_prober.go:28] interesting pod/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Sep 11 14:36:17.485913 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:17.485450    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh" podUID="88197d9f-d033-4403-aef8-dd65332427a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Sep 11 14:36:18.484634 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:18.484595    2581 patch_prober.go:28] interesting pod/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Sep 11 14:36:18.484837 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:18.484649    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh" podUID="88197d9f-d033-4403-aef8-dd65332427a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Sep 11 14:36:19.485227 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:19.485190    2581 patch_prober.go:28] interesting pod/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Sep 11 14:36:19.485639 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:19.485250    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh" podUID="88197d9f-d033-4403-aef8-dd65332427a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Sep 11 14:36:20.606935 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:20.606897    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh"
Sep 11 14:36:20.607463 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:20.607302    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh"
Sep 11 14:36:20.609025 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:20.609003    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh"
Sep 11 14:36:21.078728 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:21.078689    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8"]
Sep 11 14:36:21.084550 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:21.084525    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8"
Sep 11 14:36:21.087178 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:21.087153    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-1c284e-openshift-default-dockercfg-kt4sk\""
Sep 11 14:36:21.100960 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:21.100929    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8"]
Sep 11 14:36:21.157439 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:21.157406    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887-istio-data\") pod \"e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8\" (UID: \"1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887\") " pod="openshift-ingress/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8"
Sep 11 14:36:21.157439 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:21.157438    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887-istio-envoy\") pod \"e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8\" (UID: \"1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887\") " pod="openshift-ingress/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8"
Sep 11 14:36:21.157674 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:21.157469    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887-workload-socket\") pod \"e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8\" (UID: \"1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887\") " pod="openshift-ingress/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8"
Sep 11 14:36:21.157674 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:21.157491    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trltp\" (UniqueName: \"kubernetes.io/projected/1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887-kube-api-access-trltp\") pod \"e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8\" (UID: \"1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887\") " pod="openshift-ingress/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8"
Sep 11 14:36:21.157674 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:21.157508    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887-workload-certs\") pod \"e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8\" (UID: \"1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887\") " pod="openshift-ingress/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8"
Sep 11 14:36:21.157674 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:21.157522    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887-istiod-ca-cert\") pod \"e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8\" (UID: \"1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887\") " pod="openshift-ingress/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8"
Sep 11 14:36:21.157674 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:21.157539    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887-istio-podinfo\") pod \"e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8\" (UID: \"1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887\") " pod="openshift-ingress/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8"
Sep 11 14:36:21.157674 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:21.157640    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887-credential-socket\") pod \"e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8\" (UID: \"1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887\") " pod="openshift-ingress/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8"
Sep 11 14:36:21.157929 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:21.157707    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887-istio-token\") pod \"e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8\" (UID: \"1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887\") " pod="openshift-ingress/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8"
Sep 11 14:36:21.258275 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:21.258239    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887-istio-data\") pod \"e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8\" (UID: \"1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887\") " pod="openshift-ingress/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8"
Sep 11 14:36:21.258275 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:21.258280    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887-istio-envoy\") pod \"e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8\" (UID: \"1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887\") " pod="openshift-ingress/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8"
Sep 11 14:36:21.258526 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:21.258313    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887-workload-socket\") pod \"e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8\" (UID: \"1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887\") " pod="openshift-ingress/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8"
Sep 11 14:36:21.258526 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:21.258434    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trltp\" (UniqueName: \"kubernetes.io/projected/1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887-kube-api-access-trltp\") pod \"e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8\" (UID: \"1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887\") " pod="openshift-ingress/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8"
Sep 11 14:36:21.258526 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:21.258467    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887-workload-certs\") pod \"e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8\" (UID: \"1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887\") " pod="openshift-ingress/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8"
Sep 11 14:36:21.258526 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:21.258496    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887-istiod-ca-cert\") pod \"e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8\" (UID: \"1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887\") " pod="openshift-ingress/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8"
Sep 11 14:36:21.258741 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:21.258623    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887-istio-podinfo\") pod \"e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8\" (UID: \"1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887\") " pod="openshift-ingress/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8"
Sep 11 14:36:21.258741 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:21.258654    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887-workload-socket\") pod \"e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8\" (UID: \"1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887\") " pod="openshift-ingress/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8"
Sep 11 14:36:21.258741 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:21.258681    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887-istio-data\") pod \"e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8\" (UID: \"1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887\") " pod="openshift-ingress/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8"
Sep 11 14:36:21.258741 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:21.258720    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887-credential-socket\") pod \"e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8\" (UID: \"1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887\") " pod="openshift-ingress/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8"
Sep 11 14:36:21.258741 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:21.258737    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887-workload-certs\") pod \"e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8\" (UID: \"1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887\") " pod="openshift-ingress/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8"
Sep 11 14:36:21.258952 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:21.258813    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887-istio-token\") pod \"e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8\" (UID: \"1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887\") " pod="openshift-ingress/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8"
Sep 11 14:36:21.259021 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:21.258946    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887-credential-socket\") pod \"e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8\" (UID: \"1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887\") " pod="openshift-ingress/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8"
Sep 11 14:36:21.259360 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:21.259339    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887-istiod-ca-cert\") pod \"e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8\" (UID: \"1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887\") " pod="openshift-ingress/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8"
Sep 11 14:36:21.260708 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:21.260676    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887-istio-envoy\") pod \"e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8\" (UID: \"1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887\") " pod="openshift-ingress/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8"
Sep 11 14:36:21.261032 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:21.261013    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887-istio-podinfo\") pod \"e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8\" (UID: \"1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887\") " pod="openshift-ingress/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8"
Sep 11 14:36:21.272764 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:21.272740    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887-istio-token\") pod \"e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8\" (UID: \"1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887\") " pod="openshift-ingress/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8"
Sep 11 14:36:21.272911 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:21.272820    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trltp\" (UniqueName: \"kubernetes.io/projected/1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887-kube-api-access-trltp\") pod \"e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8\" (UID: \"1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887\") " pod="openshift-ingress/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8"
Sep 11 14:36:21.394842 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:21.394753    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8"
Sep 11 14:36:21.678308 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:21.678156    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8"]
Sep 11 14:36:21.680843 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:36:21.680815    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d0b8dcf_6ab3_40ac_8b31_3eca2d6b2887.slice/crio-e120b8b5759224652e388193d05c9c5d0db753bb8053429df224d584263e1b3f WatchSource:0}: Error finding container e120b8b5759224652e388193d05c9c5d0db753bb8053429df224d584263e1b3f: Status 404 returned error can't find the container with id e120b8b5759224652e388193d05c9c5d0db753bb8053429df224d584263e1b3f
Sep 11 14:36:21.683171 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:21.683101    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:36:21.683257 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:21.683208    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:36:21.683257 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:21.683236    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:36:22.353012 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:22.352935    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8" event={"ID":"1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887","Type":"ContainerStarted","Data":"07ed191a516fa6ba406ef6c649c57c30e676d7f97c33a1a4328f78259aef0bc8"}
Sep 11 14:36:22.353351 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:22.353329    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8" event={"ID":"1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887","Type":"ContainerStarted","Data":"e120b8b5759224652e388193d05c9c5d0db753bb8053429df224d584263e1b3f"}
Sep 11 14:36:22.384937 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:22.384888    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8" podStartSLOduration=1.38487047 podStartE2EDuration="1.38487047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:36:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:36:22.384212032 +0000 UTC m=+1285.534762096" watchObservedRunningTime="2026-09-11 14:36:22.38487047 +0000 UTC m=+1285.535420536"
Sep 11 14:36:22.395290 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:22.395260    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8"
Sep 11 14:36:22.400388 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:22.400363    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8"
Sep 11 14:36:23.357476 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:23.357441    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8"
Sep 11 14:36:23.358492 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:23.358471    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8"
Sep 11 14:36:26.119116 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:26.119082    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c3340176-pgzzf"]
Sep 11 14:36:26.122523 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:26.122506    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c3340176-pgzzf"
Sep 11 14:36:26.125411 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:26.125390    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 14:36:26.125523 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:26.125445    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hcm4g\""
Sep 11 14:36:26.132777 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:26.132748    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c3340176-pgzzf"]
Sep 11 14:36:26.210776 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:26.210732    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e93f908-2ce9-4c4f-822e-435ed8f5aaec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c3340176-pgzzf\" (UID: \"8e93f908-2ce9-4c4f-822e-435ed8f5aaec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c3340176-pgzzf"
Sep 11 14:36:26.312036 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:26.311996    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e93f908-2ce9-4c4f-822e-435ed8f5aaec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c3340176-pgzzf\" (UID: \"8e93f908-2ce9-4c4f-822e-435ed8f5aaec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c3340176-pgzzf"
Sep 11 14:36:26.312628 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:26.312608    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e93f908-2ce9-4c4f-822e-435ed8f5aaec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c3340176-pgzzf\" (UID: \"8e93f908-2ce9-4c4f-822e-435ed8f5aaec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c3340176-pgzzf"
Sep 11 14:36:26.433809 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:26.433707    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c3340176-pgzzf"
Sep 11 14:36:26.718557 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:26.718529    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c3340176-pgzzf"]
Sep 11 14:36:26.720330 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:36:26.720302    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e93f908_2ce9_4c4f_822e_435ed8f5aaec.slice/crio-b47bc9effb5062afe728807adb0c9a42313796794d2ec7a0e6a987788f216fe8 WatchSource:0}: Error finding container b47bc9effb5062afe728807adb0c9a42313796794d2ec7a0e6a987788f216fe8: Status 404 returned error can't find the container with id b47bc9effb5062afe728807adb0c9a42313796794d2ec7a0e6a987788f216fe8
Sep 11 14:36:27.380178 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:27.380145    2581 generic.go:356] "Generic (PLEG): container finished" podID="8e93f908-2ce9-4c4f-822e-435ed8f5aaec" containerID="d3d14a566bca24f0ea195dde255a63741427e8f36339738c9eaf7e5411e6ae6f" exitCode=0
Sep 11 14:36:27.380593 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:27.380234    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c3340176-pgzzf" event={"ID":"8e93f908-2ce9-4c4f-822e-435ed8f5aaec","Type":"ContainerDied","Data":"d3d14a566bca24f0ea195dde255a63741427e8f36339738c9eaf7e5411e6ae6f"}
Sep 11 14:36:27.380593 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:27.380295    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c3340176-pgzzf" event={"ID":"8e93f908-2ce9-4c4f-822e-435ed8f5aaec","Type":"ContainerStarted","Data":"b47bc9effb5062afe728807adb0c9a42313796794d2ec7a0e6a987788f216fe8"}
Sep 11 14:36:28.514644 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:28.514619    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c3340176-pgzzf"
Sep 11 14:36:28.632306 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:28.632269    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e93f908-2ce9-4c4f-822e-435ed8f5aaec-maas-api-service-ca\") pod \"8e93f908-2ce9-4c4f-822e-435ed8f5aaec\" (UID: \"8e93f908-2ce9-4c4f-822e-435ed8f5aaec\") "
Sep 11 14:36:28.632636 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:28.632611    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e93f908-2ce9-4c4f-822e-435ed8f5aaec-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8e93f908-2ce9-4c4f-822e-435ed8f5aaec" (UID: "8e93f908-2ce9-4c4f-822e-435ed8f5aaec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:36:28.734133 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:28.734037    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e93f908-2ce9-4c4f-822e-435ed8f5aaec-maas-api-service-ca\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:36:29.391157 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:29.391126    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c3340176-pgzzf"
Sep 11 14:36:29.391332 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:29.391157    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c3340176-pgzzf" event={"ID":"8e93f908-2ce9-4c4f-822e-435ed8f5aaec","Type":"ContainerDied","Data":"b47bc9effb5062afe728807adb0c9a42313796794d2ec7a0e6a987788f216fe8"}
Sep 11 14:36:29.391332 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:29.391208    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b47bc9effb5062afe728807adb0c9a42313796794d2ec7a0e6a987788f216fe8"
Sep 11 14:36:37.524642 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:37.524607    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c3340176-pgzzf"]
Sep 11 14:36:37.529553 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:37.529525    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c3340176-pgzzf"]
Sep 11 14:36:39.503588 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:39.503521    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e93f908-2ce9-4c4f-822e-435ed8f5aaec" path="/var/lib/kubelet/pods/8e93f908-2ce9-4c4f-822e-435ed8f5aaec/volumes"
Sep 11 14:36:43.466521 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:43.466487    2581 generic.go:356] "Generic (PLEG): container finished" podID="1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887" containerID="07ed191a516fa6ba406ef6c649c57c30e676d7f97c33a1a4328f78259aef0bc8" exitCode=0
Sep 11 14:36:43.466947 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:43.466557    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8" event={"ID":"1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887","Type":"ContainerDied","Data":"07ed191a516fa6ba406ef6c649c57c30e676d7f97c33a1a4328f78259aef0bc8"}
Sep 11 14:36:43.467028 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:43.466997    2581 scope.go:117] "RemoveContainer" containerID="07ed191a516fa6ba406ef6c649c57c30e676d7f97c33a1a4328f78259aef0bc8"
Sep 11 14:36:43.468020 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:43.467992    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:36:43.468105 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:43.468058    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:36:43.468105 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:43.468087    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:36:44.473719 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:44.473674    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8" event={"ID":"1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887","Type":"ContainerStarted","Data":"b611a569df4ff66649b38e0234e5dde907fe5e0154f96bbb0afda33b83a8b80a"}
Sep 11 14:36:45.396261 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:45.396205    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8"
Sep 11 14:36:45.396500 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:45.396369    2581 patch_prober.go:28] interesting pod/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 11 14:36:45.396500 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:45.396446    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8" podUID="1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 11 14:36:46.396061 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:46.396012    2581 patch_prober.go:28] interesting pod/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 11 14:36:46.396589 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:46.396090    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8" podUID="1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 11 14:36:47.395536 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:47.395491    2581 patch_prober.go:28] interesting pod/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 11 14:36:47.395748 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:47.395562    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8" podUID="1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 11 14:36:48.395783 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:48.395739    2581 patch_prober.go:28] interesting pod/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 11 14:36:48.397153 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:48.397114    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8" podUID="1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 11 14:36:49.396207 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:49.396159    2581 patch_prober.go:28] interesting pod/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 11 14:36:49.396741 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:49.396261    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8" podUID="1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 11 14:36:50.399893 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:50.399858    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8"
Sep 11 14:36:50.400366 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:50.400250    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8"
Sep 11 14:36:50.401017 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:50.400996    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8"
Sep 11 14:36:54.471365 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:54.471325    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-09b16c/auth-test-model-09b16c-kserve-7df9b7df79-rg87k"]
Sep 11 14:36:54.471834 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:54.471676    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e93f908-2ce9-4c4f-822e-435ed8f5aaec" containerName="revoke-keys"
Sep 11 14:36:54.471834 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:54.471689    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e93f908-2ce9-4c4f-822e-435ed8f5aaec" containerName="revoke-keys"
Sep 11 14:36:54.471834 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:54.471756    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e93f908-2ce9-4c4f-822e-435ed8f5aaec" containerName="revoke-keys"
Sep 11 14:36:54.474788 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:54.474771    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-09b16c/auth-test-model-09b16c-kserve-7df9b7df79-rg87k"
Sep 11 14:36:54.480520 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:54.480497    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-09b16c\"/\"kube-root-ca.crt\""
Sep 11 14:36:54.480520 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:54.480511    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-09b16c\"/\"openshift-service-ca.crt\""
Sep 11 14:36:54.480520 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:54.480497    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-09b16c\"/\"auth-test-model-09b16c-kserve-self-signed-certs\""
Sep 11 14:36:54.480723 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:54.480499    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-09b16c\"/\"default-dockercfg-g4l97\""
Sep 11 14:36:54.495448 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:54.495420    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-09b16c/auth-test-model-09b16c-kserve-7df9b7df79-rg87k"]
Sep 11 14:36:54.586453 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:54.586413    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpbg9\" (UniqueName: \"kubernetes.io/projected/e7eeaf9b-5b0d-4208-9839-8bbf719de427-kube-api-access-jpbg9\") pod \"auth-test-model-09b16c-kserve-7df9b7df79-rg87k\" (UID: \"e7eeaf9b-5b0d-4208-9839-8bbf719de427\") " pod="ai-tenant-e2e-shared-a-w3-09b16c/auth-test-model-09b16c-kserve-7df9b7df79-rg87k"
Sep 11 14:36:54.586649 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:54.586521    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7eeaf9b-5b0d-4208-9839-8bbf719de427-tls-certs\") pod \"auth-test-model-09b16c-kserve-7df9b7df79-rg87k\" (UID: \"e7eeaf9b-5b0d-4208-9839-8bbf719de427\") " pod="ai-tenant-e2e-shared-a-w3-09b16c/auth-test-model-09b16c-kserve-7df9b7df79-rg87k"
Sep 11 14:36:54.586649 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:54.586609    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7eeaf9b-5b0d-4208-9839-8bbf719de427-model-cache\") pod \"auth-test-model-09b16c-kserve-7df9b7df79-rg87k\" (UID: \"e7eeaf9b-5b0d-4208-9839-8bbf719de427\") " pod="ai-tenant-e2e-shared-a-w3-09b16c/auth-test-model-09b16c-kserve-7df9b7df79-rg87k"
Sep 11 14:36:54.586786 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:54.586672    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7eeaf9b-5b0d-4208-9839-8bbf719de427-dshm\") pod \"auth-test-model-09b16c-kserve-7df9b7df79-rg87k\" (UID: \"e7eeaf9b-5b0d-4208-9839-8bbf719de427\") " pod="ai-tenant-e2e-shared-a-w3-09b16c/auth-test-model-09b16c-kserve-7df9b7df79-rg87k"
Sep 11 14:36:54.586786 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:54.586707    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7eeaf9b-5b0d-4208-9839-8bbf719de427-tmp-dir\") pod \"auth-test-model-09b16c-kserve-7df9b7df79-rg87k\" (UID: \"e7eeaf9b-5b0d-4208-9839-8bbf719de427\") " pod="ai-tenant-e2e-shared-a-w3-09b16c/auth-test-model-09b16c-kserve-7df9b7df79-rg87k"
Sep 11 14:36:54.586786 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:54.586727    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7eeaf9b-5b0d-4208-9839-8bbf719de427-home\") pod \"auth-test-model-09b16c-kserve-7df9b7df79-rg87k\" (UID: \"e7eeaf9b-5b0d-4208-9839-8bbf719de427\") " pod="ai-tenant-e2e-shared-a-w3-09b16c/auth-test-model-09b16c-kserve-7df9b7df79-rg87k"
Sep 11 14:36:54.687876 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:54.687842    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7eeaf9b-5b0d-4208-9839-8bbf719de427-tls-certs\") pod \"auth-test-model-09b16c-kserve-7df9b7df79-rg87k\" (UID: \"e7eeaf9b-5b0d-4208-9839-8bbf719de427\") " pod="ai-tenant-e2e-shared-a-w3-09b16c/auth-test-model-09b16c-kserve-7df9b7df79-rg87k"
Sep 11 14:36:54.688074 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:54.687898    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7eeaf9b-5b0d-4208-9839-8bbf719de427-model-cache\") pod \"auth-test-model-09b16c-kserve-7df9b7df79-rg87k\" (UID: \"e7eeaf9b-5b0d-4208-9839-8bbf719de427\") " pod="ai-tenant-e2e-shared-a-w3-09b16c/auth-test-model-09b16c-kserve-7df9b7df79-rg87k"
Sep 11 14:36:54.688074 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:54.687931    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7eeaf9b-5b0d-4208-9839-8bbf719de427-dshm\") pod \"auth-test-model-09b16c-kserve-7df9b7df79-rg87k\" (UID: \"e7eeaf9b-5b0d-4208-9839-8bbf719de427\") " pod="ai-tenant-e2e-shared-a-w3-09b16c/auth-test-model-09b16c-kserve-7df9b7df79-rg87k"
Sep 11 14:36:54.688074 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:54.687949    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7eeaf9b-5b0d-4208-9839-8bbf719de427-tmp-dir\") pod \"auth-test-model-09b16c-kserve-7df9b7df79-rg87k\" (UID: \"e7eeaf9b-5b0d-4208-9839-8bbf719de427\") " pod="ai-tenant-e2e-shared-a-w3-09b16c/auth-test-model-09b16c-kserve-7df9b7df79-rg87k"
Sep 11 14:36:54.688074 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:54.687992    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7eeaf9b-5b0d-4208-9839-8bbf719de427-home\") pod \"auth-test-model-09b16c-kserve-7df9b7df79-rg87k\" (UID: \"e7eeaf9b-5b0d-4208-9839-8bbf719de427\") " pod="ai-tenant-e2e-shared-a-w3-09b16c/auth-test-model-09b16c-kserve-7df9b7df79-rg87k"
Sep 11 14:36:54.688074 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:54.688044    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpbg9\" (UniqueName: \"kubernetes.io/projected/e7eeaf9b-5b0d-4208-9839-8bbf719de427-kube-api-access-jpbg9\") pod \"auth-test-model-09b16c-kserve-7df9b7df79-rg87k\" (UID: \"e7eeaf9b-5b0d-4208-9839-8bbf719de427\") " pod="ai-tenant-e2e-shared-a-w3-09b16c/auth-test-model-09b16c-kserve-7df9b7df79-rg87k"
Sep 11 14:36:54.688489 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:54.688466    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7eeaf9b-5b0d-4208-9839-8bbf719de427-tmp-dir\") pod \"auth-test-model-09b16c-kserve-7df9b7df79-rg87k\" (UID: \"e7eeaf9b-5b0d-4208-9839-8bbf719de427\") " pod="ai-tenant-e2e-shared-a-w3-09b16c/auth-test-model-09b16c-kserve-7df9b7df79-rg87k"
Sep 11 14:36:54.688572 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:54.688477    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7eeaf9b-5b0d-4208-9839-8bbf719de427-model-cache\") pod \"auth-test-model-09b16c-kserve-7df9b7df79-rg87k\" (UID: \"e7eeaf9b-5b0d-4208-9839-8bbf719de427\") " pod="ai-tenant-e2e-shared-a-w3-09b16c/auth-test-model-09b16c-kserve-7df9b7df79-rg87k"
Sep 11 14:36:54.688572 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:54.688535    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7eeaf9b-5b0d-4208-9839-8bbf719de427-home\") pod \"auth-test-model-09b16c-kserve-7df9b7df79-rg87k\" (UID: \"e7eeaf9b-5b0d-4208-9839-8bbf719de427\") " pod="ai-tenant-e2e-shared-a-w3-09b16c/auth-test-model-09b16c-kserve-7df9b7df79-rg87k"
Sep 11 14:36:54.690317 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:54.690294    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7eeaf9b-5b0d-4208-9839-8bbf719de427-dshm\") pod \"auth-test-model-09b16c-kserve-7df9b7df79-rg87k\" (UID: \"e7eeaf9b-5b0d-4208-9839-8bbf719de427\") " pod="ai-tenant-e2e-shared-a-w3-09b16c/auth-test-model-09b16c-kserve-7df9b7df79-rg87k"
Sep 11 14:36:54.690590 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:54.690572    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7eeaf9b-5b0d-4208-9839-8bbf719de427-tls-certs\") pod \"auth-test-model-09b16c-kserve-7df9b7df79-rg87k\" (UID: \"e7eeaf9b-5b0d-4208-9839-8bbf719de427\") " pod="ai-tenant-e2e-shared-a-w3-09b16c/auth-test-model-09b16c-kserve-7df9b7df79-rg87k"
Sep 11 14:36:54.701019 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:54.700994    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpbg9\" (UniqueName: \"kubernetes.io/projected/e7eeaf9b-5b0d-4208-9839-8bbf719de427-kube-api-access-jpbg9\") pod \"auth-test-model-09b16c-kserve-7df9b7df79-rg87k\" (UID: \"e7eeaf9b-5b0d-4208-9839-8bbf719de427\") " pod="ai-tenant-e2e-shared-a-w3-09b16c/auth-test-model-09b16c-kserve-7df9b7df79-rg87k"
Sep 11 14:36:54.783959 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:54.783924    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-09b16c/auth-test-model-09b16c-kserve-7df9b7df79-rg87k"
Sep 11 14:36:55.081764 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:55.081673    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-09b16c/auth-test-model-09b16c-kserve-7df9b7df79-rg87k"]
Sep 11 14:36:55.086062 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:36:55.086035    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7eeaf9b_5b0d_4208_9839_8bbf719de427.slice/crio-4d3c8586fac2d4877681ba561adcc6dc44a52dfbcec761f307e5e5d50ea7370b WatchSource:0}: Error finding container 4d3c8586fac2d4877681ba561adcc6dc44a52dfbcec761f307e5e5d50ea7370b: Status 404 returned error can't find the container with id 4d3c8586fac2d4877681ba561adcc6dc44a52dfbcec761f307e5e5d50ea7370b
Sep 11 14:36:55.537750 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:55.537708    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-09b16c/auth-test-model-09b16c-kserve-7df9b7df79-rg87k" event={"ID":"e7eeaf9b-5b0d-4208-9839-8bbf719de427","Type":"ContainerStarted","Data":"91d6d4a269e43c164232ab1e82566dee61efc78d8fa209e1f40b2ed5b3b7cee5"}
Sep 11 14:36:55.538167 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:55.537761    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-09b16c/auth-test-model-09b16c-kserve-7df9b7df79-rg87k" event={"ID":"e7eeaf9b-5b0d-4208-9839-8bbf719de427","Type":"ContainerStarted","Data":"4d3c8586fac2d4877681ba561adcc6dc44a52dfbcec761f307e5e5d50ea7370b"}
Sep 11 14:36:55.567273 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:36:55.567222    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-09b16c/auth-test-model-09b16c-kserve-7df9b7df79-rg87k" podStartSLOduration=1.567205554 podStartE2EDuration="1.567205554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:36:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:36:55.565332115 +0000 UTC m=+1318.715882181" watchObservedRunningTime="2026-09-11 14:36:55.567205554 +0000 UTC m=+1318.717755619"
Sep 11 14:37:04.784490 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:37:04.784454    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-09b16c/auth-test-model-09b16c-kserve-7df9b7df79-rg87k"
Sep 11 14:37:04.785117 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:37:04.784771    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-09b16c/auth-test-model-09b16c-kserve-7df9b7df79-rg87k"
Sep 11 14:37:04.796995 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:37:04.796958    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-09b16c/auth-test-model-09b16c-kserve-7df9b7df79-rg87k"
Sep 11 14:37:05.602188 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:37:05.602157    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-09b16c/auth-test-model-09b16c-kserve-7df9b7df79-rg87k"
Sep 11 14:37:28.545515 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:37:28.545430    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-1c284e/auth-test-model-1c284e-kserve-dbc5df587-ckv8h"]
Sep 11 14:37:28.548947 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:37:28.548926    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1c284e/auth-test-model-1c284e-kserve-dbc5df587-ckv8h"
Sep 11 14:37:28.554097 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:37:28.554074    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-1c284e\"/\"openshift-service-ca.crt\""
Sep 11 14:37:28.554222 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:37:28.554074    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-1c284e\"/\"auth-test-model-1c284e-kserve-self-signed-certs\""
Sep 11 14:37:28.554222 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:37:28.554080    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-1c284e\"/\"kube-root-ca.crt\""
Sep 11 14:37:28.554342 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:37:28.554092    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-1c284e\"/\"default-dockercfg-62xn9\""
Sep 11 14:37:28.568000 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:37:28.567956    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1c284e/auth-test-model-1c284e-kserve-dbc5df587-ckv8h"]
Sep 11 14:37:28.611289 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:37:28.611255    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3ec3968-777d-4d0a-adea-f33d1e43c0b0-home\") pod \"auth-test-model-1c284e-kserve-dbc5df587-ckv8h\" (UID: \"c3ec3968-777d-4d0a-adea-f33d1e43c0b0\") " pod="ai-tenant-e2e-shared-b-w3-1c284e/auth-test-model-1c284e-kserve-dbc5df587-ckv8h"
Sep 11 14:37:28.611519 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:37:28.611298    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb6gt\" (UniqueName: \"kubernetes.io/projected/c3ec3968-777d-4d0a-adea-f33d1e43c0b0-kube-api-access-xb6gt\") pod \"auth-test-model-1c284e-kserve-dbc5df587-ckv8h\" (UID: \"c3ec3968-777d-4d0a-adea-f33d1e43c0b0\") " pod="ai-tenant-e2e-shared-b-w3-1c284e/auth-test-model-1c284e-kserve-dbc5df587-ckv8h"
Sep 11 14:37:28.611616 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:37:28.611554    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3ec3968-777d-4d0a-adea-f33d1e43c0b0-tls-certs\") pod \"auth-test-model-1c284e-kserve-dbc5df587-ckv8h\" (UID: \"c3ec3968-777d-4d0a-adea-f33d1e43c0b0\") " pod="ai-tenant-e2e-shared-b-w3-1c284e/auth-test-model-1c284e-kserve-dbc5df587-ckv8h"
Sep 11 14:37:28.611616 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:37:28.611597    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3ec3968-777d-4d0a-adea-f33d1e43c0b0-model-cache\") pod \"auth-test-model-1c284e-kserve-dbc5df587-ckv8h\" (UID: \"c3ec3968-777d-4d0a-adea-f33d1e43c0b0\") " pod="ai-tenant-e2e-shared-b-w3-1c284e/auth-test-model-1c284e-kserve-dbc5df587-ckv8h"
Sep 11 14:37:28.611800 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:37:28.611778    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3ec3968-777d-4d0a-adea-f33d1e43c0b0-tmp-dir\") pod \"auth-test-model-1c284e-kserve-dbc5df587-ckv8h\" (UID: \"c3ec3968-777d-4d0a-adea-f33d1e43c0b0\") " pod="ai-tenant-e2e-shared-b-w3-1c284e/auth-test-model-1c284e-kserve-dbc5df587-ckv8h"
Sep 11 14:37:28.611943 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:37:28.611869    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3ec3968-777d-4d0a-adea-f33d1e43c0b0-dshm\") pod \"auth-test-model-1c284e-kserve-dbc5df587-ckv8h\" (UID: \"c3ec3968-777d-4d0a-adea-f33d1e43c0b0\") " pod="ai-tenant-e2e-shared-b-w3-1c284e/auth-test-model-1c284e-kserve-dbc5df587-ckv8h"
Sep 11 14:37:28.712876 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:37:28.712843    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3ec3968-777d-4d0a-adea-f33d1e43c0b0-tls-certs\") pod \"auth-test-model-1c284e-kserve-dbc5df587-ckv8h\" (UID: \"c3ec3968-777d-4d0a-adea-f33d1e43c0b0\") " pod="ai-tenant-e2e-shared-b-w3-1c284e/auth-test-model-1c284e-kserve-dbc5df587-ckv8h"
Sep 11 14:37:28.712876 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:37:28.712880    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3ec3968-777d-4d0a-adea-f33d1e43c0b0-model-cache\") pod \"auth-test-model-1c284e-kserve-dbc5df587-ckv8h\" (UID: \"c3ec3968-777d-4d0a-adea-f33d1e43c0b0\") " pod="ai-tenant-e2e-shared-b-w3-1c284e/auth-test-model-1c284e-kserve-dbc5df587-ckv8h"
Sep 11 14:37:28.713151 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:37:28.712900    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3ec3968-777d-4d0a-adea-f33d1e43c0b0-tmp-dir\") pod \"auth-test-model-1c284e-kserve-dbc5df587-ckv8h\" (UID: \"c3ec3968-777d-4d0a-adea-f33d1e43c0b0\") " pod="ai-tenant-e2e-shared-b-w3-1c284e/auth-test-model-1c284e-kserve-dbc5df587-ckv8h"
Sep 11 14:37:28.713151 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:37:28.712937    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3ec3968-777d-4d0a-adea-f33d1e43c0b0-dshm\") pod \"auth-test-model-1c284e-kserve-dbc5df587-ckv8h\" (UID: \"c3ec3968-777d-4d0a-adea-f33d1e43c0b0\") " pod="ai-tenant-e2e-shared-b-w3-1c284e/auth-test-model-1c284e-kserve-dbc5df587-ckv8h"
Sep 11 14:37:28.713151 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:37:28.712984    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3ec3968-777d-4d0a-adea-f33d1e43c0b0-home\") pod \"auth-test-model-1c284e-kserve-dbc5df587-ckv8h\" (UID: \"c3ec3968-777d-4d0a-adea-f33d1e43c0b0\") " pod="ai-tenant-e2e-shared-b-w3-1c284e/auth-test-model-1c284e-kserve-dbc5df587-ckv8h"
Sep 11 14:37:28.713151 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:37:28.713011    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb6gt\" (UniqueName: \"kubernetes.io/projected/c3ec3968-777d-4d0a-adea-f33d1e43c0b0-kube-api-access-xb6gt\") pod \"auth-test-model-1c284e-kserve-dbc5df587-ckv8h\" (UID: \"c3ec3968-777d-4d0a-adea-f33d1e43c0b0\") " pod="ai-tenant-e2e-shared-b-w3-1c284e/auth-test-model-1c284e-kserve-dbc5df587-ckv8h"
Sep 11 14:37:28.713359 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:37:28.713294    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3ec3968-777d-4d0a-adea-f33d1e43c0b0-model-cache\") pod \"auth-test-model-1c284e-kserve-dbc5df587-ckv8h\" (UID: \"c3ec3968-777d-4d0a-adea-f33d1e43c0b0\") " pod="ai-tenant-e2e-shared-b-w3-1c284e/auth-test-model-1c284e-kserve-dbc5df587-ckv8h"
Sep 11 14:37:28.713359 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:37:28.713333    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3ec3968-777d-4d0a-adea-f33d1e43c0b0-tmp-dir\") pod \"auth-test-model-1c284e-kserve-dbc5df587-ckv8h\" (UID: \"c3ec3968-777d-4d0a-adea-f33d1e43c0b0\") " pod="ai-tenant-e2e-shared-b-w3-1c284e/auth-test-model-1c284e-kserve-dbc5df587-ckv8h"
Sep 11 14:37:28.713462 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:37:28.713395    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3ec3968-777d-4d0a-adea-f33d1e43c0b0-home\") pod \"auth-test-model-1c284e-kserve-dbc5df587-ckv8h\" (UID: \"c3ec3968-777d-4d0a-adea-f33d1e43c0b0\") " pod="ai-tenant-e2e-shared-b-w3-1c284e/auth-test-model-1c284e-kserve-dbc5df587-ckv8h"
Sep 11 14:37:28.715294 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:37:28.715270    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3ec3968-777d-4d0a-adea-f33d1e43c0b0-dshm\") pod \"auth-test-model-1c284e-kserve-dbc5df587-ckv8h\" (UID: \"c3ec3968-777d-4d0a-adea-f33d1e43c0b0\") " pod="ai-tenant-e2e-shared-b-w3-1c284e/auth-test-model-1c284e-kserve-dbc5df587-ckv8h"
Sep 11 14:37:28.715406 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:37:28.715372    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3ec3968-777d-4d0a-adea-f33d1e43c0b0-tls-certs\") pod \"auth-test-model-1c284e-kserve-dbc5df587-ckv8h\" (UID: \"c3ec3968-777d-4d0a-adea-f33d1e43c0b0\") " pod="ai-tenant-e2e-shared-b-w3-1c284e/auth-test-model-1c284e-kserve-dbc5df587-ckv8h"
Sep 11 14:37:28.727979 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:37:28.727954    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb6gt\" (UniqueName: \"kubernetes.io/projected/c3ec3968-777d-4d0a-adea-f33d1e43c0b0-kube-api-access-xb6gt\") pod \"auth-test-model-1c284e-kserve-dbc5df587-ckv8h\" (UID: \"c3ec3968-777d-4d0a-adea-f33d1e43c0b0\") " pod="ai-tenant-e2e-shared-b-w3-1c284e/auth-test-model-1c284e-kserve-dbc5df587-ckv8h"
Sep 11 14:37:28.858233 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:37:28.858135    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1c284e/auth-test-model-1c284e-kserve-dbc5df587-ckv8h"
Sep 11 14:37:29.144979 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:37:29.144942    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1c284e/auth-test-model-1c284e-kserve-dbc5df587-ckv8h"]
Sep 11 14:37:29.146609 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:37:29.146578    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc3ec3968_777d_4d0a_adea_f33d1e43c0b0.slice/crio-81ef3516dc389b529cf11886a28de871df88e8b79399879c387b19f4f9df84c7 WatchSource:0}: Error finding container 81ef3516dc389b529cf11886a28de871df88e8b79399879c387b19f4f9df84c7: Status 404 returned error can't find the container with id 81ef3516dc389b529cf11886a28de871df88e8b79399879c387b19f4f9df84c7
Sep 11 14:37:29.718739 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:37:29.718695    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1c284e/auth-test-model-1c284e-kserve-dbc5df587-ckv8h" event={"ID":"c3ec3968-777d-4d0a-adea-f33d1e43c0b0","Type":"ContainerStarted","Data":"407410ae258e24f15d80e68f00729430cdf9bc9a9a31844a921aed46db4841e0"}
Sep 11 14:37:29.718739 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:37:29.718747    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1c284e/auth-test-model-1c284e-kserve-dbc5df587-ckv8h" event={"ID":"c3ec3968-777d-4d0a-adea-f33d1e43c0b0","Type":"ContainerStarted","Data":"81ef3516dc389b529cf11886a28de871df88e8b79399879c387b19f4f9df84c7"}
Sep 11 14:37:29.744461 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:37:29.744409    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-1c284e/auth-test-model-1c284e-kserve-dbc5df587-ckv8h" podStartSLOduration=1.744391944 podStartE2EDuration="1.744391944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:37:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:37:29.743824313 +0000 UTC m=+1352.894374378" watchObservedRunningTime="2026-09-11 14:37:29.744391944 +0000 UTC m=+1352.894942007"
Sep 11 14:37:38.858575 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:37:38.858537    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-1c284e/auth-test-model-1c284e-kserve-dbc5df587-ckv8h"
Sep 11 14:37:38.859367 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:37:38.858590    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-1c284e/auth-test-model-1c284e-kserve-dbc5df587-ckv8h"
Sep 11 14:37:38.871134 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:37:38.871108    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-1c284e/auth-test-model-1c284e-kserve-dbc5df587-ckv8h"
Sep 11 14:37:39.782749 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:37:39.782720    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-1c284e/auth-test-model-1c284e-kserve-dbc5df587-ckv8h"
Sep 11 14:37:48.891136 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:37:48.891097    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-455f2e-85b7989885-2qzkr"]
Sep 11 14:37:48.894903 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:37:48.894884    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-455f2e-85b7989885-2qzkr"
Sep 11 14:37:48.897731 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:37:48.897705    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-455f2e\""
Sep 11 14:37:48.897731 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:37:48.897720    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-455f2e-dockercfg-szvql\""
Sep 11 14:37:48.908438 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:37:48.908400    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-455f2e-85b7989885-2qzkr"]
Sep 11 14:37:49.011449 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:37:49.011414    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb4f6189-03d0-4410-86f7-f5438a766546-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-455f2e-85b7989885-2qzkr\" (UID: \"cb4f6189-03d0-4410-86f7-f5438a766546\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-455f2e-85b7989885-2qzkr"
Sep 11 14:37:49.011614 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:37:49.011462    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz5bj\" (UniqueName: \"kubernetes.io/projected/cb4f6189-03d0-4410-86f7-f5438a766546-kube-api-access-fz5bj\") pod \"payload-processing-e2e-shared-b-w2-455f2e-85b7989885-2qzkr\" (UID: \"cb4f6189-03d0-4410-86f7-f5438a766546\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-455f2e-85b7989885-2qzkr"
Sep 11 14:37:49.112605 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:37:49.112550    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz5bj\" (UniqueName: \"kubernetes.io/projected/cb4f6189-03d0-4410-86f7-f5438a766546-kube-api-access-fz5bj\") pod \"payload-processing-e2e-shared-b-w2-455f2e-85b7989885-2qzkr\" (UID: \"cb4f6189-03d0-4410-86f7-f5438a766546\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-455f2e-85b7989885-2qzkr"
Sep 11 14:37:49.112806 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:37:49.112694    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb4f6189-03d0-4410-86f7-f5438a766546-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-455f2e-85b7989885-2qzkr\" (UID: \"cb4f6189-03d0-4410-86f7-f5438a766546\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-455f2e-85b7989885-2qzkr"
Sep 11 14:37:49.119078 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:37:49.113918    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb4f6189-03d0-4410-86f7-f5438a766546-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-455f2e-85b7989885-2qzkr\" (UID: \"cb4f6189-03d0-4410-86f7-f5438a766546\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-455f2e-85b7989885-2qzkr"
Sep 11 14:37:49.124772 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:37:49.124742    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz5bj\" (UniqueName: \"kubernetes.io/projected/cb4f6189-03d0-4410-86f7-f5438a766546-kube-api-access-fz5bj\") pod \"payload-processing-e2e-shared-b-w2-455f2e-85b7989885-2qzkr\" (UID: \"cb4f6189-03d0-4410-86f7-f5438a766546\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-455f2e-85b7989885-2qzkr"
Sep 11 14:37:49.210036 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:37:49.209902    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-455f2e-85b7989885-2qzkr"
Sep 11 14:37:49.496946 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:37:49.496914    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb4f6189_03d0_4410_86f7_f5438a766546.slice/crio-4e5df498c931ec004127cd3797fe88f8808725acc9514f5479da58ffe6ffa4ef WatchSource:0}: Error finding container 4e5df498c931ec004127cd3797fe88f8808725acc9514f5479da58ffe6ffa4ef: Status 404 returned error can't find the container with id 4e5df498c931ec004127cd3797fe88f8808725acc9514f5479da58ffe6ffa4ef
Sep 11 14:37:49.499797 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:37:49.499774    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-455f2e-85b7989885-2qzkr"]
Sep 11 14:37:49.825672 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:37:49.825566    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-455f2e-85b7989885-2qzkr" event={"ID":"cb4f6189-03d0-4410-86f7-f5438a766546","Type":"ContainerStarted","Data":"8d0490d6f6a432a667c90f78cf436971e9e75c01ff4bdce416ebe901b3496d94"}
Sep 11 14:37:49.825672 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:37:49.825620    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-455f2e-85b7989885-2qzkr" event={"ID":"cb4f6189-03d0-4410-86f7-f5438a766546","Type":"ContainerStarted","Data":"4e5df498c931ec004127cd3797fe88f8808725acc9514f5479da58ffe6ffa4ef"}
Sep 11 14:37:49.825672 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:37:49.825646    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-455f2e-85b7989885-2qzkr"
Sep 11 14:37:49.852330 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:37:49.852275    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-455f2e-85b7989885-2qzkr" podStartSLOduration=1.852256183 podStartE2EDuration="1.852256183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:37:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:37:49.850960257 +0000 UTC m=+1373.001510318" watchObservedRunningTime="2026-09-11 14:37:49.852256183 +0000 UTC m=+1373.002806225"
Sep 11 14:37:57.473534 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:37:57.473505    2581 scope.go:117] "RemoveContainer" containerID="9d06538d8c5d02b135a3bb4da31d70f2e0d44ed2da2e24a98ec5339d909b65b2"
Sep 11 14:38:00.833739 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:38:00.833706    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-455f2e-85b7989885-2qzkr"
Sep 11 14:38:33.031545 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:38:33.031506    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2b947f70-74c7f8966-lqjdn"]
Sep 11 14:38:33.035020 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:38:33.035002    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b947f70-74c7f8966-lqjdn"
Sep 11 14:38:33.038848 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:38:33.038826    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-2b947f70\""
Sep 11 14:38:33.054924 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:38:33.054897    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2b947f70-74c7f8966-lqjdn"]
Sep 11 14:38:33.150229 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:38:33.150189    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da8e262b-3b19-49fb-80b0-1eda4b67fd8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2b947f70-74c7f8966-lqjdn\" (UID: \"da8e262b-3b19-49fb-80b0-1eda4b67fd8e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2b947f70-74c7f8966-lqjdn"
Sep 11 14:38:33.150402 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:38:33.150273    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s285p\" (UniqueName: \"kubernetes.io/projected/da8e262b-3b19-49fb-80b0-1eda4b67fd8e-kube-api-access-s285p\") pod \"payload-pre-processing-e2e-mt-2b947f70-74c7f8966-lqjdn\" (UID: \"da8e262b-3b19-49fb-80b0-1eda4b67fd8e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2b947f70-74c7f8966-lqjdn"
Sep 11 14:38:33.251783 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:38:33.251743    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da8e262b-3b19-49fb-80b0-1eda4b67fd8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2b947f70-74c7f8966-lqjdn\" (UID: \"da8e262b-3b19-49fb-80b0-1eda4b67fd8e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2b947f70-74c7f8966-lqjdn"
Sep 11 14:38:33.251992 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:38:33.251802    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s285p\" (UniqueName: \"kubernetes.io/projected/da8e262b-3b19-49fb-80b0-1eda4b67fd8e-kube-api-access-s285p\") pod \"payload-pre-processing-e2e-mt-2b947f70-74c7f8966-lqjdn\" (UID: \"da8e262b-3b19-49fb-80b0-1eda4b67fd8e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2b947f70-74c7f8966-lqjdn"
Sep 11 14:38:33.252524 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:38:33.252504    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da8e262b-3b19-49fb-80b0-1eda4b67fd8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2b947f70-74c7f8966-lqjdn\" (UID: \"da8e262b-3b19-49fb-80b0-1eda4b67fd8e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2b947f70-74c7f8966-lqjdn"
Sep 11 14:38:33.263137 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:38:33.263111    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s285p\" (UniqueName: \"kubernetes.io/projected/da8e262b-3b19-49fb-80b0-1eda4b67fd8e-kube-api-access-s285p\") pod \"payload-pre-processing-e2e-mt-2b947f70-74c7f8966-lqjdn\" (UID: \"da8e262b-3b19-49fb-80b0-1eda4b67fd8e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2b947f70-74c7f8966-lqjdn"
Sep 11 14:38:33.345078 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:38:33.344990    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b947f70-74c7f8966-lqjdn"
Sep 11 14:38:33.640965 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:38:33.640935    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2b947f70-74c7f8966-lqjdn"]
Sep 11 14:38:33.643488 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:38:33.643460    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda8e262b_3b19_49fb_80b0_1eda4b67fd8e.slice/crio-7737aa6650dade24afb5b04968b89ea7544deaee151840ca6dea5c7f6ef29626 WatchSource:0}: Error finding container 7737aa6650dade24afb5b04968b89ea7544deaee151840ca6dea5c7f6ef29626: Status 404 returned error can't find the container with id 7737aa6650dade24afb5b04968b89ea7544deaee151840ca6dea5c7f6ef29626
Sep 11 14:38:34.064288 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:38:34.064244    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b947f70-74c7f8966-lqjdn" event={"ID":"da8e262b-3b19-49fb-80b0-1eda4b67fd8e","Type":"ContainerStarted","Data":"8f774a65ded438b95ac1f3f1aa94b518767c756337693e79a5258eeb7c5bd4dd"}
Sep 11 14:38:34.064709 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:38:34.064313    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b947f70-74c7f8966-lqjdn" event={"ID":"da8e262b-3b19-49fb-80b0-1eda4b67fd8e","Type":"ContainerStarted","Data":"7737aa6650dade24afb5b04968b89ea7544deaee151840ca6dea5c7f6ef29626"}
Sep 11 14:38:34.064709 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:38:34.064339    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b947f70-74c7f8966-lqjdn"
Sep 11 14:38:34.093213 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:38:34.093159    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b947f70-74c7f8966-lqjdn" podStartSLOduration=1.093141245 podStartE2EDuration="1.093141245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:38:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:38:34.091716744 +0000 UTC m=+1417.242266807" watchObservedRunningTime="2026-09-11 14:38:34.093141245 +0000 UTC m=+1417.243691309"
Sep 11 14:38:45.072426 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:38:45.072392    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b947f70-74c7f8966-lqjdn"
Sep 11 14:38:57.495577 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:38:57.495490    2581 scope.go:117] "RemoveContainer" containerID="768d9479b2f64b80b09c9c97d92d1fda8c8a93b3fcaef1e645c0f3ac1fe62119"
Sep 11 14:38:57.504956 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:38:57.504931    2581 scope.go:117] "RemoveContainer" containerID="adef2127a688f6e6dfd01ae0818d48a64384ef124836c1691f4273dc4052a22b"
Sep 11 14:39:10.032756 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:39:10.032721    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-2b947f70/e2e-lifecycle-model-2b947f70-kserve-7f89bffb87-mr7nc"]
Sep 11 14:39:10.037481 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:39:10.037463    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2b947f70/e2e-lifecycle-model-2b947f70-kserve-7f89bffb87-mr7nc"
Sep 11 14:39:10.042355 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:39:10.042333    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-2b947f70\"/\"openshift-service-ca.crt\""
Sep 11 14:39:10.042467 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:39:10.042409    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-2b947f70\"/\"kube-root-ca.crt\""
Sep 11 14:39:10.042467 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:39:10.042450    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-2b947f70\"/\"e2e-lifecycle-model-2b947f70-kserve-self-signed-certs\""
Sep 11 14:39:10.042538 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:39:10.042528    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-2b947f70\"/\"default-dockercfg-rllh8\""
Sep 11 14:39:10.054769 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:39:10.054745    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-2b947f70/e2e-lifecycle-model-2b947f70-kserve-7f89bffb87-mr7nc"]
Sep 11 14:39:10.209158 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:39:10.209127    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/658f5de9-c6e7-41f2-b46e-316e9812a56f-dshm\") pod \"e2e-lifecycle-model-2b947f70-kserve-7f89bffb87-mr7nc\" (UID: \"658f5de9-c6e7-41f2-b46e-316e9812a56f\") " pod="ai-tenant-e2e-mt-2b947f70/e2e-lifecycle-model-2b947f70-kserve-7f89bffb87-mr7nc"
Sep 11 14:39:10.209158 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:39:10.209160    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/658f5de9-c6e7-41f2-b46e-316e9812a56f-model-cache\") pod \"e2e-lifecycle-model-2b947f70-kserve-7f89bffb87-mr7nc\" (UID: \"658f5de9-c6e7-41f2-b46e-316e9812a56f\") " pod="ai-tenant-e2e-mt-2b947f70/e2e-lifecycle-model-2b947f70-kserve-7f89bffb87-mr7nc"
Sep 11 14:39:10.209398 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:39:10.209181    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/658f5de9-c6e7-41f2-b46e-316e9812a56f-tls-certs\") pod \"e2e-lifecycle-model-2b947f70-kserve-7f89bffb87-mr7nc\" (UID: \"658f5de9-c6e7-41f2-b46e-316e9812a56f\") " pod="ai-tenant-e2e-mt-2b947f70/e2e-lifecycle-model-2b947f70-kserve-7f89bffb87-mr7nc"
Sep 11 14:39:10.209398 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:39:10.209323    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/658f5de9-c6e7-41f2-b46e-316e9812a56f-home\") pod \"e2e-lifecycle-model-2b947f70-kserve-7f89bffb87-mr7nc\" (UID: \"658f5de9-c6e7-41f2-b46e-316e9812a56f\") " pod="ai-tenant-e2e-mt-2b947f70/e2e-lifecycle-model-2b947f70-kserve-7f89bffb87-mr7nc"
Sep 11 14:39:10.209398 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:39:10.209354    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlgft\" (UniqueName: \"kubernetes.io/projected/658f5de9-c6e7-41f2-b46e-316e9812a56f-kube-api-access-dlgft\") pod \"e2e-lifecycle-model-2b947f70-kserve-7f89bffb87-mr7nc\" (UID: \"658f5de9-c6e7-41f2-b46e-316e9812a56f\") " pod="ai-tenant-e2e-mt-2b947f70/e2e-lifecycle-model-2b947f70-kserve-7f89bffb87-mr7nc"
Sep 11 14:39:10.209522 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:39:10.209416    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/658f5de9-c6e7-41f2-b46e-316e9812a56f-tmp-dir\") pod \"e2e-lifecycle-model-2b947f70-kserve-7f89bffb87-mr7nc\" (UID: \"658f5de9-c6e7-41f2-b46e-316e9812a56f\") " pod="ai-tenant-e2e-mt-2b947f70/e2e-lifecycle-model-2b947f70-kserve-7f89bffb87-mr7nc"
Sep 11 14:39:10.310466 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:39:10.310379    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/658f5de9-c6e7-41f2-b46e-316e9812a56f-tmp-dir\") pod \"e2e-lifecycle-model-2b947f70-kserve-7f89bffb87-mr7nc\" (UID: \"658f5de9-c6e7-41f2-b46e-316e9812a56f\") " pod="ai-tenant-e2e-mt-2b947f70/e2e-lifecycle-model-2b947f70-kserve-7f89bffb87-mr7nc"
Sep 11 14:39:10.310466 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:39:10.310449    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/658f5de9-c6e7-41f2-b46e-316e9812a56f-dshm\") pod \"e2e-lifecycle-model-2b947f70-kserve-7f89bffb87-mr7nc\" (UID: \"658f5de9-c6e7-41f2-b46e-316e9812a56f\") " pod="ai-tenant-e2e-mt-2b947f70/e2e-lifecycle-model-2b947f70-kserve-7f89bffb87-mr7nc"
Sep 11 14:39:10.310681 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:39:10.310471    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/658f5de9-c6e7-41f2-b46e-316e9812a56f-model-cache\") pod \"e2e-lifecycle-model-2b947f70-kserve-7f89bffb87-mr7nc\" (UID: \"658f5de9-c6e7-41f2-b46e-316e9812a56f\") " pod="ai-tenant-e2e-mt-2b947f70/e2e-lifecycle-model-2b947f70-kserve-7f89bffb87-mr7nc"
Sep 11 14:39:10.310681 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:39:10.310490    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/658f5de9-c6e7-41f2-b46e-316e9812a56f-tls-certs\") pod \"e2e-lifecycle-model-2b947f70-kserve-7f89bffb87-mr7nc\" (UID: \"658f5de9-c6e7-41f2-b46e-316e9812a56f\") " pod="ai-tenant-e2e-mt-2b947f70/e2e-lifecycle-model-2b947f70-kserve-7f89bffb87-mr7nc"
Sep 11 14:39:10.310681 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:39:10.310567    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/658f5de9-c6e7-41f2-b46e-316e9812a56f-home\") pod \"e2e-lifecycle-model-2b947f70-kserve-7f89bffb87-mr7nc\" (UID: \"658f5de9-c6e7-41f2-b46e-316e9812a56f\") " pod="ai-tenant-e2e-mt-2b947f70/e2e-lifecycle-model-2b947f70-kserve-7f89bffb87-mr7nc"
Sep 11 14:39:10.310681 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:39:10.310596    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlgft\" (UniqueName: \"kubernetes.io/projected/658f5de9-c6e7-41f2-b46e-316e9812a56f-kube-api-access-dlgft\") pod \"e2e-lifecycle-model-2b947f70-kserve-7f89bffb87-mr7nc\" (UID: \"658f5de9-c6e7-41f2-b46e-316e9812a56f\") " pod="ai-tenant-e2e-mt-2b947f70/e2e-lifecycle-model-2b947f70-kserve-7f89bffb87-mr7nc"
Sep 11 14:39:10.310901 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:39:10.310830    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/658f5de9-c6e7-41f2-b46e-316e9812a56f-tmp-dir\") pod \"e2e-lifecycle-model-2b947f70-kserve-7f89bffb87-mr7nc\" (UID: \"658f5de9-c6e7-41f2-b46e-316e9812a56f\") " pod="ai-tenant-e2e-mt-2b947f70/e2e-lifecycle-model-2b947f70-kserve-7f89bffb87-mr7nc"
Sep 11 14:39:10.310901 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:39:10.310853    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/658f5de9-c6e7-41f2-b46e-316e9812a56f-model-cache\") pod \"e2e-lifecycle-model-2b947f70-kserve-7f89bffb87-mr7nc\" (UID: \"658f5de9-c6e7-41f2-b46e-316e9812a56f\") " pod="ai-tenant-e2e-mt-2b947f70/e2e-lifecycle-model-2b947f70-kserve-7f89bffb87-mr7nc"
Sep 11 14:39:10.311052 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:39:10.311010    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/658f5de9-c6e7-41f2-b46e-316e9812a56f-home\") pod \"e2e-lifecycle-model-2b947f70-kserve-7f89bffb87-mr7nc\" (UID: \"658f5de9-c6e7-41f2-b46e-316e9812a56f\") " pod="ai-tenant-e2e-mt-2b947f70/e2e-lifecycle-model-2b947f70-kserve-7f89bffb87-mr7nc"
Sep 11 14:39:10.312766 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:39:10.312739    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/658f5de9-c6e7-41f2-b46e-316e9812a56f-dshm\") pod \"e2e-lifecycle-model-2b947f70-kserve-7f89bffb87-mr7nc\" (UID: \"658f5de9-c6e7-41f2-b46e-316e9812a56f\") " pod="ai-tenant-e2e-mt-2b947f70/e2e-lifecycle-model-2b947f70-kserve-7f89bffb87-mr7nc"
Sep 11 14:39:10.313067 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:39:10.313049    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/658f5de9-c6e7-41f2-b46e-316e9812a56f-tls-certs\") pod \"e2e-lifecycle-model-2b947f70-kserve-7f89bffb87-mr7nc\" (UID: \"658f5de9-c6e7-41f2-b46e-316e9812a56f\") " pod="ai-tenant-e2e-mt-2b947f70/e2e-lifecycle-model-2b947f70-kserve-7f89bffb87-mr7nc"
Sep 11 14:39:10.323050 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:39:10.323028    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlgft\" (UniqueName: \"kubernetes.io/projected/658f5de9-c6e7-41f2-b46e-316e9812a56f-kube-api-access-dlgft\") pod \"e2e-lifecycle-model-2b947f70-kserve-7f89bffb87-mr7nc\" (UID: \"658f5de9-c6e7-41f2-b46e-316e9812a56f\") " pod="ai-tenant-e2e-mt-2b947f70/e2e-lifecycle-model-2b947f70-kserve-7f89bffb87-mr7nc"
Sep 11 14:39:10.346947 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:39:10.346912    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2b947f70/e2e-lifecycle-model-2b947f70-kserve-7f89bffb87-mr7nc"
Sep 11 14:39:10.623417 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:39:10.623394    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-2b947f70/e2e-lifecycle-model-2b947f70-kserve-7f89bffb87-mr7nc"]
Sep 11 14:39:10.625810 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:39:10.625779    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod658f5de9_c6e7_41f2_b46e_316e9812a56f.slice/crio-42636f499fde3796b73674ba45a6fe06f9e83dba79afa8f10299dea77bde2965 WatchSource:0}: Error finding container 42636f499fde3796b73674ba45a6fe06f9e83dba79afa8f10299dea77bde2965: Status 404 returned error can't find the container with id 42636f499fde3796b73674ba45a6fe06f9e83dba79afa8f10299dea77bde2965
Sep 11 14:39:11.261759 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:39:11.261717    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2b947f70/e2e-lifecycle-model-2b947f70-kserve-7f89bffb87-mr7nc" event={"ID":"658f5de9-c6e7-41f2-b46e-316e9812a56f","Type":"ContainerStarted","Data":"7b1e52e3791189a2a3287800635550373b0700a48f8a65360bad841f985d6d26"}
Sep 11 14:39:11.262190 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:39:11.261769    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2b947f70/e2e-lifecycle-model-2b947f70-kserve-7f89bffb87-mr7nc" event={"ID":"658f5de9-c6e7-41f2-b46e-316e9812a56f","Type":"ContainerStarted","Data":"42636f499fde3796b73674ba45a6fe06f9e83dba79afa8f10299dea77bde2965"}
Sep 11 14:39:11.293253 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:39:11.293200    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-2b947f70/e2e-lifecycle-model-2b947f70-kserve-7f89bffb87-mr7nc" podStartSLOduration=1.29318393 podStartE2EDuration="1.29318393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:39:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:39:11.291479353 +0000 UTC m=+1454.442029416" watchObservedRunningTime="2026-09-11 14:39:11.29318393 +0000 UTC m=+1454.443733994"
Sep 11 14:39:20.347517 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:39:20.347482    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-2b947f70/e2e-lifecycle-model-2b947f70-kserve-7f89bffb87-mr7nc"
Sep 11 14:39:20.347999 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:39:20.347629    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-2b947f70/e2e-lifecycle-model-2b947f70-kserve-7f89bffb87-mr7nc"
Sep 11 14:39:20.360089 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:39:20.360063    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-2b947f70/e2e-lifecycle-model-2b947f70-kserve-7f89bffb87-mr7nc"
Sep 11 14:39:21.323122 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:39:21.323094    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-2b947f70/e2e-lifecycle-model-2b947f70-kserve-7f89bffb87-mr7nc"
Sep 11 14:39:52.980298 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:39:52.980258    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b947f70-dtm4n"]
Sep 11 14:39:52.983805 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:39:52.983786    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b947f70-dtm4n"
Sep 11 14:39:52.986765 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:39:52.986740    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hcm4g\""
Sep 11 14:39:52.986765 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:39:52.986756    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 14:39:52.996696 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:39:52.996665    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b947f70-dtm4n"]
Sep 11 14:39:53.124438 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:39:53.124396    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f404627b-5b49-41c5-955b-74dda6975a35-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2b947f70-dtm4n\" (UID: \"f404627b-5b49-41c5-955b-74dda6975a35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b947f70-dtm4n"
Sep 11 14:39:53.225434 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:39:53.225401    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f404627b-5b49-41c5-955b-74dda6975a35-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2b947f70-dtm4n\" (UID: \"f404627b-5b49-41c5-955b-74dda6975a35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b947f70-dtm4n"
Sep 11 14:39:53.226176 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:39:53.226149    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f404627b-5b49-41c5-955b-74dda6975a35-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2b947f70-dtm4n\" (UID: \"f404627b-5b49-41c5-955b-74dda6975a35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b947f70-dtm4n"
Sep 11 14:39:53.294165 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:39:53.294122    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b947f70-dtm4n"
Sep 11 14:39:53.568918 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:39:53.568891    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b947f70-dtm4n"]
Sep 11 14:39:53.570802 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:39:53.570772    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf404627b_5b49_41c5_955b_74dda6975a35.slice/crio-22a4799ab1ceb06e0b2ab17d759e62fb6525b413fb8fdd15d90362c9579bb682 WatchSource:0}: Error finding container 22a4799ab1ceb06e0b2ab17d759e62fb6525b413fb8fdd15d90362c9579bb682: Status 404 returned error can't find the container with id 22a4799ab1ceb06e0b2ab17d759e62fb6525b413fb8fdd15d90362c9579bb682
Sep 11 14:39:54.483578 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:39:54.483480    2581 generic.go:356] "Generic (PLEG): container finished" podID="f404627b-5b49-41c5-955b-74dda6975a35" containerID="67f304e08cb6fe9c8c76f51198253c3690104442744021aec9400d4ddad4e963" exitCode=0
Sep 11 14:39:54.483578 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:39:54.483526    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b947f70-dtm4n" event={"ID":"f404627b-5b49-41c5-955b-74dda6975a35","Type":"ContainerDied","Data":"67f304e08cb6fe9c8c76f51198253c3690104442744021aec9400d4ddad4e963"}
Sep 11 14:39:54.484061 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:39:54.483583    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b947f70-dtm4n" event={"ID":"f404627b-5b49-41c5-955b-74dda6975a35","Type":"ContainerStarted","Data":"22a4799ab1ceb06e0b2ab17d759e62fb6525b413fb8fdd15d90362c9579bb682"}
Sep 11 14:39:55.627534 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:39:55.627507    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b947f70-dtm4n"
Sep 11 14:39:55.647731 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:39:55.647700    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f404627b-5b49-41c5-955b-74dda6975a35-maas-api-service-ca\") pod \"f404627b-5b49-41c5-955b-74dda6975a35\" (UID: \"f404627b-5b49-41c5-955b-74dda6975a35\") "
Sep 11 14:39:55.650086 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:39:55.648586    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f404627b-5b49-41c5-955b-74dda6975a35-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f404627b-5b49-41c5-955b-74dda6975a35" (UID: "f404627b-5b49-41c5-955b-74dda6975a35"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:39:55.749841 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:39:55.749805    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f404627b-5b49-41c5-955b-74dda6975a35-maas-api-service-ca\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:39:56.494732 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:39:56.494693    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b947f70-dtm4n" event={"ID":"f404627b-5b49-41c5-955b-74dda6975a35","Type":"ContainerDied","Data":"22a4799ab1ceb06e0b2ab17d759e62fb6525b413fb8fdd15d90362c9579bb682"}
Sep 11 14:39:56.494942 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:39:56.494734    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b947f70-dtm4n"
Sep 11 14:39:56.494942 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:39:56.494741    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22a4799ab1ceb06e0b2ab17d759e62fb6525b413fb8fdd15d90362c9579bb682"
Sep 11 14:39:57.531747 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:39:57.531713    2581 scope.go:117] "RemoveContainer" containerID="e9a4580fe27c714b8b3a3d21b9edb5303fe110d46fd1b316c9ae14ce9c99d220"
Sep 11 14:39:57.540181 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:39:57.540158    2581 scope.go:117] "RemoveContainer" containerID="f0ff2ac905bd6941bd1ab4570b5b55ece4f310a6006268e70d84aa2808f2be81"
Sep 11 14:40:03.505929 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:03.505897    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-09b16c/auth-test-model-09b16c-kserve-7df9b7df79-rg87k"]
Sep 11 14:40:04.694536 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:04.694502    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1c284e/auth-test-model-1c284e-kserve-dbc5df587-ckv8h"]
Sep 11 14:40:09.329227 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:09.329196    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2b947f70-74c7f8966-lqjdn"]
Sep 11 14:40:09.329682 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:09.329427    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b947f70-74c7f8966-lqjdn" podUID="da8e262b-3b19-49fb-80b0-1eda4b67fd8e" containerName="payload-pre-processing" containerID="cri-o://8f774a65ded438b95ac1f3f1aa94b518767c756337693e79a5258eeb7c5bd4dd" gracePeriod=30
Sep 11 14:40:09.563437 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:09.563407    2581 generic.go:356] "Generic (PLEG): container finished" podID="da8e262b-3b19-49fb-80b0-1eda4b67fd8e" containerID="8f774a65ded438b95ac1f3f1aa94b518767c756337693e79a5258eeb7c5bd4dd" exitCode=0
Sep 11 14:40:09.563574 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:09.563481    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b947f70-74c7f8966-lqjdn" event={"ID":"da8e262b-3b19-49fb-80b0-1eda4b67fd8e","Type":"ContainerDied","Data":"8f774a65ded438b95ac1f3f1aa94b518767c756337693e79a5258eeb7c5bd4dd"}
Sep 11 14:40:09.598679 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:09.598096    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b947f70-74c7f8966-lqjdn"
Sep 11 14:40:09.688069 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:09.688037    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b947f70-dtm4n"]
Sep 11 14:40:09.691263 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:09.691229    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da8e262b-3b19-49fb-80b0-1eda4b67fd8e-plugins-config-volume\") pod \"da8e262b-3b19-49fb-80b0-1eda4b67fd8e\" (UID: \"da8e262b-3b19-49fb-80b0-1eda4b67fd8e\") "
Sep 11 14:40:09.691465 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:09.691319    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s285p\" (UniqueName: \"kubernetes.io/projected/da8e262b-3b19-49fb-80b0-1eda4b67fd8e-kube-api-access-s285p\") pod \"da8e262b-3b19-49fb-80b0-1eda4b67fd8e\" (UID: \"da8e262b-3b19-49fb-80b0-1eda4b67fd8e\") "
Sep 11 14:40:09.691681 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:09.691653    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da8e262b-3b19-49fb-80b0-1eda4b67fd8e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "da8e262b-3b19-49fb-80b0-1eda4b67fd8e" (UID: "da8e262b-3b19-49fb-80b0-1eda4b67fd8e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:40:09.691784 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:09.691741    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da8e262b-3b19-49fb-80b0-1eda4b67fd8e-plugins-config-volume\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:40:09.693649 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:09.693620    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da8e262b-3b19-49fb-80b0-1eda4b67fd8e-kube-api-access-s285p" (OuterVolumeSpecName: "kube-api-access-s285p") pod "da8e262b-3b19-49fb-80b0-1eda4b67fd8e" (UID: "da8e262b-3b19-49fb-80b0-1eda4b67fd8e"). InnerVolumeSpecName "kube-api-access-s285p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:40:09.693872 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:09.693852    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b947f70-dtm4n"]
Sep 11 14:40:09.792885 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:09.792847    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s285p\" (UniqueName: \"kubernetes.io/projected/da8e262b-3b19-49fb-80b0-1eda4b67fd8e-kube-api-access-s285p\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:40:10.570279 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:10.570243    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b947f70-74c7f8966-lqjdn" event={"ID":"da8e262b-3b19-49fb-80b0-1eda4b67fd8e","Type":"ContainerDied","Data":"7737aa6650dade24afb5b04968b89ea7544deaee151840ca6dea5c7f6ef29626"}
Sep 11 14:40:10.570279 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:10.570276    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b947f70-74c7f8966-lqjdn"
Sep 11 14:40:10.570783 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:10.570318    2581 scope.go:117] "RemoveContainer" containerID="8f774a65ded438b95ac1f3f1aa94b518767c756337693e79a5258eeb7c5bd4dd"
Sep 11 14:40:10.602792 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:10.602761    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2b947f70-74c7f8966-lqjdn"]
Sep 11 14:40:10.612922 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:10.612888    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2b947f70-74c7f8966-lqjdn"]
Sep 11 14:40:11.503745 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:11.503705    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da8e262b-3b19-49fb-80b0-1eda4b67fd8e" path="/var/lib/kubelet/pods/da8e262b-3b19-49fb-80b0-1eda4b67fd8e/volumes"
Sep 11 14:40:11.504298 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:11.504269    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f404627b-5b49-41c5-955b-74dda6975a35" path="/var/lib/kubelet/pods/f404627b-5b49-41c5-955b-74dda6975a35/volumes"
Sep 11 14:40:18.453122 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:18.453084    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-2b947f70/e2e-lifecycle-model-2b947f70-kserve-7f89bffb87-mr7nc"]
Sep 11 14:40:18.524735 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:18.524699    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-09b16c/auth-test-model-09b16c-kserve-7df9b7df79-rg87k" podUID="e7eeaf9b-5b0d-4208-9839-8bbf719de427" containerName="main" containerID="cri-o://91d6d4a269e43c164232ab1e82566dee61efc78d8fa209e1f40b2ed5b3b7cee5" gracePeriod=45
Sep 11 14:40:18.773059 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:18.773026    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-09b16c/auth-test-model-09b16c-kserve-7df9b7df79-rg87k"
Sep 11 14:40:18.889305 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:18.889266    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jpbg9\" (UniqueName: \"kubernetes.io/projected/e7eeaf9b-5b0d-4208-9839-8bbf719de427-kube-api-access-jpbg9\") pod \"e7eeaf9b-5b0d-4208-9839-8bbf719de427\" (UID: \"e7eeaf9b-5b0d-4208-9839-8bbf719de427\") "
Sep 11 14:40:18.889479 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:18.889326    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7eeaf9b-5b0d-4208-9839-8bbf719de427-dshm\") pod \"e7eeaf9b-5b0d-4208-9839-8bbf719de427\" (UID: \"e7eeaf9b-5b0d-4208-9839-8bbf719de427\") "
Sep 11 14:40:18.889479 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:18.889458    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7eeaf9b-5b0d-4208-9839-8bbf719de427-tls-certs\") pod \"e7eeaf9b-5b0d-4208-9839-8bbf719de427\" (UID: \"e7eeaf9b-5b0d-4208-9839-8bbf719de427\") "
Sep 11 14:40:18.889570 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:18.889482    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7eeaf9b-5b0d-4208-9839-8bbf719de427-home\") pod \"e7eeaf9b-5b0d-4208-9839-8bbf719de427\" (UID: \"e7eeaf9b-5b0d-4208-9839-8bbf719de427\") "
Sep 11 14:40:18.889570 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:18.889518    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7eeaf9b-5b0d-4208-9839-8bbf719de427-model-cache\") pod \"e7eeaf9b-5b0d-4208-9839-8bbf719de427\" (UID: \"e7eeaf9b-5b0d-4208-9839-8bbf719de427\") "
Sep 11 14:40:18.889646 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:18.889575    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7eeaf9b-5b0d-4208-9839-8bbf719de427-tmp-dir\") pod \"e7eeaf9b-5b0d-4208-9839-8bbf719de427\" (UID: \"e7eeaf9b-5b0d-4208-9839-8bbf719de427\") "
Sep 11 14:40:18.889801 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:18.889771    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7eeaf9b-5b0d-4208-9839-8bbf719de427-home" (OuterVolumeSpecName: "home") pod "e7eeaf9b-5b0d-4208-9839-8bbf719de427" (UID: "e7eeaf9b-5b0d-4208-9839-8bbf719de427"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:40:18.889947 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:18.889817    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7eeaf9b-5b0d-4208-9839-8bbf719de427-model-cache" (OuterVolumeSpecName: "model-cache") pod "e7eeaf9b-5b0d-4208-9839-8bbf719de427" (UID: "e7eeaf9b-5b0d-4208-9839-8bbf719de427"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:40:18.890050 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:18.889940    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7eeaf9b-5b0d-4208-9839-8bbf719de427-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e7eeaf9b-5b0d-4208-9839-8bbf719de427" (UID: "e7eeaf9b-5b0d-4208-9839-8bbf719de427"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:40:18.890113 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:18.890047    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7eeaf9b-5b0d-4208-9839-8bbf719de427-home\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:40:18.890113 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:18.890069    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7eeaf9b-5b0d-4208-9839-8bbf719de427-model-cache\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:40:18.891649 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:18.891619    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7eeaf9b-5b0d-4208-9839-8bbf719de427-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e7eeaf9b-5b0d-4208-9839-8bbf719de427" (UID: "e7eeaf9b-5b0d-4208-9839-8bbf719de427"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:40:18.891745 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:18.891661    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7eeaf9b-5b0d-4208-9839-8bbf719de427-dshm" (OuterVolumeSpecName: "dshm") pod "e7eeaf9b-5b0d-4208-9839-8bbf719de427" (UID: "e7eeaf9b-5b0d-4208-9839-8bbf719de427"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:40:18.891745 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:18.891677    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7eeaf9b-5b0d-4208-9839-8bbf719de427-kube-api-access-jpbg9" (OuterVolumeSpecName: "kube-api-access-jpbg9") pod "e7eeaf9b-5b0d-4208-9839-8bbf719de427" (UID: "e7eeaf9b-5b0d-4208-9839-8bbf719de427"). InnerVolumeSpecName "kube-api-access-jpbg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:40:18.991471 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:18.991435    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7eeaf9b-5b0d-4208-9839-8bbf719de427-tls-certs\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:40:18.991471 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:18.991464    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7eeaf9b-5b0d-4208-9839-8bbf719de427-tmp-dir\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:40:18.991471 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:18.991474    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpbg9\" (UniqueName: \"kubernetes.io/projected/e7eeaf9b-5b0d-4208-9839-8bbf719de427-kube-api-access-jpbg9\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:40:18.991704 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:18.991484    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7eeaf9b-5b0d-4208-9839-8bbf719de427-dshm\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:40:19.623290 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:19.623250    2581 generic.go:356] "Generic (PLEG): container finished" podID="e7eeaf9b-5b0d-4208-9839-8bbf719de427" containerID="91d6d4a269e43c164232ab1e82566dee61efc78d8fa209e1f40b2ed5b3b7cee5" exitCode=0
Sep 11 14:40:19.623740 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:19.623303    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-09b16c/auth-test-model-09b16c-kserve-7df9b7df79-rg87k"
Sep 11 14:40:19.623740 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:19.623328    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-09b16c/auth-test-model-09b16c-kserve-7df9b7df79-rg87k" event={"ID":"e7eeaf9b-5b0d-4208-9839-8bbf719de427","Type":"ContainerDied","Data":"91d6d4a269e43c164232ab1e82566dee61efc78d8fa209e1f40b2ed5b3b7cee5"}
Sep 11 14:40:19.623740 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:19.623364    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-09b16c/auth-test-model-09b16c-kserve-7df9b7df79-rg87k" event={"ID":"e7eeaf9b-5b0d-4208-9839-8bbf719de427","Type":"ContainerDied","Data":"4d3c8586fac2d4877681ba561adcc6dc44a52dfbcec761f307e5e5d50ea7370b"}
Sep 11 14:40:19.623740 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:19.623383    2581 scope.go:117] "RemoveContainer" containerID="91d6d4a269e43c164232ab1e82566dee61efc78d8fa209e1f40b2ed5b3b7cee5"
Sep 11 14:40:19.631775 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:19.631754    2581 scope.go:117] "RemoveContainer" containerID="91d6d4a269e43c164232ab1e82566dee61efc78d8fa209e1f40b2ed5b3b7cee5"
Sep 11 14:40:19.632076 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:40:19.632048    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91d6d4a269e43c164232ab1e82566dee61efc78d8fa209e1f40b2ed5b3b7cee5\": container with ID starting with 91d6d4a269e43c164232ab1e82566dee61efc78d8fa209e1f40b2ed5b3b7cee5 not found: ID does not exist" containerID="91d6d4a269e43c164232ab1e82566dee61efc78d8fa209e1f40b2ed5b3b7cee5"
Sep 11 14:40:19.632162 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:19.632085    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91d6d4a269e43c164232ab1e82566dee61efc78d8fa209e1f40b2ed5b3b7cee5"} err="failed to get container status \"91d6d4a269e43c164232ab1e82566dee61efc78d8fa209e1f40b2ed5b3b7cee5\": rpc error: code = NotFound desc = could not find container \"91d6d4a269e43c164232ab1e82566dee61efc78d8fa209e1f40b2ed5b3b7cee5\": container with ID starting with 91d6d4a269e43c164232ab1e82566dee61efc78d8fa209e1f40b2ed5b3b7cee5 not found: ID does not exist"
Sep 11 14:40:19.653005 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:19.652953    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-09b16c/auth-test-model-09b16c-kserve-7df9b7df79-rg87k"]
Sep 11 14:40:19.658144 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:19.658119    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-09b16c/auth-test-model-09b16c-kserve-7df9b7df79-rg87k"]
Sep 11 14:40:19.713396 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:19.713339    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-1c284e/auth-test-model-1c284e-kserve-dbc5df587-ckv8h" podUID="c3ec3968-777d-4d0a-adea-f33d1e43c0b0" containerName="main" containerID="cri-o://407410ae258e24f15d80e68f00729430cdf9bc9a9a31844a921aed46db4841e0" gracePeriod=45
Sep 11 14:40:19.772757 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:19.772687    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-1c284e/auth-test-model-1c284e-kserve-dbc5df587-ckv8h" podUID="c3ec3968-777d-4d0a-adea-f33d1e43c0b0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": dial tcp 10.132.0.57:8000: connect: connection refused"
Sep 11 14:40:19.964558 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:19.964533    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1c284e/auth-test-model-1c284e-kserve-dbc5df587-ckv8h"
Sep 11 14:40:20.102023 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:20.101900    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3ec3968-777d-4d0a-adea-f33d1e43c0b0-home\") pod \"c3ec3968-777d-4d0a-adea-f33d1e43c0b0\" (UID: \"c3ec3968-777d-4d0a-adea-f33d1e43c0b0\") "
Sep 11 14:40:20.102023 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:20.102016    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xb6gt\" (UniqueName: \"kubernetes.io/projected/c3ec3968-777d-4d0a-adea-f33d1e43c0b0-kube-api-access-xb6gt\") pod \"c3ec3968-777d-4d0a-adea-f33d1e43c0b0\" (UID: \"c3ec3968-777d-4d0a-adea-f33d1e43c0b0\") "
Sep 11 14:40:20.102260 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:20.102036    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3ec3968-777d-4d0a-adea-f33d1e43c0b0-tls-certs\") pod \"c3ec3968-777d-4d0a-adea-f33d1e43c0b0\" (UID: \"c3ec3968-777d-4d0a-adea-f33d1e43c0b0\") "
Sep 11 14:40:20.102260 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:20.102054    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3ec3968-777d-4d0a-adea-f33d1e43c0b0-dshm\") pod \"c3ec3968-777d-4d0a-adea-f33d1e43c0b0\" (UID: \"c3ec3968-777d-4d0a-adea-f33d1e43c0b0\") "
Sep 11 14:40:20.102260 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:20.102098    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3ec3968-777d-4d0a-adea-f33d1e43c0b0-model-cache\") pod \"c3ec3968-777d-4d0a-adea-f33d1e43c0b0\" (UID: \"c3ec3968-777d-4d0a-adea-f33d1e43c0b0\") "
Sep 11 14:40:20.102260 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:20.102147    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3ec3968-777d-4d0a-adea-f33d1e43c0b0-tmp-dir\") pod \"c3ec3968-777d-4d0a-adea-f33d1e43c0b0\" (UID: \"c3ec3968-777d-4d0a-adea-f33d1e43c0b0\") "
Sep 11 14:40:20.102467 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:20.102265    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3ec3968-777d-4d0a-adea-f33d1e43c0b0-home" (OuterVolumeSpecName: "home") pod "c3ec3968-777d-4d0a-adea-f33d1e43c0b0" (UID: "c3ec3968-777d-4d0a-adea-f33d1e43c0b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:40:20.102531 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:20.102464    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3ec3968-777d-4d0a-adea-f33d1e43c0b0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c3ec3968-777d-4d0a-adea-f33d1e43c0b0" (UID: "c3ec3968-777d-4d0a-adea-f33d1e43c0b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:40:20.102531 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:20.102479    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3ec3968-777d-4d0a-adea-f33d1e43c0b0-model-cache" (OuterVolumeSpecName: "model-cache") pod "c3ec3968-777d-4d0a-adea-f33d1e43c0b0" (UID: "c3ec3968-777d-4d0a-adea-f33d1e43c0b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:40:20.102628 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:20.102576    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3ec3968-777d-4d0a-adea-f33d1e43c0b0-tmp-dir\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:40:20.102628 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:20.102596    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3ec3968-777d-4d0a-adea-f33d1e43c0b0-home\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:40:20.104395 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:20.104370    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3ec3968-777d-4d0a-adea-f33d1e43c0b0-kube-api-access-xb6gt" (OuterVolumeSpecName: "kube-api-access-xb6gt") pod "c3ec3968-777d-4d0a-adea-f33d1e43c0b0" (UID: "c3ec3968-777d-4d0a-adea-f33d1e43c0b0"). InnerVolumeSpecName "kube-api-access-xb6gt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:40:20.104677 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:20.104651    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3ec3968-777d-4d0a-adea-f33d1e43c0b0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c3ec3968-777d-4d0a-adea-f33d1e43c0b0" (UID: "c3ec3968-777d-4d0a-adea-f33d1e43c0b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:40:20.104677 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:20.104670    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3ec3968-777d-4d0a-adea-f33d1e43c0b0-dshm" (OuterVolumeSpecName: "dshm") pod "c3ec3968-777d-4d0a-adea-f33d1e43c0b0" (UID: "c3ec3968-777d-4d0a-adea-f33d1e43c0b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:40:20.203241 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:20.203201    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xb6gt\" (UniqueName: \"kubernetes.io/projected/c3ec3968-777d-4d0a-adea-f33d1e43c0b0-kube-api-access-xb6gt\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:40:20.203241 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:20.203233    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3ec3968-777d-4d0a-adea-f33d1e43c0b0-tls-certs\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:40:20.203241 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:20.203243    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3ec3968-777d-4d0a-adea-f33d1e43c0b0-dshm\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:40:20.203241 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:20.203251    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3ec3968-777d-4d0a-adea-f33d1e43c0b0-model-cache\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:40:20.630633 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:20.630590    2581 generic.go:356] "Generic (PLEG): container finished" podID="c3ec3968-777d-4d0a-adea-f33d1e43c0b0" containerID="407410ae258e24f15d80e68f00729430cdf9bc9a9a31844a921aed46db4841e0" exitCode=0
Sep 11 14:40:20.630633 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:20.630635    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1c284e/auth-test-model-1c284e-kserve-dbc5df587-ckv8h"
Sep 11 14:40:20.631217 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:20.630680    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1c284e/auth-test-model-1c284e-kserve-dbc5df587-ckv8h" event={"ID":"c3ec3968-777d-4d0a-adea-f33d1e43c0b0","Type":"ContainerDied","Data":"407410ae258e24f15d80e68f00729430cdf9bc9a9a31844a921aed46db4841e0"}
Sep 11 14:40:20.631217 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:20.630738    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1c284e/auth-test-model-1c284e-kserve-dbc5df587-ckv8h" event={"ID":"c3ec3968-777d-4d0a-adea-f33d1e43c0b0","Type":"ContainerDied","Data":"81ef3516dc389b529cf11886a28de871df88e8b79399879c387b19f4f9df84c7"}
Sep 11 14:40:20.631217 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:20.630764    2581 scope.go:117] "RemoveContainer" containerID="407410ae258e24f15d80e68f00729430cdf9bc9a9a31844a921aed46db4841e0"
Sep 11 14:40:20.640290 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:20.640273    2581 scope.go:117] "RemoveContainer" containerID="407410ae258e24f15d80e68f00729430cdf9bc9a9a31844a921aed46db4841e0"
Sep 11 14:40:20.640549 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:40:20.640530    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"407410ae258e24f15d80e68f00729430cdf9bc9a9a31844a921aed46db4841e0\": container with ID starting with 407410ae258e24f15d80e68f00729430cdf9bc9a9a31844a921aed46db4841e0 not found: ID does not exist" containerID="407410ae258e24f15d80e68f00729430cdf9bc9a9a31844a921aed46db4841e0"
Sep 11 14:40:20.640615 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:20.640557    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"407410ae258e24f15d80e68f00729430cdf9bc9a9a31844a921aed46db4841e0"} err="failed to get container status \"407410ae258e24f15d80e68f00729430cdf9bc9a9a31844a921aed46db4841e0\": rpc error: code = NotFound desc = could not find container \"407410ae258e24f15d80e68f00729430cdf9bc9a9a31844a921aed46db4841e0\": container with ID starting with 407410ae258e24f15d80e68f00729430cdf9bc9a9a31844a921aed46db4841e0 not found: ID does not exist"
Sep 11 14:40:20.671096 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:20.671063    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1c284e/auth-test-model-1c284e-kserve-dbc5df587-ckv8h"]
Sep 11 14:40:20.699202 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:20.699166    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1c284e/auth-test-model-1c284e-kserve-dbc5df587-ckv8h"]
Sep 11 14:40:21.499678 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:21.499647    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3ec3968-777d-4d0a-adea-f33d1e43c0b0" path="/var/lib/kubelet/pods/c3ec3968-777d-4d0a-adea-f33d1e43c0b0/volumes"
Sep 11 14:40:21.500086 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:21.500073    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7eeaf9b-5b0d-4208-9839-8bbf719de427" path="/var/lib/kubelet/pods/e7eeaf9b-5b0d-4208-9839-8bbf719de427/volumes"
Sep 11 14:40:27.626606 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:27.626569    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-09b16c/sub-test-model-09b16c-kserve-7454455f57-f6gtb"]
Sep 11 14:40:27.627122 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:27.627107    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7eeaf9b-5b0d-4208-9839-8bbf719de427" containerName="main"
Sep 11 14:40:27.627199 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:27.627127    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7eeaf9b-5b0d-4208-9839-8bbf719de427" containerName="main"
Sep 11 14:40:27.627199 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:27.627153    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da8e262b-3b19-49fb-80b0-1eda4b67fd8e" containerName="payload-pre-processing"
Sep 11 14:40:27.627199 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:27.627163    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="da8e262b-3b19-49fb-80b0-1eda4b67fd8e" containerName="payload-pre-processing"
Sep 11 14:40:27.627199 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:27.627182    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3ec3968-777d-4d0a-adea-f33d1e43c0b0" containerName="main"
Sep 11 14:40:27.627199 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:27.627191    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3ec3968-777d-4d0a-adea-f33d1e43c0b0" containerName="main"
Sep 11 14:40:27.627445 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:27.627214    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f404627b-5b49-41c5-955b-74dda6975a35" containerName="revoke-keys"
Sep 11 14:40:27.627445 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:27.627223    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f404627b-5b49-41c5-955b-74dda6975a35" containerName="revoke-keys"
Sep 11 14:40:27.627445 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:27.627319    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3ec3968-777d-4d0a-adea-f33d1e43c0b0" containerName="main"
Sep 11 14:40:27.627445 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:27.627332    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7eeaf9b-5b0d-4208-9839-8bbf719de427" containerName="main"
Sep 11 14:40:27.627445 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:27.627342    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f404627b-5b49-41c5-955b-74dda6975a35" containerName="revoke-keys"
Sep 11 14:40:27.627445 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:27.627353    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="da8e262b-3b19-49fb-80b0-1eda4b67fd8e" containerName="payload-pre-processing"
Sep 11 14:40:27.632113 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:27.632093    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-09b16c/sub-test-model-09b16c-kserve-7454455f57-f6gtb"
Sep 11 14:40:27.634936 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:27.634914    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-09b16c\"/\"openshift-service-ca.crt\""
Sep 11 14:40:27.636333 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:27.636308    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-09b16c\"/\"kube-root-ca.crt\""
Sep 11 14:40:27.636463 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:27.636315    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-09b16c\"/\"default-dockercfg-g4l97\""
Sep 11 14:40:27.636536 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:27.636457    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-09b16c\"/\"sub-test-model-09b16c-kserve-self-signed-certs\""
Sep 11 14:40:27.644439 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:27.644413    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-09b16c/sub-test-model-09b16c-kserve-7454455f57-f6gtb"]
Sep 11 14:40:27.674561 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:27.674528    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff-dshm\") pod \"sub-test-model-09b16c-kserve-7454455f57-f6gtb\" (UID: \"c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff\") " pod="ai-tenant-e2e-shared-a-w3-09b16c/sub-test-model-09b16c-kserve-7454455f57-f6gtb"
Sep 11 14:40:27.674561 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:27.674562    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff-tmp-dir\") pod \"sub-test-model-09b16c-kserve-7454455f57-f6gtb\" (UID: \"c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff\") " pod="ai-tenant-e2e-shared-a-w3-09b16c/sub-test-model-09b16c-kserve-7454455f57-f6gtb"
Sep 11 14:40:27.674800 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:27.674593    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scmxx\" (UniqueName: \"kubernetes.io/projected/c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff-kube-api-access-scmxx\") pod \"sub-test-model-09b16c-kserve-7454455f57-f6gtb\" (UID: \"c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff\") " pod="ai-tenant-e2e-shared-a-w3-09b16c/sub-test-model-09b16c-kserve-7454455f57-f6gtb"
Sep 11 14:40:27.674800 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:27.674614    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff-home\") pod \"sub-test-model-09b16c-kserve-7454455f57-f6gtb\" (UID: \"c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff\") " pod="ai-tenant-e2e-shared-a-w3-09b16c/sub-test-model-09b16c-kserve-7454455f57-f6gtb"
Sep 11 14:40:27.674800 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:27.674704    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff-model-cache\") pod \"sub-test-model-09b16c-kserve-7454455f57-f6gtb\" (UID: \"c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff\") " pod="ai-tenant-e2e-shared-a-w3-09b16c/sub-test-model-09b16c-kserve-7454455f57-f6gtb"
Sep 11 14:40:27.674800 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:27.674761    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff-tls-certs\") pod \"sub-test-model-09b16c-kserve-7454455f57-f6gtb\" (UID: \"c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff\") " pod="ai-tenant-e2e-shared-a-w3-09b16c/sub-test-model-09b16c-kserve-7454455f57-f6gtb"
Sep 11 14:40:27.775578 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:27.775546    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scmxx\" (UniqueName: \"kubernetes.io/projected/c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff-kube-api-access-scmxx\") pod \"sub-test-model-09b16c-kserve-7454455f57-f6gtb\" (UID: \"c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff\") " pod="ai-tenant-e2e-shared-a-w3-09b16c/sub-test-model-09b16c-kserve-7454455f57-f6gtb"
Sep 11 14:40:27.775774 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:27.775590    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff-home\") pod \"sub-test-model-09b16c-kserve-7454455f57-f6gtb\" (UID: \"c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff\") " pod="ai-tenant-e2e-shared-a-w3-09b16c/sub-test-model-09b16c-kserve-7454455f57-f6gtb"
Sep 11 14:40:27.775774 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:27.775640    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff-model-cache\") pod \"sub-test-model-09b16c-kserve-7454455f57-f6gtb\" (UID: \"c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff\") " pod="ai-tenant-e2e-shared-a-w3-09b16c/sub-test-model-09b16c-kserve-7454455f57-f6gtb"
Sep 11 14:40:27.775774 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:27.775690    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff-tls-certs\") pod \"sub-test-model-09b16c-kserve-7454455f57-f6gtb\" (UID: \"c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff\") " pod="ai-tenant-e2e-shared-a-w3-09b16c/sub-test-model-09b16c-kserve-7454455f57-f6gtb"
Sep 11 14:40:27.775774 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:27.775758    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff-dshm\") pod \"sub-test-model-09b16c-kserve-7454455f57-f6gtb\" (UID: \"c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff\") " pod="ai-tenant-e2e-shared-a-w3-09b16c/sub-test-model-09b16c-kserve-7454455f57-f6gtb"
Sep 11 14:40:27.776004 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:27.775786    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff-tmp-dir\") pod \"sub-test-model-09b16c-kserve-7454455f57-f6gtb\" (UID: \"c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff\") " pod="ai-tenant-e2e-shared-a-w3-09b16c/sub-test-model-09b16c-kserve-7454455f57-f6gtb"
Sep 11 14:40:27.776071 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:27.776049    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff-home\") pod \"sub-test-model-09b16c-kserve-7454455f57-f6gtb\" (UID: \"c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff\") " pod="ai-tenant-e2e-shared-a-w3-09b16c/sub-test-model-09b16c-kserve-7454455f57-f6gtb"
Sep 11 14:40:27.776131 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:27.776088    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff-model-cache\") pod \"sub-test-model-09b16c-kserve-7454455f57-f6gtb\" (UID: \"c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff\") " pod="ai-tenant-e2e-shared-a-w3-09b16c/sub-test-model-09b16c-kserve-7454455f57-f6gtb"
Sep 11 14:40:27.776214 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:27.776194    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff-tmp-dir\") pod \"sub-test-model-09b16c-kserve-7454455f57-f6gtb\" (UID: \"c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff\") " pod="ai-tenant-e2e-shared-a-w3-09b16c/sub-test-model-09b16c-kserve-7454455f57-f6gtb"
Sep 11 14:40:27.777962 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:27.777935    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff-dshm\") pod \"sub-test-model-09b16c-kserve-7454455f57-f6gtb\" (UID: \"c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff\") " pod="ai-tenant-e2e-shared-a-w3-09b16c/sub-test-model-09b16c-kserve-7454455f57-f6gtb"
Sep 11 14:40:27.778239 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:27.778222    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff-tls-certs\") pod \"sub-test-model-09b16c-kserve-7454455f57-f6gtb\" (UID: \"c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff\") " pod="ai-tenant-e2e-shared-a-w3-09b16c/sub-test-model-09b16c-kserve-7454455f57-f6gtb"
Sep 11 14:40:27.787987 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:27.787945    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scmxx\" (UniqueName: \"kubernetes.io/projected/c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff-kube-api-access-scmxx\") pod \"sub-test-model-09b16c-kserve-7454455f57-f6gtb\" (UID: \"c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff\") " pod="ai-tenant-e2e-shared-a-w3-09b16c/sub-test-model-09b16c-kserve-7454455f57-f6gtb"
Sep 11 14:40:27.942474 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:27.942350    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-09b16c/sub-test-model-09b16c-kserve-7454455f57-f6gtb"
Sep 11 14:40:28.223692 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:28.223655    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-09b16c/sub-test-model-09b16c-kserve-7454455f57-f6gtb"]
Sep 11 14:40:28.224827 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:40:28.224802    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc2e0d2ec_33d2_49de_8826_9b71f7a2a3ff.slice/crio-d5e9995f9aeea9f69e51c9aa2c54e17830093cb48190143304701f35698cb695 WatchSource:0}: Error finding container d5e9995f9aeea9f69e51c9aa2c54e17830093cb48190143304701f35698cb695: Status 404 returned error can't find the container with id d5e9995f9aeea9f69e51c9aa2c54e17830093cb48190143304701f35698cb695
Sep 11 14:40:28.676946 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:28.676909    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-09b16c/sub-test-model-09b16c-kserve-7454455f57-f6gtb" event={"ID":"c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff","Type":"ContainerStarted","Data":"b2b9735726c06908af36bffe9130461c632ae3836526c91f0c68ffd652a305d2"}
Sep 11 14:40:28.676946 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:28.676950    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-09b16c/sub-test-model-09b16c-kserve-7454455f57-f6gtb" event={"ID":"c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff","Type":"ContainerStarted","Data":"d5e9995f9aeea9f69e51c9aa2c54e17830093cb48190143304701f35698cb695"}
Sep 11 14:40:28.712226 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:28.712164    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-09b16c/sub-test-model-09b16c-kserve-7454455f57-f6gtb" podStartSLOduration=1.712143575 podStartE2EDuration="1.712143575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:40:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:40:28.710171281 +0000 UTC m=+1531.860721344" watchObservedRunningTime="2026-09-11 14:40:28.712143575 +0000 UTC m=+1531.862693639"
Sep 11 14:40:33.473737 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:33.473689    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-2b947f70/e2e-lifecycle-model-2b947f70-kserve-7f89bffb87-mr7nc" podUID="658f5de9-c6e7-41f2-b46e-316e9812a56f" containerName="main" containerID="cri-o://7b1e52e3791189a2a3287800635550373b0700a48f8a65360bad841f985d6d26" gracePeriod=45
Sep 11 14:40:33.704416 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:33.704385    2581 generic.go:356] "Generic (PLEG): container finished" podID="658f5de9-c6e7-41f2-b46e-316e9812a56f" containerID="7b1e52e3791189a2a3287800635550373b0700a48f8a65360bad841f985d6d26" exitCode=0
Sep 11 14:40:33.704572 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:33.704459    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2b947f70/e2e-lifecycle-model-2b947f70-kserve-7f89bffb87-mr7nc" event={"ID":"658f5de9-c6e7-41f2-b46e-316e9812a56f","Type":"ContainerDied","Data":"7b1e52e3791189a2a3287800635550373b0700a48f8a65360bad841f985d6d26"}
Sep 11 14:40:33.728547 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:33.728482    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2b947f70/e2e-lifecycle-model-2b947f70-kserve-7f89bffb87-mr7nc"
Sep 11 14:40:33.832780 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:33.832748    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/658f5de9-c6e7-41f2-b46e-316e9812a56f-dshm\") pod \"658f5de9-c6e7-41f2-b46e-316e9812a56f\" (UID: \"658f5de9-c6e7-41f2-b46e-316e9812a56f\") "
Sep 11 14:40:33.832944 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:33.832852    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/658f5de9-c6e7-41f2-b46e-316e9812a56f-tmp-dir\") pod \"658f5de9-c6e7-41f2-b46e-316e9812a56f\" (UID: \"658f5de9-c6e7-41f2-b46e-316e9812a56f\") "
Sep 11 14:40:33.832944 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:33.832897    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/658f5de9-c6e7-41f2-b46e-316e9812a56f-model-cache\") pod \"658f5de9-c6e7-41f2-b46e-316e9812a56f\" (UID: \"658f5de9-c6e7-41f2-b46e-316e9812a56f\") "
Sep 11 14:40:33.832944 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:33.832923    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/658f5de9-c6e7-41f2-b46e-316e9812a56f-tls-certs\") pod \"658f5de9-c6e7-41f2-b46e-316e9812a56f\" (UID: \"658f5de9-c6e7-41f2-b46e-316e9812a56f\") "
Sep 11 14:40:33.833236 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:33.832960    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/658f5de9-c6e7-41f2-b46e-316e9812a56f-home\") pod \"658f5de9-c6e7-41f2-b46e-316e9812a56f\" (UID: \"658f5de9-c6e7-41f2-b46e-316e9812a56f\") "
Sep 11 14:40:33.833236 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:33.833027    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dlgft\" (UniqueName: \"kubernetes.io/projected/658f5de9-c6e7-41f2-b46e-316e9812a56f-kube-api-access-dlgft\") pod \"658f5de9-c6e7-41f2-b46e-316e9812a56f\" (UID: \"658f5de9-c6e7-41f2-b46e-316e9812a56f\") "
Sep 11 14:40:33.833236 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:33.833188    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/658f5de9-c6e7-41f2-b46e-316e9812a56f-model-cache" (OuterVolumeSpecName: "model-cache") pod "658f5de9-c6e7-41f2-b46e-316e9812a56f" (UID: "658f5de9-c6e7-41f2-b46e-316e9812a56f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:40:33.833236 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:33.833205    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/658f5de9-c6e7-41f2-b46e-316e9812a56f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "658f5de9-c6e7-41f2-b46e-316e9812a56f" (UID: "658f5de9-c6e7-41f2-b46e-316e9812a56f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:40:33.833236 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:33.833232    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/658f5de9-c6e7-41f2-b46e-316e9812a56f-home" (OuterVolumeSpecName: "home") pod "658f5de9-c6e7-41f2-b46e-316e9812a56f" (UID: "658f5de9-c6e7-41f2-b46e-316e9812a56f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:40:33.833416 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:33.833376    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/658f5de9-c6e7-41f2-b46e-316e9812a56f-tmp-dir\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:40:33.833416 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:33.833392    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/658f5de9-c6e7-41f2-b46e-316e9812a56f-model-cache\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:40:33.833416 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:33.833410    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/658f5de9-c6e7-41f2-b46e-316e9812a56f-home\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:40:33.835101 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:33.835071    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/658f5de9-c6e7-41f2-b46e-316e9812a56f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "658f5de9-c6e7-41f2-b46e-316e9812a56f" (UID: "658f5de9-c6e7-41f2-b46e-316e9812a56f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:40:33.835205 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:33.835071    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/658f5de9-c6e7-41f2-b46e-316e9812a56f-dshm" (OuterVolumeSpecName: "dshm") pod "658f5de9-c6e7-41f2-b46e-316e9812a56f" (UID: "658f5de9-c6e7-41f2-b46e-316e9812a56f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:40:33.835205 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:33.835099    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/658f5de9-c6e7-41f2-b46e-316e9812a56f-kube-api-access-dlgft" (OuterVolumeSpecName: "kube-api-access-dlgft") pod "658f5de9-c6e7-41f2-b46e-316e9812a56f" (UID: "658f5de9-c6e7-41f2-b46e-316e9812a56f"). InnerVolumeSpecName "kube-api-access-dlgft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:40:33.933905 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:33.933867    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/658f5de9-c6e7-41f2-b46e-316e9812a56f-tls-certs\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:40:33.933905 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:33.933896    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlgft\" (UniqueName: \"kubernetes.io/projected/658f5de9-c6e7-41f2-b46e-316e9812a56f-kube-api-access-dlgft\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:40:33.933905 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:33.933906    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/658f5de9-c6e7-41f2-b46e-316e9812a56f-dshm\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:40:34.717490 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:34.717451    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2b947f70/e2e-lifecycle-model-2b947f70-kserve-7f89bffb87-mr7nc" event={"ID":"658f5de9-c6e7-41f2-b46e-316e9812a56f","Type":"ContainerDied","Data":"42636f499fde3796b73674ba45a6fe06f9e83dba79afa8f10299dea77bde2965"}
Sep 11 14:40:34.717916 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:34.717501    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2b947f70/e2e-lifecycle-model-2b947f70-kserve-7f89bffb87-mr7nc"
Sep 11 14:40:34.717916 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:34.717512    2581 scope.go:117] "RemoveContainer" containerID="7b1e52e3791189a2a3287800635550373b0700a48f8a65360bad841f985d6d26"
Sep 11 14:40:34.752538 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:34.752505    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-2b947f70/e2e-lifecycle-model-2b947f70-kserve-7f89bffb87-mr7nc"]
Sep 11 14:40:34.758250 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:34.758224    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-2b947f70/e2e-lifecycle-model-2b947f70-kserve-7f89bffb87-mr7nc"]
Sep 11 14:40:35.499754 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:35.499712    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="658f5de9-c6e7-41f2-b46e-316e9812a56f" path="/var/lib/kubelet/pods/658f5de9-c6e7-41f2-b46e-316e9812a56f/volumes"
Sep 11 14:40:37.942576 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:37.942530    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-09b16c/sub-test-model-09b16c-kserve-7454455f57-f6gtb"
Sep 11 14:40:37.943121 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:37.942599    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-09b16c/sub-test-model-09b16c-kserve-7454455f57-f6gtb"
Sep 11 14:40:37.955318 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:37.955287    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-09b16c/sub-test-model-09b16c-kserve-7454455f57-f6gtb"
Sep 11 14:40:38.750028 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:38.749999    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-09b16c/sub-test-model-09b16c-kserve-7454455f57-f6gtb"
Sep 11 14:40:44.726246 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:44.726213    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-1c284e/sub-test-model-1c284e-kserve-66b545b97-57qnl"]
Sep 11 14:40:44.726618 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:44.726573    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="658f5de9-c6e7-41f2-b46e-316e9812a56f" containerName="main"
Sep 11 14:40:44.726618 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:44.726583    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="658f5de9-c6e7-41f2-b46e-316e9812a56f" containerName="main"
Sep 11 14:40:44.726694 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:44.726640    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="658f5de9-c6e7-41f2-b46e-316e9812a56f" containerName="main"
Sep 11 14:40:44.731110 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:44.731093    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1c284e/sub-test-model-1c284e-kserve-66b545b97-57qnl"
Sep 11 14:40:44.735676 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:44.735650    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-1c284e\"/\"default-dockercfg-62xn9\""
Sep 11 14:40:44.735779 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:44.735705    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-1c284e\"/\"openshift-service-ca.crt\""
Sep 11 14:40:44.735890 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:44.735877    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-1c284e\"/\"kube-root-ca.crt\""
Sep 11 14:40:44.735931 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:44.735908    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-1c284e\"/\"sub-test-model-1c284e-kserve-self-signed-certs\""
Sep 11 14:40:44.762320 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:44.762290    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1c284e/sub-test-model-1c284e-kserve-66b545b97-57qnl"]
Sep 11 14:40:44.846950 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:44.846911    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8-tls-certs\") pod \"sub-test-model-1c284e-kserve-66b545b97-57qnl\" (UID: \"d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8\") " pod="ai-tenant-e2e-shared-b-w3-1c284e/sub-test-model-1c284e-kserve-66b545b97-57qnl"
Sep 11 14:40:44.847153 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:44.846984    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8-home\") pod \"sub-test-model-1c284e-kserve-66b545b97-57qnl\" (UID: \"d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8\") " pod="ai-tenant-e2e-shared-b-w3-1c284e/sub-test-model-1c284e-kserve-66b545b97-57qnl"
Sep 11 14:40:44.847153 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:44.847067    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8-tmp-dir\") pod \"sub-test-model-1c284e-kserve-66b545b97-57qnl\" (UID: \"d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8\") " pod="ai-tenant-e2e-shared-b-w3-1c284e/sub-test-model-1c284e-kserve-66b545b97-57qnl"
Sep 11 14:40:44.847153 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:44.847097    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds627\" (UniqueName: \"kubernetes.io/projected/d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8-kube-api-access-ds627\") pod \"sub-test-model-1c284e-kserve-66b545b97-57qnl\" (UID: \"d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8\") " pod="ai-tenant-e2e-shared-b-w3-1c284e/sub-test-model-1c284e-kserve-66b545b97-57qnl"
Sep 11 14:40:44.847153 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:44.847114    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8-model-cache\") pod \"sub-test-model-1c284e-kserve-66b545b97-57qnl\" (UID: \"d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8\") " pod="ai-tenant-e2e-shared-b-w3-1c284e/sub-test-model-1c284e-kserve-66b545b97-57qnl"
Sep 11 14:40:44.847296 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:44.847168    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8-dshm\") pod \"sub-test-model-1c284e-kserve-66b545b97-57qnl\" (UID: \"d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8\") " pod="ai-tenant-e2e-shared-b-w3-1c284e/sub-test-model-1c284e-kserve-66b545b97-57qnl"
Sep 11 14:40:44.948345 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:44.948306    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8-tls-certs\") pod \"sub-test-model-1c284e-kserve-66b545b97-57qnl\" (UID: \"d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8\") " pod="ai-tenant-e2e-shared-b-w3-1c284e/sub-test-model-1c284e-kserve-66b545b97-57qnl"
Sep 11 14:40:44.948564 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:44.948360    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8-home\") pod \"sub-test-model-1c284e-kserve-66b545b97-57qnl\" (UID: \"d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8\") " pod="ai-tenant-e2e-shared-b-w3-1c284e/sub-test-model-1c284e-kserve-66b545b97-57qnl"
Sep 11 14:40:44.948564 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:44.948411    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8-tmp-dir\") pod \"sub-test-model-1c284e-kserve-66b545b97-57qnl\" (UID: \"d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8\") " pod="ai-tenant-e2e-shared-b-w3-1c284e/sub-test-model-1c284e-kserve-66b545b97-57qnl"
Sep 11 14:40:44.948564 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:44.948429    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds627\" (UniqueName: \"kubernetes.io/projected/d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8-kube-api-access-ds627\") pod \"sub-test-model-1c284e-kserve-66b545b97-57qnl\" (UID: \"d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8\") " pod="ai-tenant-e2e-shared-b-w3-1c284e/sub-test-model-1c284e-kserve-66b545b97-57qnl"
Sep 11 14:40:44.948564 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:44.948446    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8-model-cache\") pod \"sub-test-model-1c284e-kserve-66b545b97-57qnl\" (UID: \"d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8\") " pod="ai-tenant-e2e-shared-b-w3-1c284e/sub-test-model-1c284e-kserve-66b545b97-57qnl"
Sep 11 14:40:44.948564 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:44.948472    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8-dshm\") pod \"sub-test-model-1c284e-kserve-66b545b97-57qnl\" (UID: \"d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8\") " pod="ai-tenant-e2e-shared-b-w3-1c284e/sub-test-model-1c284e-kserve-66b545b97-57qnl"
Sep 11 14:40:44.948828 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:44.948789    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8-tmp-dir\") pod \"sub-test-model-1c284e-kserve-66b545b97-57qnl\" (UID: \"d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8\") " pod="ai-tenant-e2e-shared-b-w3-1c284e/sub-test-model-1c284e-kserve-66b545b97-57qnl"
Sep 11 14:40:44.948900 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:44.948877    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8-home\") pod \"sub-test-model-1c284e-kserve-66b545b97-57qnl\" (UID: \"d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8\") " pod="ai-tenant-e2e-shared-b-w3-1c284e/sub-test-model-1c284e-kserve-66b545b97-57qnl"
Sep 11 14:40:44.948952 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:44.948928    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8-model-cache\") pod \"sub-test-model-1c284e-kserve-66b545b97-57qnl\" (UID: \"d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8\") " pod="ai-tenant-e2e-shared-b-w3-1c284e/sub-test-model-1c284e-kserve-66b545b97-57qnl"
Sep 11 14:40:44.950868 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:44.950842    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8-dshm\") pod \"sub-test-model-1c284e-kserve-66b545b97-57qnl\" (UID: \"d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8\") " pod="ai-tenant-e2e-shared-b-w3-1c284e/sub-test-model-1c284e-kserve-66b545b97-57qnl"
Sep 11 14:40:44.951077 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:44.951056    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8-tls-certs\") pod \"sub-test-model-1c284e-kserve-66b545b97-57qnl\" (UID: \"d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8\") " pod="ai-tenant-e2e-shared-b-w3-1c284e/sub-test-model-1c284e-kserve-66b545b97-57qnl"
Sep 11 14:40:44.960533 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:44.960512    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds627\" (UniqueName: \"kubernetes.io/projected/d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8-kube-api-access-ds627\") pod \"sub-test-model-1c284e-kserve-66b545b97-57qnl\" (UID: \"d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8\") " pod="ai-tenant-e2e-shared-b-w3-1c284e/sub-test-model-1c284e-kserve-66b545b97-57qnl"
Sep 11 14:40:45.040135 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:45.040099    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1c284e/sub-test-model-1c284e-kserve-66b545b97-57qnl"
Sep 11 14:40:45.330785 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:45.330757    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1c284e/sub-test-model-1c284e-kserve-66b545b97-57qnl"]
Sep 11 14:40:45.332273 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:40:45.332247    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6b90074_5cc6_4c18_ba9a_8ac1698ab3a8.slice/crio-84e0bb4eaa5e03492cef86d4f73584b45ba35566a26f528d22eb99b68285fb03 WatchSource:0}: Error finding container 84e0bb4eaa5e03492cef86d4f73584b45ba35566a26f528d22eb99b68285fb03: Status 404 returned error can't find the container with id 84e0bb4eaa5e03492cef86d4f73584b45ba35566a26f528d22eb99b68285fb03
Sep 11 14:40:45.778498 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:45.778462    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1c284e/sub-test-model-1c284e-kserve-66b545b97-57qnl" event={"ID":"d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8","Type":"ContainerStarted","Data":"b1bb4f0ed537a8492d1b479e91e22cb492c47c87c1aee76e1bc7898a9aecb3bf"}
Sep 11 14:40:45.778498 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:45.778504    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1c284e/sub-test-model-1c284e-kserve-66b545b97-57qnl" event={"ID":"d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8","Type":"ContainerStarted","Data":"84e0bb4eaa5e03492cef86d4f73584b45ba35566a26f528d22eb99b68285fb03"}
Sep 11 14:40:45.808781 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:45.808730    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-1c284e/sub-test-model-1c284e-kserve-66b545b97-57qnl" podStartSLOduration=1.808713278 podStartE2EDuration="1.808713278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:40:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:40:45.80691304 +0000 UTC m=+1548.957463108" watchObservedRunningTime="2026-09-11 14:40:45.808713278 +0000 UTC m=+1548.959263341"
Sep 11 14:40:55.040426 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:55.040387    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-1c284e/sub-test-model-1c284e-kserve-66b545b97-57qnl"
Sep 11 14:40:55.040906 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:55.040445    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-1c284e/sub-test-model-1c284e-kserve-66b545b97-57qnl"
Sep 11 14:40:55.053072 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:55.053045    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-1c284e/sub-test-model-1c284e-kserve-66b545b97-57qnl"
Sep 11 14:40:55.838653 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:40:55.838621    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-1c284e/sub-test-model-1c284e-kserve-66b545b97-57qnl"
Sep 11 14:41:24.628535 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:24.628499    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-09b16c/sub-test-model-09b16c-kserve-7454455f57-f6gtb"]
Sep 11 14:41:25.596120 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:25.596089    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1c284e/sub-test-model-1c284e-kserve-66b545b97-57qnl"]
Sep 11 14:41:39.647245 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:39.647197    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-09b16c/sub-test-model-09b16c-kserve-7454455f57-f6gtb" podUID="c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff" containerName="main" containerID="cri-o://b2b9735726c06908af36bffe9130461c632ae3836526c91f0c68ffd652a305d2" gracePeriod=45
Sep 11 14:41:39.891317 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:39.891288    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-09b16c/sub-test-model-09b16c-kserve-7454455f57-f6gtb"
Sep 11 14:41:39.994800 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:39.994772    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-scmxx\" (UniqueName: \"kubernetes.io/projected/c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff-kube-api-access-scmxx\") pod \"c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff\" (UID: \"c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff\") "
Sep 11 14:41:39.994998 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:39.994826    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff-dshm\") pod \"c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff\" (UID: \"c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff\") "
Sep 11 14:41:39.994998 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:39.994856    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff-tls-certs\") pod \"c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff\" (UID: \"c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff\") "
Sep 11 14:41:39.994998 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:39.994896    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff-home\") pod \"c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff\" (UID: \"c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff\") "
Sep 11 14:41:39.994998 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:39.994913    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff-tmp-dir\") pod \"c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff\" (UID: \"c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff\") "
Sep 11 14:41:39.994998 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:39.994946    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff-model-cache\") pod \"c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff\" (UID: \"c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff\") "
Sep 11 14:41:39.995296 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:39.995212    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff" (UID: "c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:41:39.995296 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:39.995224    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff-home" (OuterVolumeSpecName: "home") pod "c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff" (UID: "c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:41:39.995296 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:39.995237    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff-model-cache" (OuterVolumeSpecName: "model-cache") pod "c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff" (UID: "c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:41:39.997193 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:39.997160    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff" (UID: "c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:41:39.997193 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:39.997179    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff-kube-api-access-scmxx" (OuterVolumeSpecName: "kube-api-access-scmxx") pod "c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff" (UID: "c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff"). InnerVolumeSpecName "kube-api-access-scmxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:41:39.997336 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:39.997242    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff-dshm" (OuterVolumeSpecName: "dshm") pod "c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff" (UID: "c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:41:40.066126 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:40.066090    2581 generic.go:356] "Generic (PLEG): container finished" podID="c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff" containerID="b2b9735726c06908af36bffe9130461c632ae3836526c91f0c68ffd652a305d2" exitCode=0
Sep 11 14:41:40.066329 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:40.066135    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-09b16c/sub-test-model-09b16c-kserve-7454455f57-f6gtb"
Sep 11 14:41:40.066329 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:40.066180    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-09b16c/sub-test-model-09b16c-kserve-7454455f57-f6gtb" event={"ID":"c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff","Type":"ContainerDied","Data":"b2b9735726c06908af36bffe9130461c632ae3836526c91f0c68ffd652a305d2"}
Sep 11 14:41:40.066329 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:40.066227    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-09b16c/sub-test-model-09b16c-kserve-7454455f57-f6gtb" event={"ID":"c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff","Type":"ContainerDied","Data":"d5e9995f9aeea9f69e51c9aa2c54e17830093cb48190143304701f35698cb695"}
Sep 11 14:41:40.066329 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:40.066248    2581 scope.go:117] "RemoveContainer" containerID="b2b9735726c06908af36bffe9130461c632ae3836526c91f0c68ffd652a305d2"
Sep 11 14:41:40.076052 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:40.076031    2581 scope.go:117] "RemoveContainer" containerID="b2b9735726c06908af36bffe9130461c632ae3836526c91f0c68ffd652a305d2"
Sep 11 14:41:40.076342 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:41:40.076321    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2b9735726c06908af36bffe9130461c632ae3836526c91f0c68ffd652a305d2\": container with ID starting with b2b9735726c06908af36bffe9130461c632ae3836526c91f0c68ffd652a305d2 not found: ID does not exist" containerID="b2b9735726c06908af36bffe9130461c632ae3836526c91f0c68ffd652a305d2"
Sep 11 14:41:40.076399 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:40.076350    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2b9735726c06908af36bffe9130461c632ae3836526c91f0c68ffd652a305d2"} err="failed to get container status \"b2b9735726c06908af36bffe9130461c632ae3836526c91f0c68ffd652a305d2\": rpc error: code = NotFound desc = could not find container \"b2b9735726c06908af36bffe9130461c632ae3836526c91f0c68ffd652a305d2\": container with ID starting with b2b9735726c06908af36bffe9130461c632ae3836526c91f0c68ffd652a305d2 not found: ID does not exist"
Sep 11 14:41:40.093996 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:40.093953    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-09b16c/sub-test-model-09b16c-kserve-7454455f57-f6gtb"]
Sep 11 14:41:40.096421 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:40.096400    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scmxx\" (UniqueName: \"kubernetes.io/projected/c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff-kube-api-access-scmxx\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:41:40.096477 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:40.096424    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff-dshm\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:41:40.096477 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:40.096435    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff-tls-certs\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:41:40.096477 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:40.096444    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff-home\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:41:40.096477 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:40.096472    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff-tmp-dir\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:41:40.096608 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:40.096482    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff-model-cache\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:41:40.104047 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:40.104016    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-09b16c/sub-test-model-09b16c-kserve-7454455f57-f6gtb"]
Sep 11 14:41:40.614650 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:40.614614    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-1c284e/sub-test-model-1c284e-kserve-66b545b97-57qnl" podUID="d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8" containerName="main" containerID="cri-o://b1bb4f0ed537a8492d1b479e91e22cb492c47c87c1aee76e1bc7898a9aecb3bf" gracePeriod=45
Sep 11 14:41:40.864963 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:40.864902    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1c284e/sub-test-model-1c284e-kserve-66b545b97-57qnl"
Sep 11 14:41:40.903961 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:40.903929    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8-model-cache\") pod \"d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8\" (UID: \"d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8\") "
Sep 11 14:41:40.904157 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:40.904049    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8-home\") pod \"d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8\" (UID: \"d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8\") "
Sep 11 14:41:40.904157 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:40.904080    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8-tmp-dir\") pod \"d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8\" (UID: \"d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8\") "
Sep 11 14:41:40.904157 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:40.904111    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8-dshm\") pod \"d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8\" (UID: \"d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8\") "
Sep 11 14:41:40.904157 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:40.904150    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ds627\" (UniqueName: \"kubernetes.io/projected/d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8-kube-api-access-ds627\") pod \"d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8\" (UID: \"d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8\") "
Sep 11 14:41:40.904380 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:40.904196    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8-tls-certs\") pod \"d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8\" (UID: \"d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8\") "
Sep 11 14:41:40.904380 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:40.904192    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8-model-cache" (OuterVolumeSpecName: "model-cache") pod "d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8" (UID: "d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:41:40.904380 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:40.904315    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8" (UID: "d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:41:40.904380 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:40.904334    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8-home" (OuterVolumeSpecName: "home") pod "d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8" (UID: "d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:41:40.904571 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:40.904548    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8-model-cache\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:41:40.904571 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:40.904562    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8-home\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:41:40.904571 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:40.904570    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8-tmp-dir\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:41:40.906297 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:40.906268    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8" (UID: "d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:41:40.906414 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:40.906353    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8-kube-api-access-ds627" (OuterVolumeSpecName: "kube-api-access-ds627") pod "d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8" (UID: "d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8"). InnerVolumeSpecName "kube-api-access-ds627". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:41:40.906636 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:40.906614    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8-dshm" (OuterVolumeSpecName: "dshm") pod "d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8" (UID: "d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:41:41.005903 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:41.005857    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8-dshm\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:41:41.005903 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:41.005895    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ds627\" (UniqueName: \"kubernetes.io/projected/d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8-kube-api-access-ds627\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:41:41.005903 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:41.005911    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8-tls-certs\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:41:41.073090 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:41.073049    2581 generic.go:356] "Generic (PLEG): container finished" podID="d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8" containerID="b1bb4f0ed537a8492d1b479e91e22cb492c47c87c1aee76e1bc7898a9aecb3bf" exitCode=0
Sep 11 14:41:41.073252 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:41.073095    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1c284e/sub-test-model-1c284e-kserve-66b545b97-57qnl"
Sep 11 14:41:41.073252 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:41.073134    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1c284e/sub-test-model-1c284e-kserve-66b545b97-57qnl" event={"ID":"d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8","Type":"ContainerDied","Data":"b1bb4f0ed537a8492d1b479e91e22cb492c47c87c1aee76e1bc7898a9aecb3bf"}
Sep 11 14:41:41.073252 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:41.073178    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1c284e/sub-test-model-1c284e-kserve-66b545b97-57qnl" event={"ID":"d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8","Type":"ContainerDied","Data":"84e0bb4eaa5e03492cef86d4f73584b45ba35566a26f528d22eb99b68285fb03"}
Sep 11 14:41:41.073252 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:41.073197    2581 scope.go:117] "RemoveContainer" containerID="b1bb4f0ed537a8492d1b479e91e22cb492c47c87c1aee76e1bc7898a9aecb3bf"
Sep 11 14:41:41.082522 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:41.082501    2581 scope.go:117] "RemoveContainer" containerID="b1bb4f0ed537a8492d1b479e91e22cb492c47c87c1aee76e1bc7898a9aecb3bf"
Sep 11 14:41:41.082797 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:41:41.082777    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1bb4f0ed537a8492d1b479e91e22cb492c47c87c1aee76e1bc7898a9aecb3bf\": container with ID starting with b1bb4f0ed537a8492d1b479e91e22cb492c47c87c1aee76e1bc7898a9aecb3bf not found: ID does not exist" containerID="b1bb4f0ed537a8492d1b479e91e22cb492c47c87c1aee76e1bc7898a9aecb3bf"
Sep 11 14:41:41.082867 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:41.082809    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1bb4f0ed537a8492d1b479e91e22cb492c47c87c1aee76e1bc7898a9aecb3bf"} err="failed to get container status \"b1bb4f0ed537a8492d1b479e91e22cb492c47c87c1aee76e1bc7898a9aecb3bf\": rpc error: code = NotFound desc = could not find container \"b1bb4f0ed537a8492d1b479e91e22cb492c47c87c1aee76e1bc7898a9aecb3bf\": container with ID starting with b1bb4f0ed537a8492d1b479e91e22cb492c47c87c1aee76e1bc7898a9aecb3bf not found: ID does not exist"
Sep 11 14:41:41.110153 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:41.110120    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1c284e/sub-test-model-1c284e-kserve-66b545b97-57qnl"]
Sep 11 14:41:41.119027 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:41.118939    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1c284e/sub-test-model-1c284e-kserve-66b545b97-57qnl"]
Sep 11 14:41:41.501394 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:41.501359    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff" path="/var/lib/kubelet/pods/c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff/volumes"
Sep 11 14:41:41.501983 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:41.501950    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8" path="/var/lib/kubelet/pods/d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8/volumes"
Sep 11 14:41:47.228754 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:47.228716    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-09b16c/rate-test-model-09b16c-kserve-6db7db4cff-h4qdc"]
Sep 11 14:41:47.231057 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:47.229099    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff" containerName="main"
Sep 11 14:41:47.231057 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:47.229110    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff" containerName="main"
Sep 11 14:41:47.231057 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:47.229123    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8" containerName="main"
Sep 11 14:41:47.231057 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:47.229129    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8" containerName="main"
Sep 11 14:41:47.231057 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:47.229182    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6b90074-5cc6-4c18-ba9a-8ac1698ab3a8" containerName="main"
Sep 11 14:41:47.231057 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:47.229194    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2e0d2ec-33d2-49de-8826-9b71f7a2a3ff" containerName="main"
Sep 11 14:41:47.231945 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:47.231929    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-09b16c/rate-test-model-09b16c-kserve-6db7db4cff-h4qdc"
Sep 11 14:41:47.240885 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:47.240856    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-09b16c\"/\"default-dockercfg-g4l97\""
Sep 11 14:41:47.241043 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:47.240856    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-09b16c\"/\"kube-root-ca.crt\""
Sep 11 14:41:47.241764 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:47.241747    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-09b16c\"/\"openshift-service-ca.crt\""
Sep 11 14:41:47.243880 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:47.243864    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-09b16c\"/\"rate-test-model-09b16c-kserve-self-signed-certs\""
Sep 11 14:41:47.262903 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:47.262871    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-09b16c/rate-test-model-09b16c-kserve-6db7db4cff-h4qdc"]
Sep 11 14:41:47.366362 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:47.366325    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b62dc2d-e00c-4947-80e0-31afcd8371b0-home\") pod \"rate-test-model-09b16c-kserve-6db7db4cff-h4qdc\" (UID: \"0b62dc2d-e00c-4947-80e0-31afcd8371b0\") " pod="ai-tenant-e2e-shared-a-w3-09b16c/rate-test-model-09b16c-kserve-6db7db4cff-h4qdc"
Sep 11 14:41:47.366696 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:47.366665    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b62dc2d-e00c-4947-80e0-31afcd8371b0-model-cache\") pod \"rate-test-model-09b16c-kserve-6db7db4cff-h4qdc\" (UID: \"0b62dc2d-e00c-4947-80e0-31afcd8371b0\") " pod="ai-tenant-e2e-shared-a-w3-09b16c/rate-test-model-09b16c-kserve-6db7db4cff-h4qdc"
Sep 11 14:41:47.366844 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:47.366731    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b62dc2d-e00c-4947-80e0-31afcd8371b0-tmp-dir\") pod \"rate-test-model-09b16c-kserve-6db7db4cff-h4qdc\" (UID: \"0b62dc2d-e00c-4947-80e0-31afcd8371b0\") " pod="ai-tenant-e2e-shared-a-w3-09b16c/rate-test-model-09b16c-kserve-6db7db4cff-h4qdc"
Sep 11 14:41:47.366916 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:47.366833    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b62dc2d-e00c-4947-80e0-31afcd8371b0-tls-certs\") pod \"rate-test-model-09b16c-kserve-6db7db4cff-h4qdc\" (UID: \"0b62dc2d-e00c-4947-80e0-31afcd8371b0\") " pod="ai-tenant-e2e-shared-a-w3-09b16c/rate-test-model-09b16c-kserve-6db7db4cff-h4qdc"
Sep 11 14:41:47.366916 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:47.366894    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b62dc2d-e00c-4947-80e0-31afcd8371b0-dshm\") pod \"rate-test-model-09b16c-kserve-6db7db4cff-h4qdc\" (UID: \"0b62dc2d-e00c-4947-80e0-31afcd8371b0\") " pod="ai-tenant-e2e-shared-a-w3-09b16c/rate-test-model-09b16c-kserve-6db7db4cff-h4qdc"
Sep 11 14:41:47.367079 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:47.366925    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8wbn\" (UniqueName: \"kubernetes.io/projected/0b62dc2d-e00c-4947-80e0-31afcd8371b0-kube-api-access-h8wbn\") pod \"rate-test-model-09b16c-kserve-6db7db4cff-h4qdc\" (UID: \"0b62dc2d-e00c-4947-80e0-31afcd8371b0\") " pod="ai-tenant-e2e-shared-a-w3-09b16c/rate-test-model-09b16c-kserve-6db7db4cff-h4qdc"
Sep 11 14:41:47.467836 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:47.467802    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b62dc2d-e00c-4947-80e0-31afcd8371b0-tls-certs\") pod \"rate-test-model-09b16c-kserve-6db7db4cff-h4qdc\" (UID: \"0b62dc2d-e00c-4947-80e0-31afcd8371b0\") " pod="ai-tenant-e2e-shared-a-w3-09b16c/rate-test-model-09b16c-kserve-6db7db4cff-h4qdc"
Sep 11 14:41:47.468064 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:47.467845    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b62dc2d-e00c-4947-80e0-31afcd8371b0-dshm\") pod \"rate-test-model-09b16c-kserve-6db7db4cff-h4qdc\" (UID: \"0b62dc2d-e00c-4947-80e0-31afcd8371b0\") " pod="ai-tenant-e2e-shared-a-w3-09b16c/rate-test-model-09b16c-kserve-6db7db4cff-h4qdc"
Sep 11 14:41:47.468064 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:47.467867    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8wbn\" (UniqueName: \"kubernetes.io/projected/0b62dc2d-e00c-4947-80e0-31afcd8371b0-kube-api-access-h8wbn\") pod \"rate-test-model-09b16c-kserve-6db7db4cff-h4qdc\" (UID: \"0b62dc2d-e00c-4947-80e0-31afcd8371b0\") " pod="ai-tenant-e2e-shared-a-w3-09b16c/rate-test-model-09b16c-kserve-6db7db4cff-h4qdc"
Sep 11 14:41:47.468064 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:47.467930    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b62dc2d-e00c-4947-80e0-31afcd8371b0-home\") pod \"rate-test-model-09b16c-kserve-6db7db4cff-h4qdc\" (UID: \"0b62dc2d-e00c-4947-80e0-31afcd8371b0\") " pod="ai-tenant-e2e-shared-a-w3-09b16c/rate-test-model-09b16c-kserve-6db7db4cff-h4qdc"
Sep 11 14:41:47.468064 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:47.467956    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b62dc2d-e00c-4947-80e0-31afcd8371b0-model-cache\") pod \"rate-test-model-09b16c-kserve-6db7db4cff-h4qdc\" (UID: \"0b62dc2d-e00c-4947-80e0-31afcd8371b0\") " pod="ai-tenant-e2e-shared-a-w3-09b16c/rate-test-model-09b16c-kserve-6db7db4cff-h4qdc"
Sep 11 14:41:47.468064 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:47.468008    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b62dc2d-e00c-4947-80e0-31afcd8371b0-tmp-dir\") pod \"rate-test-model-09b16c-kserve-6db7db4cff-h4qdc\" (UID: \"0b62dc2d-e00c-4947-80e0-31afcd8371b0\") " pod="ai-tenant-e2e-shared-a-w3-09b16c/rate-test-model-09b16c-kserve-6db7db4cff-h4qdc"
Sep 11 14:41:47.468466 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:47.468440    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b62dc2d-e00c-4947-80e0-31afcd8371b0-model-cache\") pod \"rate-test-model-09b16c-kserve-6db7db4cff-h4qdc\" (UID: \"0b62dc2d-e00c-4947-80e0-31afcd8371b0\") " pod="ai-tenant-e2e-shared-a-w3-09b16c/rate-test-model-09b16c-kserve-6db7db4cff-h4qdc"
Sep 11 14:41:47.468466 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:47.468455    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b62dc2d-e00c-4947-80e0-31afcd8371b0-tmp-dir\") pod \"rate-test-model-09b16c-kserve-6db7db4cff-h4qdc\" (UID: \"0b62dc2d-e00c-4947-80e0-31afcd8371b0\") " pod="ai-tenant-e2e-shared-a-w3-09b16c/rate-test-model-09b16c-kserve-6db7db4cff-h4qdc"
Sep 11 14:41:47.468598 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:47.468543    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b62dc2d-e00c-4947-80e0-31afcd8371b0-home\") pod \"rate-test-model-09b16c-kserve-6db7db4cff-h4qdc\" (UID: \"0b62dc2d-e00c-4947-80e0-31afcd8371b0\") " pod="ai-tenant-e2e-shared-a-w3-09b16c/rate-test-model-09b16c-kserve-6db7db4cff-h4qdc"
Sep 11 14:41:47.470173 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:47.470148    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b62dc2d-e00c-4947-80e0-31afcd8371b0-dshm\") pod \"rate-test-model-09b16c-kserve-6db7db4cff-h4qdc\" (UID: \"0b62dc2d-e00c-4947-80e0-31afcd8371b0\") " pod="ai-tenant-e2e-shared-a-w3-09b16c/rate-test-model-09b16c-kserve-6db7db4cff-h4qdc"
Sep 11 14:41:47.470524 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:47.470501    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b62dc2d-e00c-4947-80e0-31afcd8371b0-tls-certs\") pod \"rate-test-model-09b16c-kserve-6db7db4cff-h4qdc\" (UID: \"0b62dc2d-e00c-4947-80e0-31afcd8371b0\") " pod="ai-tenant-e2e-shared-a-w3-09b16c/rate-test-model-09b16c-kserve-6db7db4cff-h4qdc"
Sep 11 14:41:47.487450 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:47.487383    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8wbn\" (UniqueName: \"kubernetes.io/projected/0b62dc2d-e00c-4947-80e0-31afcd8371b0-kube-api-access-h8wbn\") pod \"rate-test-model-09b16c-kserve-6db7db4cff-h4qdc\" (UID: \"0b62dc2d-e00c-4947-80e0-31afcd8371b0\") " pod="ai-tenant-e2e-shared-a-w3-09b16c/rate-test-model-09b16c-kserve-6db7db4cff-h4qdc"
Sep 11 14:41:47.541575 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:47.541536    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-09b16c/rate-test-model-09b16c-kserve-6db7db4cff-h4qdc"
Sep 11 14:41:47.822420 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:47.822392    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-09b16c/rate-test-model-09b16c-kserve-6db7db4cff-h4qdc"]
Sep 11 14:41:47.824522 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:41:47.824492    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b62dc2d_e00c_4947_80e0_31afcd8371b0.slice/crio-41cdcc367ec64ac0fc23ff7bf5637b898aa75ca0b09ad052a645193bb9ca1d55 WatchSource:0}: Error finding container 41cdcc367ec64ac0fc23ff7bf5637b898aa75ca0b09ad052a645193bb9ca1d55: Status 404 returned error can't find the container with id 41cdcc367ec64ac0fc23ff7bf5637b898aa75ca0b09ad052a645193bb9ca1d55
Sep 11 14:41:48.111000 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:48.110892    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-09b16c/rate-test-model-09b16c-kserve-6db7db4cff-h4qdc" event={"ID":"0b62dc2d-e00c-4947-80e0-31afcd8371b0","Type":"ContainerStarted","Data":"0e2e239e7fe2b1a8013cab50481ce64a80a10376e5ae4f086204fbe6375e2450"}
Sep 11 14:41:48.111000 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:48.110944    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-09b16c/rate-test-model-09b16c-kserve-6db7db4cff-h4qdc" event={"ID":"0b62dc2d-e00c-4947-80e0-31afcd8371b0","Type":"ContainerStarted","Data":"41cdcc367ec64ac0fc23ff7bf5637b898aa75ca0b09ad052a645193bb9ca1d55"}
Sep 11 14:41:48.136050 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:48.135994    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-09b16c/rate-test-model-09b16c-kserve-6db7db4cff-h4qdc" podStartSLOduration=1.135965678 podStartE2EDuration="1.135965678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:41:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:41:48.134258316 +0000 UTC m=+1611.284808396" watchObservedRunningTime="2026-09-11 14:41:48.135965678 +0000 UTC m=+1611.286515743"
Sep 11 14:41:57.541939 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:57.541842    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-09b16c/rate-test-model-09b16c-kserve-6db7db4cff-h4qdc"
Sep 11 14:41:57.541939 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:57.541899    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-09b16c/rate-test-model-09b16c-kserve-6db7db4cff-h4qdc"
Sep 11 14:41:57.554318 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:57.554292    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-09b16c/rate-test-model-09b16c-kserve-6db7db4cff-h4qdc"
Sep 11 14:41:57.589739 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:57.589707    2581 scope.go:117] "RemoveContainer" containerID="7f82ce0f4228f338a94de1b19ed44d664f39319a1ac6d719d0a265725e071dff"
Sep 11 14:41:57.598055 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:57.598034    2581 scope.go:117] "RemoveContainer" containerID="dad5922a64c1308d360a55c2718ab7791d3cacfe85348de1f57dcbf1145af2c7"
Sep 11 14:41:58.170293 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:41:58.170266    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-09b16c/rate-test-model-09b16c-kserve-6db7db4cff-h4qdc"
Sep 11 14:42:07.855473 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:07.855437    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb"]
Sep 11 14:42:07.859138 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:07.859115    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb"
Sep 11 14:42:07.865513 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:07.865490    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-ef0ff5-openshift-default-dockercfg-c89hk\""
Sep 11 14:42:07.895507 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:07.895475    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb"]
Sep 11 14:42:07.960885 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:07.960845    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1682346d-e312-4750-90f8-28db3c360404-istiod-ca-cert\") pod \"e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb\" (UID: \"1682346d-e312-4750-90f8-28db3c360404\") " pod="openshift-ingress/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb"
Sep 11 14:42:07.961088 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:07.960905    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1682346d-e312-4750-90f8-28db3c360404-istio-data\") pod \"e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb\" (UID: \"1682346d-e312-4750-90f8-28db3c360404\") " pod="openshift-ingress/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb"
Sep 11 14:42:07.961088 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:07.960927    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzd2w\" (UniqueName: \"kubernetes.io/projected/1682346d-e312-4750-90f8-28db3c360404-kube-api-access-xzd2w\") pod \"e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb\" (UID: \"1682346d-e312-4750-90f8-28db3c360404\") " pod="openshift-ingress/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb"
Sep 11 14:42:07.961088 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:07.960948    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1682346d-e312-4750-90f8-28db3c360404-credential-socket\") pod \"e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb\" (UID: \"1682346d-e312-4750-90f8-28db3c360404\") " pod="openshift-ingress/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb"
Sep 11 14:42:07.961088 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:07.960994    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1682346d-e312-4750-90f8-28db3c360404-istio-envoy\") pod \"e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb\" (UID: \"1682346d-e312-4750-90f8-28db3c360404\") " pod="openshift-ingress/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb"
Sep 11 14:42:07.961088 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:07.961033    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1682346d-e312-4750-90f8-28db3c360404-workload-socket\") pod \"e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb\" (UID: \"1682346d-e312-4750-90f8-28db3c360404\") " pod="openshift-ingress/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb"
Sep 11 14:42:07.961320 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:07.961099    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1682346d-e312-4750-90f8-28db3c360404-istio-token\") pod \"e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb\" (UID: \"1682346d-e312-4750-90f8-28db3c360404\") " pod="openshift-ingress/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb"
Sep 11 14:42:07.961320 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:07.961198    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1682346d-e312-4750-90f8-28db3c360404-workload-certs\") pod \"e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb\" (UID: \"1682346d-e312-4750-90f8-28db3c360404\") " pod="openshift-ingress/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb"
Sep 11 14:42:07.961320 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:07.961239    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1682346d-e312-4750-90f8-28db3c360404-istio-podinfo\") pod \"e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb\" (UID: \"1682346d-e312-4750-90f8-28db3c360404\") " pod="openshift-ingress/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb"
Sep 11 14:42:08.062274 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:08.062240    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1682346d-e312-4750-90f8-28db3c360404-workload-certs\") pod \"e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb\" (UID: \"1682346d-e312-4750-90f8-28db3c360404\") " pod="openshift-ingress/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb"
Sep 11 14:42:08.062274 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:08.062278    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1682346d-e312-4750-90f8-28db3c360404-istio-podinfo\") pod \"e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb\" (UID: \"1682346d-e312-4750-90f8-28db3c360404\") " pod="openshift-ingress/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb"
Sep 11 14:42:08.062509 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:08.062318    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1682346d-e312-4750-90f8-28db3c360404-istiod-ca-cert\") pod \"e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb\" (UID: \"1682346d-e312-4750-90f8-28db3c360404\") " pod="openshift-ingress/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb"
Sep 11 14:42:08.062509 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:08.062349    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1682346d-e312-4750-90f8-28db3c360404-istio-data\") pod \"e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb\" (UID: \"1682346d-e312-4750-90f8-28db3c360404\") " pod="openshift-ingress/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb"
Sep 11 14:42:08.062509 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:08.062371    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzd2w\" (UniqueName: \"kubernetes.io/projected/1682346d-e312-4750-90f8-28db3c360404-kube-api-access-xzd2w\") pod \"e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb\" (UID: \"1682346d-e312-4750-90f8-28db3c360404\") " pod="openshift-ingress/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb"
Sep 11 14:42:08.062509 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:08.062387    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1682346d-e312-4750-90f8-28db3c360404-credential-socket\") pod \"e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb\" (UID: \"1682346d-e312-4750-90f8-28db3c360404\") " pod="openshift-ingress/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb"
Sep 11 14:42:08.062509 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:08.062443    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1682346d-e312-4750-90f8-28db3c360404-istio-envoy\") pod \"e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb\" (UID: \"1682346d-e312-4750-90f8-28db3c360404\") " pod="openshift-ingress/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb"
Sep 11 14:42:08.062509 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:08.062494    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1682346d-e312-4750-90f8-28db3c360404-workload-socket\") pod \"e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb\" (UID: \"1682346d-e312-4750-90f8-28db3c360404\") " pod="openshift-ingress/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb"
Sep 11 14:42:08.062828 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:08.062566    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1682346d-e312-4750-90f8-28db3c360404-istio-token\") pod \"e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb\" (UID: \"1682346d-e312-4750-90f8-28db3c360404\") " pod="openshift-ingress/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb"
Sep 11 14:42:08.062828 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:08.062695    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1682346d-e312-4750-90f8-28db3c360404-workload-certs\") pod \"e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb\" (UID: \"1682346d-e312-4750-90f8-28db3c360404\") " pod="openshift-ingress/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb"
Sep 11 14:42:08.062828 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:08.062745    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1682346d-e312-4750-90f8-28db3c360404-credential-socket\") pod \"e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb\" (UID: \"1682346d-e312-4750-90f8-28db3c360404\") " pod="openshift-ingress/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb"
Sep 11 14:42:08.063080 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:08.063059    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1682346d-e312-4750-90f8-28db3c360404-istio-data\") pod \"e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb\" (UID: \"1682346d-e312-4750-90f8-28db3c360404\") " pod="openshift-ingress/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb"
Sep 11 14:42:08.063149 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:08.063119    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1682346d-e312-4750-90f8-28db3c360404-workload-socket\") pod \"e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb\" (UID: \"1682346d-e312-4750-90f8-28db3c360404\") " pod="openshift-ingress/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb"
Sep 11 14:42:08.063326 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:08.063308    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1682346d-e312-4750-90f8-28db3c360404-istiod-ca-cert\") pod \"e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb\" (UID: \"1682346d-e312-4750-90f8-28db3c360404\") " pod="openshift-ingress/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb"
Sep 11 14:42:08.064698 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:08.064669    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1682346d-e312-4750-90f8-28db3c360404-istio-envoy\") pod \"e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb\" (UID: \"1682346d-e312-4750-90f8-28db3c360404\") " pod="openshift-ingress/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb"
Sep 11 14:42:08.065031 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:08.065011    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1682346d-e312-4750-90f8-28db3c360404-istio-podinfo\") pod \"e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb\" (UID: \"1682346d-e312-4750-90f8-28db3c360404\") " pod="openshift-ingress/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb"
Sep 11 14:42:08.075089 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:08.075068    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzd2w\" (UniqueName: \"kubernetes.io/projected/1682346d-e312-4750-90f8-28db3c360404-kube-api-access-xzd2w\") pod \"e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb\" (UID: \"1682346d-e312-4750-90f8-28db3c360404\") " pod="openshift-ingress/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb"
Sep 11 14:42:08.096809 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:08.096785    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1682346d-e312-4750-90f8-28db3c360404-istio-token\") pod \"e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb\" (UID: \"1682346d-e312-4750-90f8-28db3c360404\") " pod="openshift-ingress/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb"
Sep 11 14:42:08.169335 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:08.169231    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb"
Sep 11 14:42:08.460540 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:08.460510    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb"]
Sep 11 14:42:08.462282 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:42:08.462252    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1682346d_e312_4750_90f8_28db3c360404.slice/crio-ef6931dbc1ba1809d722604615ed8f2ca95edc5fd7c048e43675a146f3904782 WatchSource:0}: Error finding container ef6931dbc1ba1809d722604615ed8f2ca95edc5fd7c048e43675a146f3904782: Status 404 returned error can't find the container with id ef6931dbc1ba1809d722604615ed8f2ca95edc5fd7c048e43675a146f3904782
Sep 11 14:42:08.464559 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:08.464523    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:42:08.464660 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:08.464603    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:42:08.464723 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:08.464675    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:42:09.213214 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:09.213172    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb" event={"ID":"1682346d-e312-4750-90f8-28db3c360404","Type":"ContainerStarted","Data":"fb4ba3c19bc83157d8fbd75930c61ce3179e27697e09b1d9f6f990a043cac27b"}
Sep 11 14:42:09.213731 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:09.213238    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb" event={"ID":"1682346d-e312-4750-90f8-28db3c360404","Type":"ContainerStarted","Data":"ef6931dbc1ba1809d722604615ed8f2ca95edc5fd7c048e43675a146f3904782"}
Sep 11 14:42:09.278442 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:09.278386    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb" podStartSLOduration=2.278370725 podStartE2EDuration="2.278370725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:42:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:42:09.275637145 +0000 UTC m=+1632.426187201" watchObservedRunningTime="2026-09-11 14:42:09.278370725 +0000 UTC m=+1632.428920788"
Sep 11 14:42:10.169809 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:10.169770    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb"
Sep 11 14:42:10.174702 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:10.174673    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb"
Sep 11 14:42:10.218113 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:10.218084    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb"
Sep 11 14:42:10.219143 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:10.219123    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb"
Sep 11 14:42:16.739888 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:16.739853    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-1c284e/rate-test-model-1c284e-kserve-c87c79c79-gwhj5"]
Sep 11 14:42:16.743670 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:16.743643    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1c284e/rate-test-model-1c284e-kserve-c87c79c79-gwhj5"
Sep 11 14:42:16.747962 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:16.747939    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-1c284e\"/\"rate-test-model-1c284e-kserve-self-signed-certs\""
Sep 11 14:42:16.748166 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:16.747939    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-1c284e\"/\"openshift-service-ca.crt\""
Sep 11 14:42:16.748254 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:16.747954    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-1c284e\"/\"kube-root-ca.crt\""
Sep 11 14:42:16.748315 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:16.747994    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-1c284e\"/\"default-dockercfg-62xn9\""
Sep 11 14:42:16.766229 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:16.766195    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1c284e/rate-test-model-1c284e-kserve-c87c79c79-gwhj5"]
Sep 11 14:42:16.845045 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:16.845007    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/837b492c-a215-4094-aadb-360a6b650671-tls-certs\") pod \"rate-test-model-1c284e-kserve-c87c79c79-gwhj5\" (UID: \"837b492c-a215-4094-aadb-360a6b650671\") " pod="ai-tenant-e2e-shared-b-w3-1c284e/rate-test-model-1c284e-kserve-c87c79c79-gwhj5"
Sep 11 14:42:16.845212 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:16.845060    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/837b492c-a215-4094-aadb-360a6b650671-dshm\") pod \"rate-test-model-1c284e-kserve-c87c79c79-gwhj5\" (UID: \"837b492c-a215-4094-aadb-360a6b650671\") " pod="ai-tenant-e2e-shared-b-w3-1c284e/rate-test-model-1c284e-kserve-c87c79c79-gwhj5"
Sep 11 14:42:16.845212 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:16.845109    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6f8r\" (UniqueName: \"kubernetes.io/projected/837b492c-a215-4094-aadb-360a6b650671-kube-api-access-l6f8r\") pod \"rate-test-model-1c284e-kserve-c87c79c79-gwhj5\" (UID: \"837b492c-a215-4094-aadb-360a6b650671\") " pod="ai-tenant-e2e-shared-b-w3-1c284e/rate-test-model-1c284e-kserve-c87c79c79-gwhj5"
Sep 11 14:42:16.845291 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:16.845209    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/837b492c-a215-4094-aadb-360a6b650671-home\") pod \"rate-test-model-1c284e-kserve-c87c79c79-gwhj5\" (UID: \"837b492c-a215-4094-aadb-360a6b650671\") " pod="ai-tenant-e2e-shared-b-w3-1c284e/rate-test-model-1c284e-kserve-c87c79c79-gwhj5"
Sep 11 14:42:16.845291 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:16.845248    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/837b492c-a215-4094-aadb-360a6b650671-model-cache\") pod \"rate-test-model-1c284e-kserve-c87c79c79-gwhj5\" (UID: \"837b492c-a215-4094-aadb-360a6b650671\") " pod="ai-tenant-e2e-shared-b-w3-1c284e/rate-test-model-1c284e-kserve-c87c79c79-gwhj5"
Sep 11 14:42:16.845291 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:16.845285    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/837b492c-a215-4094-aadb-360a6b650671-tmp-dir\") pod \"rate-test-model-1c284e-kserve-c87c79c79-gwhj5\" (UID: \"837b492c-a215-4094-aadb-360a6b650671\") " pod="ai-tenant-e2e-shared-b-w3-1c284e/rate-test-model-1c284e-kserve-c87c79c79-gwhj5"
Sep 11 14:42:16.946389 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:16.946351    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/837b492c-a215-4094-aadb-360a6b650671-tls-certs\") pod \"rate-test-model-1c284e-kserve-c87c79c79-gwhj5\" (UID: \"837b492c-a215-4094-aadb-360a6b650671\") " pod="ai-tenant-e2e-shared-b-w3-1c284e/rate-test-model-1c284e-kserve-c87c79c79-gwhj5"
Sep 11 14:42:16.946389 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:16.946394    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/837b492c-a215-4094-aadb-360a6b650671-dshm\") pod \"rate-test-model-1c284e-kserve-c87c79c79-gwhj5\" (UID: \"837b492c-a215-4094-aadb-360a6b650671\") " pod="ai-tenant-e2e-shared-b-w3-1c284e/rate-test-model-1c284e-kserve-c87c79c79-gwhj5"
Sep 11 14:42:16.946640 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:16.946500    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6f8r\" (UniqueName: \"kubernetes.io/projected/837b492c-a215-4094-aadb-360a6b650671-kube-api-access-l6f8r\") pod \"rate-test-model-1c284e-kserve-c87c79c79-gwhj5\" (UID: \"837b492c-a215-4094-aadb-360a6b650671\") " pod="ai-tenant-e2e-shared-b-w3-1c284e/rate-test-model-1c284e-kserve-c87c79c79-gwhj5"
Sep 11 14:42:16.946691 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:16.946642    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/837b492c-a215-4094-aadb-360a6b650671-home\") pod \"rate-test-model-1c284e-kserve-c87c79c79-gwhj5\" (UID: \"837b492c-a215-4094-aadb-360a6b650671\") " pod="ai-tenant-e2e-shared-b-w3-1c284e/rate-test-model-1c284e-kserve-c87c79c79-gwhj5"
Sep 11 14:42:16.946691 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:16.946677    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/837b492c-a215-4094-aadb-360a6b650671-model-cache\") pod \"rate-test-model-1c284e-kserve-c87c79c79-gwhj5\" (UID: \"837b492c-a215-4094-aadb-360a6b650671\") " pod="ai-tenant-e2e-shared-b-w3-1c284e/rate-test-model-1c284e-kserve-c87c79c79-gwhj5"
Sep 11 14:42:16.946802 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:16.946715    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/837b492c-a215-4094-aadb-360a6b650671-tmp-dir\") pod \"rate-test-model-1c284e-kserve-c87c79c79-gwhj5\" (UID: \"837b492c-a215-4094-aadb-360a6b650671\") " pod="ai-tenant-e2e-shared-b-w3-1c284e/rate-test-model-1c284e-kserve-c87c79c79-gwhj5"
Sep 11 14:42:16.947058 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:16.947034    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/837b492c-a215-4094-aadb-360a6b650671-home\") pod \"rate-test-model-1c284e-kserve-c87c79c79-gwhj5\" (UID: \"837b492c-a215-4094-aadb-360a6b650671\") " pod="ai-tenant-e2e-shared-b-w3-1c284e/rate-test-model-1c284e-kserve-c87c79c79-gwhj5"
Sep 11 14:42:16.947159 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:16.947084    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/837b492c-a215-4094-aadb-360a6b650671-model-cache\") pod \"rate-test-model-1c284e-kserve-c87c79c79-gwhj5\" (UID: \"837b492c-a215-4094-aadb-360a6b650671\") " pod="ai-tenant-e2e-shared-b-w3-1c284e/rate-test-model-1c284e-kserve-c87c79c79-gwhj5"
Sep 11 14:42:16.947159 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:16.947102    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/837b492c-a215-4094-aadb-360a6b650671-tmp-dir\") pod \"rate-test-model-1c284e-kserve-c87c79c79-gwhj5\" (UID: \"837b492c-a215-4094-aadb-360a6b650671\") " pod="ai-tenant-e2e-shared-b-w3-1c284e/rate-test-model-1c284e-kserve-c87c79c79-gwhj5"
Sep 11 14:42:16.948792 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:16.948768    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/837b492c-a215-4094-aadb-360a6b650671-dshm\") pod \"rate-test-model-1c284e-kserve-c87c79c79-gwhj5\" (UID: \"837b492c-a215-4094-aadb-360a6b650671\") " pod="ai-tenant-e2e-shared-b-w3-1c284e/rate-test-model-1c284e-kserve-c87c79c79-gwhj5"
Sep 11 14:42:16.949045 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:16.949028    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/837b492c-a215-4094-aadb-360a6b650671-tls-certs\") pod \"rate-test-model-1c284e-kserve-c87c79c79-gwhj5\" (UID: \"837b492c-a215-4094-aadb-360a6b650671\") " pod="ai-tenant-e2e-shared-b-w3-1c284e/rate-test-model-1c284e-kserve-c87c79c79-gwhj5"
Sep 11 14:42:16.959703 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:16.959675    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6f8r\" (UniqueName: \"kubernetes.io/projected/837b492c-a215-4094-aadb-360a6b650671-kube-api-access-l6f8r\") pod \"rate-test-model-1c284e-kserve-c87c79c79-gwhj5\" (UID: \"837b492c-a215-4094-aadb-360a6b650671\") " pod="ai-tenant-e2e-shared-b-w3-1c284e/rate-test-model-1c284e-kserve-c87c79c79-gwhj5"
Sep 11 14:42:17.056099 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:17.055952    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1c284e/rate-test-model-1c284e-kserve-c87c79c79-gwhj5"
Sep 11 14:42:17.361903 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:17.361819    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1c284e/rate-test-model-1c284e-kserve-c87c79c79-gwhj5"]
Sep 11 14:42:17.364729 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:42:17.364706    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod837b492c_a215_4094_aadb_360a6b650671.slice/crio-04310e35c8d5171d7b8889149a8da571f1b2f72fd5f80bca288f0db8f9706fa2 WatchSource:0}: Error finding container 04310e35c8d5171d7b8889149a8da571f1b2f72fd5f80bca288f0db8f9706fa2: Status 404 returned error can't find the container with id 04310e35c8d5171d7b8889149a8da571f1b2f72fd5f80bca288f0db8f9706fa2
Sep 11 14:42:18.268872 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:18.268836    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1c284e/rate-test-model-1c284e-kserve-c87c79c79-gwhj5" event={"ID":"837b492c-a215-4094-aadb-360a6b650671","Type":"ContainerStarted","Data":"010d8989d6cb868182304658c11cd04431fdbb321e43277eb64d6aad24f6382a"}
Sep 11 14:42:18.268872 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:18.268878    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1c284e/rate-test-model-1c284e-kserve-c87c79c79-gwhj5" event={"ID":"837b492c-a215-4094-aadb-360a6b650671","Type":"ContainerStarted","Data":"04310e35c8d5171d7b8889149a8da571f1b2f72fd5f80bca288f0db8f9706fa2"}
Sep 11 14:42:18.289896 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:18.289842    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-1c284e/rate-test-model-1c284e-kserve-c87c79c79-gwhj5" podStartSLOduration=2.289827099 podStartE2EDuration="2.289827099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:42:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:42:18.288879829 +0000 UTC m=+1641.439429895" watchObservedRunningTime="2026-09-11 14:42:18.289827099 +0000 UTC m=+1641.440377221"
Sep 11 14:42:27.057085 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:27.057050    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-1c284e/rate-test-model-1c284e-kserve-c87c79c79-gwhj5"
Sep 11 14:42:27.057582 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:27.057190    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-1c284e/rate-test-model-1c284e-kserve-c87c79c79-gwhj5"
Sep 11 14:42:27.069507 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:27.069483    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-1c284e/rate-test-model-1c284e-kserve-c87c79c79-gwhj5"
Sep 11 14:42:27.333201 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:27.333113    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-1c284e/rate-test-model-1c284e-kserve-c87c79c79-gwhj5"
Sep 11 14:42:33.354668 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:33.354628    2581 generic.go:356] "Generic (PLEG): container finished" podID="1682346d-e312-4750-90f8-28db3c360404" containerID="fb4ba3c19bc83157d8fbd75930c61ce3179e27697e09b1d9f6f990a043cac27b" exitCode=0
Sep 11 14:42:33.355240 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:33.354706    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb" event={"ID":"1682346d-e312-4750-90f8-28db3c360404","Type":"ContainerDied","Data":"fb4ba3c19bc83157d8fbd75930c61ce3179e27697e09b1d9f6f990a043cac27b"}
Sep 11 14:42:33.355313 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:33.355285    2581 scope.go:117] "RemoveContainer" containerID="fb4ba3c19bc83157d8fbd75930c61ce3179e27697e09b1d9f6f990a043cac27b"
Sep 11 14:42:33.356475 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:33.356439    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:42:33.356570 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:33.356509    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:42:33.356570 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:33.356542    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:42:34.362722 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:34.362687    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb" event={"ID":"1682346d-e312-4750-90f8-28db3c360404","Type":"ContainerStarted","Data":"b47637dc74e29506cd291d587e61a78266cec677b9231cc1b4b3580e9ec078e7"}
Sep 11 14:42:35.169929 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:35.169892    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb"
Sep 11 14:42:35.170198 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:35.170142    2581 patch_prober.go:28] interesting pod/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Sep 11 14:42:35.170261 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:35.170204    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb" podUID="1682346d-e312-4750-90f8-28db3c360404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Sep 11 14:42:36.169829 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:36.169793    2581 patch_prober.go:28] interesting pod/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Sep 11 14:42:36.170264 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:36.169848    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb" podUID="1682346d-e312-4750-90f8-28db3c360404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Sep 11 14:42:37.170409 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:37.170370    2581 patch_prober.go:28] interesting pod/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Sep 11 14:42:37.170813 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:37.170434    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb" podUID="1682346d-e312-4750-90f8-28db3c360404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Sep 11 14:42:38.169909 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:38.169869    2581 patch_prober.go:28] interesting pod/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Sep 11 14:42:38.170342 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:38.169933    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb" podUID="1682346d-e312-4750-90f8-28db3c360404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Sep 11 14:42:39.170558 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:39.170517    2581 patch_prober.go:28] interesting pod/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Sep 11 14:42:39.171017 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:39.170574    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb" podUID="1682346d-e312-4750-90f8-28db3c360404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Sep 11 14:42:40.174016 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:40.173964    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb"
Sep 11 14:42:40.174498 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:40.174324    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb"
Sep 11 14:42:40.174994 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:40.174948    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb"
Sep 11 14:42:42.550386 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:42.550347    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm"]
Sep 11 14:42:42.553032 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:42.553008    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm"
Sep 11 14:42:42.557111 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:42.557088    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-56b1b6-openshift-default-dockercfg-qjw86\""
Sep 11 14:42:42.580509 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:42.580474    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm"]
Sep 11 14:42:42.694300 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:42.694255    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f05e099-7e7a-4c36-aeb8-6072904fdb5d-istiod-ca-cert\") pod \"e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm\" (UID: \"7f05e099-7e7a-4c36-aeb8-6072904fdb5d\") " pod="openshift-ingress/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm"
Sep 11 14:42:42.694300 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:42.694298    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f05e099-7e7a-4c36-aeb8-6072904fdb5d-credential-socket\") pod \"e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm\" (UID: \"7f05e099-7e7a-4c36-aeb8-6072904fdb5d\") " pod="openshift-ingress/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm"
Sep 11 14:42:42.694539 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:42.694330    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f05e099-7e7a-4c36-aeb8-6072904fdb5d-istio-podinfo\") pod \"e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm\" (UID: \"7f05e099-7e7a-4c36-aeb8-6072904fdb5d\") " pod="openshift-ingress/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm"
Sep 11 14:42:42.694539 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:42.694347    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f05e099-7e7a-4c36-aeb8-6072904fdb5d-istio-envoy\") pod \"e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm\" (UID: \"7f05e099-7e7a-4c36-aeb8-6072904fdb5d\") " pod="openshift-ingress/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm"
Sep 11 14:42:42.694539 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:42.694407    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f05e099-7e7a-4c36-aeb8-6072904fdb5d-workload-certs\") pod \"e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm\" (UID: \"7f05e099-7e7a-4c36-aeb8-6072904fdb5d\") " pod="openshift-ingress/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm"
Sep 11 14:42:42.694539 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:42.694485    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f05e099-7e7a-4c36-aeb8-6072904fdb5d-istio-token\") pod \"e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm\" (UID: \"7f05e099-7e7a-4c36-aeb8-6072904fdb5d\") " pod="openshift-ingress/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm"
Sep 11 14:42:42.694744 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:42.694611    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f05e099-7e7a-4c36-aeb8-6072904fdb5d-istio-data\") pod \"e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm\" (UID: \"7f05e099-7e7a-4c36-aeb8-6072904fdb5d\") " pod="openshift-ingress/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm"
Sep 11 14:42:42.694744 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:42.694653    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f05e099-7e7a-4c36-aeb8-6072904fdb5d-workload-socket\") pod \"e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm\" (UID: \"7f05e099-7e7a-4c36-aeb8-6072904fdb5d\") " pod="openshift-ingress/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm"
Sep 11 14:42:42.694744 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:42.694721    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gpjt\" (UniqueName: \"kubernetes.io/projected/7f05e099-7e7a-4c36-aeb8-6072904fdb5d-kube-api-access-7gpjt\") pod \"e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm\" (UID: \"7f05e099-7e7a-4c36-aeb8-6072904fdb5d\") " pod="openshift-ingress/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm"
Sep 11 14:42:42.795681 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:42.795646    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f05e099-7e7a-4c36-aeb8-6072904fdb5d-istio-podinfo\") pod \"e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm\" (UID: \"7f05e099-7e7a-4c36-aeb8-6072904fdb5d\") " pod="openshift-ingress/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm"
Sep 11 14:42:42.795681 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:42.795681    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f05e099-7e7a-4c36-aeb8-6072904fdb5d-istio-envoy\") pod \"e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm\" (UID: \"7f05e099-7e7a-4c36-aeb8-6072904fdb5d\") " pod="openshift-ingress/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm"
Sep 11 14:42:42.795940 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:42.795702    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f05e099-7e7a-4c36-aeb8-6072904fdb5d-workload-certs\") pod \"e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm\" (UID: \"7f05e099-7e7a-4c36-aeb8-6072904fdb5d\") " pod="openshift-ingress/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm"
Sep 11 14:42:42.795940 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:42.795822    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f05e099-7e7a-4c36-aeb8-6072904fdb5d-istio-token\") pod \"e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm\" (UID: \"7f05e099-7e7a-4c36-aeb8-6072904fdb5d\") " pod="openshift-ingress/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm"
Sep 11 14:42:42.795940 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:42.795928    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f05e099-7e7a-4c36-aeb8-6072904fdb5d-istio-data\") pod \"e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm\" (UID: \"7f05e099-7e7a-4c36-aeb8-6072904fdb5d\") " pod="openshift-ingress/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm"
Sep 11 14:42:42.796140 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:42.795950    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f05e099-7e7a-4c36-aeb8-6072904fdb5d-workload-socket\") pod \"e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm\" (UID: \"7f05e099-7e7a-4c36-aeb8-6072904fdb5d\") " pod="openshift-ingress/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm"
Sep 11 14:42:42.796140 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:42.796115    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f05e099-7e7a-4c36-aeb8-6072904fdb5d-workload-certs\") pod \"e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm\" (UID: \"7f05e099-7e7a-4c36-aeb8-6072904fdb5d\") " pod="openshift-ingress/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm"
Sep 11 14:42:42.796140 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:42.796125    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gpjt\" (UniqueName: \"kubernetes.io/projected/7f05e099-7e7a-4c36-aeb8-6072904fdb5d-kube-api-access-7gpjt\") pod \"e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm\" (UID: \"7f05e099-7e7a-4c36-aeb8-6072904fdb5d\") " pod="openshift-ingress/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm"
Sep 11 14:42:42.796291 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:42.796211    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f05e099-7e7a-4c36-aeb8-6072904fdb5d-istiod-ca-cert\") pod \"e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm\" (UID: \"7f05e099-7e7a-4c36-aeb8-6072904fdb5d\") " pod="openshift-ingress/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm"
Sep 11 14:42:42.796291 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:42.796247    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f05e099-7e7a-4c36-aeb8-6072904fdb5d-credential-socket\") pod \"e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm\" (UID: \"7f05e099-7e7a-4c36-aeb8-6072904fdb5d\") " pod="openshift-ingress/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm"
Sep 11 14:42:42.796379 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:42.796292    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f05e099-7e7a-4c36-aeb8-6072904fdb5d-workload-socket\") pod \"e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm\" (UID: \"7f05e099-7e7a-4c36-aeb8-6072904fdb5d\") " pod="openshift-ingress/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm"
Sep 11 14:42:42.796379 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:42.796310    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f05e099-7e7a-4c36-aeb8-6072904fdb5d-istio-data\") pod \"e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm\" (UID: \"7f05e099-7e7a-4c36-aeb8-6072904fdb5d\") " pod="openshift-ingress/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm"
Sep 11 14:42:42.796586 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:42.796565    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f05e099-7e7a-4c36-aeb8-6072904fdb5d-credential-socket\") pod \"e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm\" (UID: \"7f05e099-7e7a-4c36-aeb8-6072904fdb5d\") " pod="openshift-ingress/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm"
Sep 11 14:42:42.796868 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:42.796850    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f05e099-7e7a-4c36-aeb8-6072904fdb5d-istiod-ca-cert\") pod \"e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm\" (UID: \"7f05e099-7e7a-4c36-aeb8-6072904fdb5d\") " pod="openshift-ingress/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm"
Sep 11 14:42:42.798247 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:42.798225    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f05e099-7e7a-4c36-aeb8-6072904fdb5d-istio-envoy\") pod \"e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm\" (UID: \"7f05e099-7e7a-4c36-aeb8-6072904fdb5d\") " pod="openshift-ingress/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm"
Sep 11 14:42:42.798363 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:42.798347    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f05e099-7e7a-4c36-aeb8-6072904fdb5d-istio-podinfo\") pod \"e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm\" (UID: \"7f05e099-7e7a-4c36-aeb8-6072904fdb5d\") " pod="openshift-ingress/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm"
Sep 11 14:42:42.806212 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:42.806153    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f05e099-7e7a-4c36-aeb8-6072904fdb5d-istio-token\") pod \"e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm\" (UID: \"7f05e099-7e7a-4c36-aeb8-6072904fdb5d\") " pod="openshift-ingress/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm"
Sep 11 14:42:42.806326 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:42.806316    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gpjt\" (UniqueName: \"kubernetes.io/projected/7f05e099-7e7a-4c36-aeb8-6072904fdb5d-kube-api-access-7gpjt\") pod \"e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm\" (UID: \"7f05e099-7e7a-4c36-aeb8-6072904fdb5d\") " pod="openshift-ingress/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm"
Sep 11 14:42:42.863598 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:42.863565    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm"
Sep 11 14:42:43.144081 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:43.144055    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm"]
Sep 11 14:42:43.145873 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:42:43.145847    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f05e099_7e7a_4c36_aeb8_6072904fdb5d.slice/crio-fc0a5e0222e76de58154ce71cbd1ca2c7b0dbfc3f9dbde4ebcb599be4cf4e89c WatchSource:0}: Error finding container fc0a5e0222e76de58154ce71cbd1ca2c7b0dbfc3f9dbde4ebcb599be4cf4e89c: Status 404 returned error can't find the container with id fc0a5e0222e76de58154ce71cbd1ca2c7b0dbfc3f9dbde4ebcb599be4cf4e89c
Sep 11 14:42:43.148204 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:43.148172    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:42:43.148276 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:43.148247    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:42:43.148322 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:43.148310    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:42:43.413480 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:43.413395    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm" event={"ID":"7f05e099-7e7a-4c36-aeb8-6072904fdb5d","Type":"ContainerStarted","Data":"5b68b97c7952729108589d41f1f0c7d99df0bd48ae9c2e1691066d995d3b4f57"}
Sep 11 14:42:43.413480 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:43.413453    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm" event={"ID":"7f05e099-7e7a-4c36-aeb8-6072904fdb5d","Type":"ContainerStarted","Data":"fc0a5e0222e76de58154ce71cbd1ca2c7b0dbfc3f9dbde4ebcb599be4cf4e89c"}
Sep 11 14:42:43.446330 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:43.446278    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm" podStartSLOduration=1.446260463 podStartE2EDuration="1.446260463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:42:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:42:43.444370179 +0000 UTC m=+1666.594920267" watchObservedRunningTime="2026-09-11 14:42:43.446260463 +0000 UTC m=+1666.596810527"
Sep 11 14:42:43.864012 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:43.863945    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm"
Sep 11 14:42:44.869336 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:44.869303    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm"
Sep 11 14:42:45.425035 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:45.425005    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm"
Sep 11 14:42:45.426090 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:45.426068    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm"
Sep 11 14:42:57.627905 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:42:57.627877    2581 scope.go:117] "RemoveContainer" containerID="d3d14a566bca24f0ea195dde255a63741427e8f36339738c9eaf7e5411e6ae6f"
Sep 11 14:43:07.556305 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:07.556276    2581 generic.go:356] "Generic (PLEG): container finished" podID="7f05e099-7e7a-4c36-aeb8-6072904fdb5d" containerID="5b68b97c7952729108589d41f1f0c7d99df0bd48ae9c2e1691066d995d3b4f57" exitCode=0
Sep 11 14:43:07.556667 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:07.556351    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm" event={"ID":"7f05e099-7e7a-4c36-aeb8-6072904fdb5d","Type":"ContainerDied","Data":"5b68b97c7952729108589d41f1f0c7d99df0bd48ae9c2e1691066d995d3b4f57"}
Sep 11 14:43:07.556765 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:07.556751    2581 scope.go:117] "RemoveContainer" containerID="5b68b97c7952729108589d41f1f0c7d99df0bd48ae9c2e1691066d995d3b4f57"
Sep 11 14:43:07.558026 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:07.558000    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:43:07.558099 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:07.558069    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:43:07.558138 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:07.558103    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:43:08.563794 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:08.563761    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm" event={"ID":"7f05e099-7e7a-4c36-aeb8-6072904fdb5d","Type":"ContainerStarted","Data":"6eea61ef49275fcfcf291c1fb57e1e37f5bfa612c5ecc9e8af10b1c6aeb0a4da"}
Sep 11 14:43:08.864226 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:08.864133    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm"
Sep 11 14:43:08.864393 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:08.864349    2581 patch_prober.go:28] interesting pod/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Sep 11 14:43:08.864439 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:08.864394    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm" podUID="7f05e099-7e7a-4c36-aeb8-6072904fdb5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Sep 11 14:43:09.864207 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:09.864166    2581 patch_prober.go:28] interesting pod/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Sep 11 14:43:09.864647 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:09.864235    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm" podUID="7f05e099-7e7a-4c36-aeb8-6072904fdb5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Sep 11 14:43:10.864850 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:10.864809    2581 patch_prober.go:28] interesting pod/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Sep 11 14:43:10.865327 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:10.864864    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm" podUID="7f05e099-7e7a-4c36-aeb8-6072904fdb5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Sep 11 14:43:11.864337 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:11.864288    2581 patch_prober.go:28] interesting pod/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Sep 11 14:43:11.864552 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:11.864377    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm" podUID="7f05e099-7e7a-4c36-aeb8-6072904fdb5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Sep 11 14:43:12.622937 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:12.622891    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-09b16c/rate-test-model-09b16c-kserve-6db7db4cff-h4qdc"]
Sep 11 14:43:12.864696 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:12.864654    2581 patch_prober.go:28] interesting pod/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Sep 11 14:43:12.864872 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:12.864711    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm" podUID="7f05e099-7e7a-4c36-aeb8-6072904fdb5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Sep 11 14:43:13.742339 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:13.742299    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1c284e/rate-test-model-1c284e-kserve-c87c79c79-gwhj5"]
Sep 11 14:43:13.934494 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:13.934457    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm"
Sep 11 14:43:13.934909 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:13.934881    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm"
Sep 11 14:43:13.936236 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:13.936211    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm"
Sep 11 14:43:14.566550 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:14.566510    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8"]
Sep 11 14:43:14.569195 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:14.569172    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8"
Sep 11 14:43:14.572480 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:14.572459    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-f0ee6c-openshift-default-dockercfg-9m2k5\""
Sep 11 14:43:14.585204 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:14.585177    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8"]
Sep 11 14:43:14.652226 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:14.652080    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d6949fd-9353-442f-8feb-64d16d9eeb28-workload-certs\") pod \"e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8\" (UID: \"8d6949fd-9353-442f-8feb-64d16d9eeb28\") " pod="openshift-ingress/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8"
Sep 11 14:43:14.652588 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:14.652553    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d6949fd-9353-442f-8feb-64d16d9eeb28-istio-envoy\") pod \"e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8\" (UID: \"8d6949fd-9353-442f-8feb-64d16d9eeb28\") " pod="openshift-ingress/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8"
Sep 11 14:43:14.652717 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:14.652703    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d6949fd-9353-442f-8feb-64d16d9eeb28-istiod-ca-cert\") pod \"e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8\" (UID: \"8d6949fd-9353-442f-8feb-64d16d9eeb28\") " pod="openshift-ingress/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8"
Sep 11 14:43:14.652850 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:14.652835    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46n9v\" (UniqueName: \"kubernetes.io/projected/8d6949fd-9353-442f-8feb-64d16d9eeb28-kube-api-access-46n9v\") pod \"e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8\" (UID: \"8d6949fd-9353-442f-8feb-64d16d9eeb28\") " pod="openshift-ingress/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8"
Sep 11 14:43:14.652989 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:14.652955    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d6949fd-9353-442f-8feb-64d16d9eeb28-workload-socket\") pod \"e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8\" (UID: \"8d6949fd-9353-442f-8feb-64d16d9eeb28\") " pod="openshift-ingress/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8"
Sep 11 14:43:14.653120 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:14.653106    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d6949fd-9353-442f-8feb-64d16d9eeb28-credential-socket\") pod \"e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8\" (UID: \"8d6949fd-9353-442f-8feb-64d16d9eeb28\") " pod="openshift-ingress/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8"
Sep 11 14:43:14.653268 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:14.653253    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d6949fd-9353-442f-8feb-64d16d9eeb28-istio-data\") pod \"e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8\" (UID: \"8d6949fd-9353-442f-8feb-64d16d9eeb28\") " pod="openshift-ingress/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8"
Sep 11 14:43:14.653433 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:14.653417    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d6949fd-9353-442f-8feb-64d16d9eeb28-istio-podinfo\") pod \"e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8\" (UID: \"8d6949fd-9353-442f-8feb-64d16d9eeb28\") " pod="openshift-ingress/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8"
Sep 11 14:43:14.653637 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:14.653622    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d6949fd-9353-442f-8feb-64d16d9eeb28-istio-token\") pod \"e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8\" (UID: \"8d6949fd-9353-442f-8feb-64d16d9eeb28\") " pod="openshift-ingress/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8"
Sep 11 14:43:14.754724 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:14.754684    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d6949fd-9353-442f-8feb-64d16d9eeb28-workload-certs\") pod \"e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8\" (UID: \"8d6949fd-9353-442f-8feb-64d16d9eeb28\") " pod="openshift-ingress/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8"
Sep 11 14:43:14.755238 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:14.754740    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d6949fd-9353-442f-8feb-64d16d9eeb28-istio-envoy\") pod \"e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8\" (UID: \"8d6949fd-9353-442f-8feb-64d16d9eeb28\") " pod="openshift-ingress/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8"
Sep 11 14:43:14.755238 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:14.754765    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d6949fd-9353-442f-8feb-64d16d9eeb28-istiod-ca-cert\") pod \"e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8\" (UID: \"8d6949fd-9353-442f-8feb-64d16d9eeb28\") " pod="openshift-ingress/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8"
Sep 11 14:43:14.755238 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:14.754788    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46n9v\" (UniqueName: \"kubernetes.io/projected/8d6949fd-9353-442f-8feb-64d16d9eeb28-kube-api-access-46n9v\") pod \"e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8\" (UID: \"8d6949fd-9353-442f-8feb-64d16d9eeb28\") " pod="openshift-ingress/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8"
Sep 11 14:43:14.755238 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:14.754813    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d6949fd-9353-442f-8feb-64d16d9eeb28-workload-socket\") pod \"e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8\" (UID: \"8d6949fd-9353-442f-8feb-64d16d9eeb28\") " pod="openshift-ingress/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8"
Sep 11 14:43:14.755238 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:14.754852    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d6949fd-9353-442f-8feb-64d16d9eeb28-credential-socket\") pod \"e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8\" (UID: \"8d6949fd-9353-442f-8feb-64d16d9eeb28\") " pod="openshift-ingress/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8"
Sep 11 14:43:14.755238 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:14.754887    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d6949fd-9353-442f-8feb-64d16d9eeb28-istio-data\") pod \"e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8\" (UID: \"8d6949fd-9353-442f-8feb-64d16d9eeb28\") " pod="openshift-ingress/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8"
Sep 11 14:43:14.755238 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:14.754958    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d6949fd-9353-442f-8feb-64d16d9eeb28-istio-podinfo\") pod \"e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8\" (UID: \"8d6949fd-9353-442f-8feb-64d16d9eeb28\") " pod="openshift-ingress/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8"
Sep 11 14:43:14.755238 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:14.755086    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d6949fd-9353-442f-8feb-64d16d9eeb28-istio-token\") pod \"e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8\" (UID: \"8d6949fd-9353-442f-8feb-64d16d9eeb28\") " pod="openshift-ingress/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8"
Sep 11 14:43:14.755797 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:14.755761    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d6949fd-9353-442f-8feb-64d16d9eeb28-workload-certs\") pod \"e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8\" (UID: \"8d6949fd-9353-442f-8feb-64d16d9eeb28\") " pod="openshift-ingress/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8"
Sep 11 14:43:14.755919 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:14.755820    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d6949fd-9353-442f-8feb-64d16d9eeb28-workload-socket\") pod \"e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8\" (UID: \"8d6949fd-9353-442f-8feb-64d16d9eeb28\") " pod="openshift-ingress/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8"
Sep 11 14:43:14.756121 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:14.756100    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d6949fd-9353-442f-8feb-64d16d9eeb28-credential-socket\") pod \"e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8\" (UID: \"8d6949fd-9353-442f-8feb-64d16d9eeb28\") " pod="openshift-ingress/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8"
Sep 11 14:43:14.756178 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:14.756138    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d6949fd-9353-442f-8feb-64d16d9eeb28-istiod-ca-cert\") pod \"e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8\" (UID: \"8d6949fd-9353-442f-8feb-64d16d9eeb28\") " pod="openshift-ingress/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8"
Sep 11 14:43:14.756230 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:14.756208    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d6949fd-9353-442f-8feb-64d16d9eeb28-istio-data\") pod \"e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8\" (UID: \"8d6949fd-9353-442f-8feb-64d16d9eeb28\") " pod="openshift-ingress/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8"
Sep 11 14:43:14.757685 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:14.757655    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d6949fd-9353-442f-8feb-64d16d9eeb28-istio-envoy\") pod \"e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8\" (UID: \"8d6949fd-9353-442f-8feb-64d16d9eeb28\") " pod="openshift-ingress/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8"
Sep 11 14:43:14.757914 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:14.757895    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d6949fd-9353-442f-8feb-64d16d9eeb28-istio-podinfo\") pod \"e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8\" (UID: \"8d6949fd-9353-442f-8feb-64d16d9eeb28\") " pod="openshift-ingress/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8"
Sep 11 14:43:14.765239 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:14.765219    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46n9v\" (UniqueName: \"kubernetes.io/projected/8d6949fd-9353-442f-8feb-64d16d9eeb28-kube-api-access-46n9v\") pod \"e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8\" (UID: \"8d6949fd-9353-442f-8feb-64d16d9eeb28\") " pod="openshift-ingress/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8"
Sep 11 14:43:14.765327 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:14.765292    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d6949fd-9353-442f-8feb-64d16d9eeb28-istio-token\") pod \"e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8\" (UID: \"8d6949fd-9353-442f-8feb-64d16d9eeb28\") " pod="openshift-ingress/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8"
Sep 11 14:43:14.880358 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:14.880256    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8"
Sep 11 14:43:15.164850 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:15.164821    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8"]
Sep 11 14:43:15.167572 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:43:15.167532    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d6949fd_9353_442f_8feb_64d16d9eeb28.slice/crio-b235669eb5e591f14e6b31a880ea2c646639f7ed0145b3fb3756b1a1d6bd29b1 WatchSource:0}: Error finding container b235669eb5e591f14e6b31a880ea2c646639f7ed0145b3fb3756b1a1d6bd29b1: Status 404 returned error can't find the container with id b235669eb5e591f14e6b31a880ea2c646639f7ed0145b3fb3756b1a1d6bd29b1
Sep 11 14:43:15.169872 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:15.169836    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:43:15.169987 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:15.169917    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:43:15.170061 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:15.170019    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:43:15.603667 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:15.603631    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8" event={"ID":"8d6949fd-9353-442f-8feb-64d16d9eeb28","Type":"ContainerStarted","Data":"b25863dc77d6e57c5bb3c4362d0c1730fb2df3474767d2cec0845b2dbc31bdb1"}
Sep 11 14:43:15.603844 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:15.603695    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8" event={"ID":"8d6949fd-9353-442f-8feb-64d16d9eeb28","Type":"ContainerStarted","Data":"b235669eb5e591f14e6b31a880ea2c646639f7ed0145b3fb3756b1a1d6bd29b1"}
Sep 11 14:43:15.632175 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:15.632105    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8" podStartSLOduration=1.632086178 podStartE2EDuration="1.632086178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:43:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:43:15.63142537 +0000 UTC m=+1698.781975435" watchObservedRunningTime="2026-09-11 14:43:15.632086178 +0000 UTC m=+1698.782636300"
Sep 11 14:43:15.881193 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:15.881100    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8"
Sep 11 14:43:16.885809 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:16.885779    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8"
Sep 11 14:43:17.614528 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:17.614485    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8"
Sep 11 14:43:17.615608 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:17.615583    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8"
Sep 11 14:43:18.747105 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:18.747072    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef0ff5-28rdg"]
Sep 11 14:43:18.749415 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:18.749397    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef0ff5-28rdg"
Sep 11 14:43:18.752251 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:18.752217    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 14:43:18.752395 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:18.752276    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hcm4g\""
Sep 11 14:43:18.762557 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:18.762531    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef0ff5-28rdg"]
Sep 11 14:43:18.901438 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:18.901393    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/802a0dd2-6cdf-4799-b564-13a166b5b2bd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ef0ff5-28rdg\" (UID: \"802a0dd2-6cdf-4799-b564-13a166b5b2bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef0ff5-28rdg"
Sep 11 14:43:19.003057 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:19.002928    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/802a0dd2-6cdf-4799-b564-13a166b5b2bd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ef0ff5-28rdg\" (UID: \"802a0dd2-6cdf-4799-b564-13a166b5b2bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef0ff5-28rdg"
Sep 11 14:43:19.003570 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:19.003548    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/802a0dd2-6cdf-4799-b564-13a166b5b2bd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ef0ff5-28rdg\" (UID: \"802a0dd2-6cdf-4799-b564-13a166b5b2bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef0ff5-28rdg"
Sep 11 14:43:19.059961 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:19.059927    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef0ff5-28rdg"
Sep 11 14:43:19.342713 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:19.342684    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef0ff5-28rdg"]
Sep 11 14:43:19.344721 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:43:19.344691    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod802a0dd2_6cdf_4799_b564_13a166b5b2bd.slice/crio-cc295b37367d1c57bf84754002004b35c3e7b0cf287803d13c28f18204a9b04d WatchSource:0}: Error finding container cc295b37367d1c57bf84754002004b35c3e7b0cf287803d13c28f18204a9b04d: Status 404 returned error can't find the container with id cc295b37367d1c57bf84754002004b35c3e7b0cf287803d13c28f18204a9b04d
Sep 11 14:43:19.625895 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:19.625856    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef0ff5-28rdg" event={"ID":"802a0dd2-6cdf-4799-b564-13a166b5b2bd","Type":"ContainerStarted","Data":"ddb4fa2f8d3acd5b5b7075639e30b3ad6921c4e96a4ec4d199d93d54dd27e0f8"}
Sep 11 14:43:19.626080 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:19.625913    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef0ff5-28rdg" event={"ID":"802a0dd2-6cdf-4799-b564-13a166b5b2bd","Type":"ContainerStarted","Data":"cc295b37367d1c57bf84754002004b35c3e7b0cf287803d13c28f18204a9b04d"}
Sep 11 14:43:19.646293 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:19.646239    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef0ff5-28rdg" podStartSLOduration=1.646222641 podStartE2EDuration="1.646222641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:43:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:43:19.645057489 +0000 UTC m=+1702.795607547" watchObservedRunningTime="2026-09-11 14:43:19.646222641 +0000 UTC m=+1702.796772705"
Sep 11 14:43:20.633400 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:20.633362    2581 generic.go:356] "Generic (PLEG): container finished" podID="802a0dd2-6cdf-4799-b564-13a166b5b2bd" containerID="ddb4fa2f8d3acd5b5b7075639e30b3ad6921c4e96a4ec4d199d93d54dd27e0f8" exitCode=0
Sep 11 14:43:20.633799 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:20.633446    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef0ff5-28rdg" event={"ID":"802a0dd2-6cdf-4799-b564-13a166b5b2bd","Type":"ContainerDied","Data":"ddb4fa2f8d3acd5b5b7075639e30b3ad6921c4e96a4ec4d199d93d54dd27e0f8"}
Sep 11 14:43:21.775245 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:21.775221    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef0ff5-28rdg"
Sep 11 14:43:21.828954 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:21.828923    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/802a0dd2-6cdf-4799-b564-13a166b5b2bd-maas-api-service-ca\") pod \"802a0dd2-6cdf-4799-b564-13a166b5b2bd\" (UID: \"802a0dd2-6cdf-4799-b564-13a166b5b2bd\") "
Sep 11 14:43:21.829270 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:21.829247    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/802a0dd2-6cdf-4799-b564-13a166b5b2bd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "802a0dd2-6cdf-4799-b564-13a166b5b2bd" (UID: "802a0dd2-6cdf-4799-b564-13a166b5b2bd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:43:21.930419 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:21.930319    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/802a0dd2-6cdf-4799-b564-13a166b5b2bd-maas-api-service-ca\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:43:22.646187 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:22.646148    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef0ff5-28rdg" event={"ID":"802a0dd2-6cdf-4799-b564-13a166b5b2bd","Type":"ContainerDied","Data":"cc295b37367d1c57bf84754002004b35c3e7b0cf287803d13c28f18204a9b04d"}
Sep 11 14:43:22.646368 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:22.646201    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc295b37367d1c57bf84754002004b35c3e7b0cf287803d13c28f18204a9b04d"
Sep 11 14:43:22.646368 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:22.646168    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef0ff5-28rdg"
Sep 11 14:43:27.644989 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:27.644913    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-09b16c/rate-test-model-09b16c-kserve-6db7db4cff-h4qdc" podUID="0b62dc2d-e00c-4947-80e0-31afcd8371b0" containerName="main" containerID="cri-o://0e2e239e7fe2b1a8013cab50481ce64a80a10376e5ae4f086204fbe6375e2450" gracePeriod=45
Sep 11 14:43:27.892720 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:27.892692    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-09b16c/rate-test-model-09b16c-kserve-6db7db4cff-h4qdc"
Sep 11 14:43:27.991554 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:27.991524    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b62dc2d-e00c-4947-80e0-31afcd8371b0-tmp-dir\") pod \"0b62dc2d-e00c-4947-80e0-31afcd8371b0\" (UID: \"0b62dc2d-e00c-4947-80e0-31afcd8371b0\") "
Sep 11 14:43:27.991714 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:27.991604    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b62dc2d-e00c-4947-80e0-31afcd8371b0-tls-certs\") pod \"0b62dc2d-e00c-4947-80e0-31afcd8371b0\" (UID: \"0b62dc2d-e00c-4947-80e0-31afcd8371b0\") "
Sep 11 14:43:27.991714 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:27.991629    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b62dc2d-e00c-4947-80e0-31afcd8371b0-dshm\") pod \"0b62dc2d-e00c-4947-80e0-31afcd8371b0\" (UID: \"0b62dc2d-e00c-4947-80e0-31afcd8371b0\") "
Sep 11 14:43:27.991714 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:27.991695    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b62dc2d-e00c-4947-80e0-31afcd8371b0-home\") pod \"0b62dc2d-e00c-4947-80e0-31afcd8371b0\" (UID: \"0b62dc2d-e00c-4947-80e0-31afcd8371b0\") "
Sep 11 14:43:27.991843 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:27.991734    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8wbn\" (UniqueName: \"kubernetes.io/projected/0b62dc2d-e00c-4947-80e0-31afcd8371b0-kube-api-access-h8wbn\") pod \"0b62dc2d-e00c-4947-80e0-31afcd8371b0\" (UID: \"0b62dc2d-e00c-4947-80e0-31afcd8371b0\") "
Sep 11 14:43:27.991843 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:27.991794    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b62dc2d-e00c-4947-80e0-31afcd8371b0-model-cache\") pod \"0b62dc2d-e00c-4947-80e0-31afcd8371b0\" (UID: \"0b62dc2d-e00c-4947-80e0-31afcd8371b0\") "
Sep 11 14:43:27.991843 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:27.991822    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b62dc2d-e00c-4947-80e0-31afcd8371b0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0b62dc2d-e00c-4947-80e0-31afcd8371b0" (UID: "0b62dc2d-e00c-4947-80e0-31afcd8371b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:43:27.992029 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:27.992001    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b62dc2d-e00c-4947-80e0-31afcd8371b0-home" (OuterVolumeSpecName: "home") pod "0b62dc2d-e00c-4947-80e0-31afcd8371b0" (UID: "0b62dc2d-e00c-4947-80e0-31afcd8371b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:43:27.992178 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:27.992154    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b62dc2d-e00c-4947-80e0-31afcd8371b0-model-cache" (OuterVolumeSpecName: "model-cache") pod "0b62dc2d-e00c-4947-80e0-31afcd8371b0" (UID: "0b62dc2d-e00c-4947-80e0-31afcd8371b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:43:27.992258 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:27.992187    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b62dc2d-e00c-4947-80e0-31afcd8371b0-tmp-dir\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:43:27.992258 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:27.992212    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b62dc2d-e00c-4947-80e0-31afcd8371b0-home\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:43:27.993817 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:27.993791    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b62dc2d-e00c-4947-80e0-31afcd8371b0-dshm" (OuterVolumeSpecName: "dshm") pod "0b62dc2d-e00c-4947-80e0-31afcd8371b0" (UID: "0b62dc2d-e00c-4947-80e0-31afcd8371b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:43:27.993914 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:27.993839    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b62dc2d-e00c-4947-80e0-31afcd8371b0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0b62dc2d-e00c-4947-80e0-31afcd8371b0" (UID: "0b62dc2d-e00c-4947-80e0-31afcd8371b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:43:27.993914 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:27.993853    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b62dc2d-e00c-4947-80e0-31afcd8371b0-kube-api-access-h8wbn" (OuterVolumeSpecName: "kube-api-access-h8wbn") pod "0b62dc2d-e00c-4947-80e0-31afcd8371b0" (UID: "0b62dc2d-e00c-4947-80e0-31afcd8371b0"). InnerVolumeSpecName "kube-api-access-h8wbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:43:28.093575 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:28.093536    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b62dc2d-e00c-4947-80e0-31afcd8371b0-tls-certs\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:43:28.093575 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:28.093566    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b62dc2d-e00c-4947-80e0-31afcd8371b0-dshm\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:43:28.093575 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:28.093576    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8wbn\" (UniqueName: \"kubernetes.io/projected/0b62dc2d-e00c-4947-80e0-31afcd8371b0-kube-api-access-h8wbn\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:43:28.093812 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:28.093586    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b62dc2d-e00c-4947-80e0-31afcd8371b0-model-cache\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:43:28.679421 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:28.679390    2581 generic.go:356] "Generic (PLEG): container finished" podID="0b62dc2d-e00c-4947-80e0-31afcd8371b0" containerID="0e2e239e7fe2b1a8013cab50481ce64a80a10376e5ae4f086204fbe6375e2450" exitCode=0
Sep 11 14:43:28.679878 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:28.679432    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-09b16c/rate-test-model-09b16c-kserve-6db7db4cff-h4qdc"
Sep 11 14:43:28.679878 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:28.679435    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-09b16c/rate-test-model-09b16c-kserve-6db7db4cff-h4qdc" event={"ID":"0b62dc2d-e00c-4947-80e0-31afcd8371b0","Type":"ContainerDied","Data":"0e2e239e7fe2b1a8013cab50481ce64a80a10376e5ae4f086204fbe6375e2450"}
Sep 11 14:43:28.679878 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:28.679555    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-09b16c/rate-test-model-09b16c-kserve-6db7db4cff-h4qdc" event={"ID":"0b62dc2d-e00c-4947-80e0-31afcd8371b0","Type":"ContainerDied","Data":"41cdcc367ec64ac0fc23ff7bf5637b898aa75ca0b09ad052a645193bb9ca1d55"}
Sep 11 14:43:28.679878 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:28.679588    2581 scope.go:117] "RemoveContainer" containerID="0e2e239e7fe2b1a8013cab50481ce64a80a10376e5ae4f086204fbe6375e2450"
Sep 11 14:43:28.690006 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:28.689981    2581 scope.go:117] "RemoveContainer" containerID="0e2e239e7fe2b1a8013cab50481ce64a80a10376e5ae4f086204fbe6375e2450"
Sep 11 14:43:28.690326 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:43:28.690307    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e2e239e7fe2b1a8013cab50481ce64a80a10376e5ae4f086204fbe6375e2450\": container with ID starting with 0e2e239e7fe2b1a8013cab50481ce64a80a10376e5ae4f086204fbe6375e2450 not found: ID does not exist" containerID="0e2e239e7fe2b1a8013cab50481ce64a80a10376e5ae4f086204fbe6375e2450"
Sep 11 14:43:28.690402 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:28.690332    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e2e239e7fe2b1a8013cab50481ce64a80a10376e5ae4f086204fbe6375e2450"} err="failed to get container status \"0e2e239e7fe2b1a8013cab50481ce64a80a10376e5ae4f086204fbe6375e2450\": rpc error: code = NotFound desc = could not find container \"0e2e239e7fe2b1a8013cab50481ce64a80a10376e5ae4f086204fbe6375e2450\": container with ID starting with 0e2e239e7fe2b1a8013cab50481ce64a80a10376e5ae4f086204fbe6375e2450 not found: ID does not exist"
Sep 11 14:43:28.704485 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:28.704455    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-09b16c/rate-test-model-09b16c-kserve-6db7db4cff-h4qdc"]
Sep 11 14:43:28.712507 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:28.712477    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-09b16c/rate-test-model-09b16c-kserve-6db7db4cff-h4qdc"]
Sep 11 14:43:28.761110 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:28.761071    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-1c284e/rate-test-model-1c284e-kserve-c87c79c79-gwhj5" podUID="837b492c-a215-4094-aadb-360a6b650671" containerName="main" containerID="cri-o://010d8989d6cb868182304658c11cd04431fdbb321e43277eb64d6aad24f6382a" gracePeriod=45
Sep 11 14:43:29.019670 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:29.019648    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1c284e/rate-test-model-1c284e-kserve-c87c79c79-gwhj5"
Sep 11 14:43:29.103358 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:29.103310    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/837b492c-a215-4094-aadb-360a6b650671-tls-certs\") pod \"837b492c-a215-4094-aadb-360a6b650671\" (UID: \"837b492c-a215-4094-aadb-360a6b650671\") "
Sep 11 14:43:29.103358 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:29.103361    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/837b492c-a215-4094-aadb-360a6b650671-model-cache\") pod \"837b492c-a215-4094-aadb-360a6b650671\" (UID: \"837b492c-a215-4094-aadb-360a6b650671\") "
Sep 11 14:43:29.103601 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:29.103394    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l6f8r\" (UniqueName: \"kubernetes.io/projected/837b492c-a215-4094-aadb-360a6b650671-kube-api-access-l6f8r\") pod \"837b492c-a215-4094-aadb-360a6b650671\" (UID: \"837b492c-a215-4094-aadb-360a6b650671\") "
Sep 11 14:43:29.103601 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:29.103537    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/837b492c-a215-4094-aadb-360a6b650671-tmp-dir\") pod \"837b492c-a215-4094-aadb-360a6b650671\" (UID: \"837b492c-a215-4094-aadb-360a6b650671\") "
Sep 11 14:43:29.103723 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:29.103618    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/837b492c-a215-4094-aadb-360a6b650671-home\") pod \"837b492c-a215-4094-aadb-360a6b650671\" (UID: \"837b492c-a215-4094-aadb-360a6b650671\") "
Sep 11 14:43:29.103723 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:29.103653    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/837b492c-a215-4094-aadb-360a6b650671-dshm\") pod \"837b492c-a215-4094-aadb-360a6b650671\" (UID: \"837b492c-a215-4094-aadb-360a6b650671\") "
Sep 11 14:43:29.103829 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:29.103650    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/837b492c-a215-4094-aadb-360a6b650671-model-cache" (OuterVolumeSpecName: "model-cache") pod "837b492c-a215-4094-aadb-360a6b650671" (UID: "837b492c-a215-4094-aadb-360a6b650671"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:43:29.103829 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:29.103802    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/837b492c-a215-4094-aadb-360a6b650671-home" (OuterVolumeSpecName: "home") pod "837b492c-a215-4094-aadb-360a6b650671" (UID: "837b492c-a215-4094-aadb-360a6b650671"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:43:29.103933 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:29.103823    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/837b492c-a215-4094-aadb-360a6b650671-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "837b492c-a215-4094-aadb-360a6b650671" (UID: "837b492c-a215-4094-aadb-360a6b650671"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:43:29.104115 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:29.104097    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/837b492c-a215-4094-aadb-360a6b650671-model-cache\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:43:29.104192 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:29.104118    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/837b492c-a215-4094-aadb-360a6b650671-tmp-dir\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:43:29.104192 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:29.104141    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/837b492c-a215-4094-aadb-360a6b650671-home\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:43:29.105622 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:29.105593    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/837b492c-a215-4094-aadb-360a6b650671-dshm" (OuterVolumeSpecName: "dshm") pod "837b492c-a215-4094-aadb-360a6b650671" (UID: "837b492c-a215-4094-aadb-360a6b650671"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:43:29.105622 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:29.105597    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/837b492c-a215-4094-aadb-360a6b650671-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "837b492c-a215-4094-aadb-360a6b650671" (UID: "837b492c-a215-4094-aadb-360a6b650671"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:43:29.105622 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:29.105594    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/837b492c-a215-4094-aadb-360a6b650671-kube-api-access-l6f8r" (OuterVolumeSpecName: "kube-api-access-l6f8r") pod "837b492c-a215-4094-aadb-360a6b650671" (UID: "837b492c-a215-4094-aadb-360a6b650671"). InnerVolumeSpecName "kube-api-access-l6f8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:43:29.205180 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:29.205142    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6f8r\" (UniqueName: \"kubernetes.io/projected/837b492c-a215-4094-aadb-360a6b650671-kube-api-access-l6f8r\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:43:29.205180 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:29.205172    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/837b492c-a215-4094-aadb-360a6b650671-dshm\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:43:29.205180 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:29.205181    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/837b492c-a215-4094-aadb-360a6b650671-tls-certs\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:43:29.499515 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:29.499482    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b62dc2d-e00c-4947-80e0-31afcd8371b0" path="/var/lib/kubelet/pods/0b62dc2d-e00c-4947-80e0-31afcd8371b0/volumes"
Sep 11 14:43:29.686841 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:29.686807    2581 generic.go:356] "Generic (PLEG): container finished" podID="837b492c-a215-4094-aadb-360a6b650671" containerID="010d8989d6cb868182304658c11cd04431fdbb321e43277eb64d6aad24f6382a" exitCode=0
Sep 11 14:43:29.687303 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:29.686853    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1c284e/rate-test-model-1c284e-kserve-c87c79c79-gwhj5"
Sep 11 14:43:29.687303 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:29.686858    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1c284e/rate-test-model-1c284e-kserve-c87c79c79-gwhj5" event={"ID":"837b492c-a215-4094-aadb-360a6b650671","Type":"ContainerDied","Data":"010d8989d6cb868182304658c11cd04431fdbb321e43277eb64d6aad24f6382a"}
Sep 11 14:43:29.687303 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:29.686986    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1c284e/rate-test-model-1c284e-kserve-c87c79c79-gwhj5" event={"ID":"837b492c-a215-4094-aadb-360a6b650671","Type":"ContainerDied","Data":"04310e35c8d5171d7b8889149a8da571f1b2f72fd5f80bca288f0db8f9706fa2"}
Sep 11 14:43:29.687303 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:29.687011    2581 scope.go:117] "RemoveContainer" containerID="010d8989d6cb868182304658c11cd04431fdbb321e43277eb64d6aad24f6382a"
Sep 11 14:43:29.696197 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:29.696174    2581 scope.go:117] "RemoveContainer" containerID="010d8989d6cb868182304658c11cd04431fdbb321e43277eb64d6aad24f6382a"
Sep 11 14:43:29.696535 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:43:29.696517    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"010d8989d6cb868182304658c11cd04431fdbb321e43277eb64d6aad24f6382a\": container with ID starting with 010d8989d6cb868182304658c11cd04431fdbb321e43277eb64d6aad24f6382a not found: ID does not exist" containerID="010d8989d6cb868182304658c11cd04431fdbb321e43277eb64d6aad24f6382a"
Sep 11 14:43:29.696605 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:29.696543    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"010d8989d6cb868182304658c11cd04431fdbb321e43277eb64d6aad24f6382a"} err="failed to get container status \"010d8989d6cb868182304658c11cd04431fdbb321e43277eb64d6aad24f6382a\": rpc error: code = NotFound desc = could not find container \"010d8989d6cb868182304658c11cd04431fdbb321e43277eb64d6aad24f6382a\": container with ID starting with 010d8989d6cb868182304658c11cd04431fdbb321e43277eb64d6aad24f6382a not found: ID does not exist"
Sep 11 14:43:29.709243 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:29.709208    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1c284e/rate-test-model-1c284e-kserve-c87c79c79-gwhj5"]
Sep 11 14:43:29.715115 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:29.715089    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1c284e/rate-test-model-1c284e-kserve-c87c79c79-gwhj5"]
Sep 11 14:43:31.500337 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:31.500301    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="837b492c-a215-4094-aadb-360a6b650671" path="/var/lib/kubelet/pods/837b492c-a215-4094-aadb-360a6b650671/volumes"
Sep 11 14:43:34.678788 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:34.678750    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef0ff5-28rdg"]
Sep 11 14:43:34.693477 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:34.693450    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef0ff5-28rdg"]
Sep 11 14:43:35.504883 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:35.504836    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="802a0dd2-6cdf-4799-b564-13a166b5b2bd" path="/var/lib/kubelet/pods/802a0dd2-6cdf-4799-b564-13a166b5b2bd/volumes"
Sep 11 14:43:43.639469 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:43.638794    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb"]
Sep 11 14:43:43.639469 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:43.639091    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb" podUID="1682346d-e312-4750-90f8-28db3c360404" containerName="istio-proxy" containerID="cri-o://b47637dc74e29506cd291d587e61a78266cec677b9231cc1b4b3580e9ec078e7" gracePeriod=30
Sep 11 14:43:43.722651 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:43.722618    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts"]
Sep 11 14:43:43.723143 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:43.723125    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="802a0dd2-6cdf-4799-b564-13a166b5b2bd" containerName="revoke-keys"
Sep 11 14:43:43.723198 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:43.723145    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="802a0dd2-6cdf-4799-b564-13a166b5b2bd" containerName="revoke-keys"
Sep 11 14:43:43.723198 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:43.723155    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b62dc2d-e00c-4947-80e0-31afcd8371b0" containerName="main"
Sep 11 14:43:43.723198 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:43.723161    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b62dc2d-e00c-4947-80e0-31afcd8371b0" containerName="main"
Sep 11 14:43:43.723198 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:43.723167    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="837b492c-a215-4094-aadb-360a6b650671" containerName="main"
Sep 11 14:43:43.723198 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:43.723172    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="837b492c-a215-4094-aadb-360a6b650671" containerName="main"
Sep 11 14:43:43.723415 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:43.723239    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="802a0dd2-6cdf-4799-b564-13a166b5b2bd" containerName="revoke-keys"
Sep 11 14:43:43.723415 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:43.723248    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="837b492c-a215-4094-aadb-360a6b650671" containerName="main"
Sep 11 14:43:43.723415 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:43.723258    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b62dc2d-e00c-4947-80e0-31afcd8371b0" containerName="main"
Sep 11 14:43:43.726496 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:43.726475    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts"
Sep 11 14:43:43.729558 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:43.729539    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-40d3a2-openshift-default-dockercfg-tc92p\""
Sep 11 14:43:43.741442 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:43.741421    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts"]
Sep 11 14:43:43.766139 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:43.766112    2581 generic.go:356] "Generic (PLEG): container finished" podID="8d6949fd-9353-442f-8feb-64d16d9eeb28" containerID="b25863dc77d6e57c5bb3c4362d0c1730fb2df3474767d2cec0845b2dbc31bdb1" exitCode=0
Sep 11 14:43:43.766315 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:43.766194    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8" event={"ID":"8d6949fd-9353-442f-8feb-64d16d9eeb28","Type":"ContainerDied","Data":"b25863dc77d6e57c5bb3c4362d0c1730fb2df3474767d2cec0845b2dbc31bdb1"}
Sep 11 14:43:43.766745 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:43.766728    2581 scope.go:117] "RemoveContainer" containerID="b25863dc77d6e57c5bb3c4362d0c1730fb2df3474767d2cec0845b2dbc31bdb1"
Sep 11 14:43:43.768083 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:43.768046    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:43:43.768190 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:43.768115    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:43:43.768190 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:43.768153    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:43:43.867166 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:43.867135    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c69a781-1386-4047-979a-8eeb4e0709df-workload-socket\") pod \"e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts\" (UID: \"4c69a781-1386-4047-979a-8eeb4e0709df\") " pod="openshift-ingress/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts"
Sep 11 14:43:43.867351 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:43.867182    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-696jz\" (UniqueName: \"kubernetes.io/projected/4c69a781-1386-4047-979a-8eeb4e0709df-kube-api-access-696jz\") pod \"e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts\" (UID: \"4c69a781-1386-4047-979a-8eeb4e0709df\") " pod="openshift-ingress/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts"
Sep 11 14:43:43.867351 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:43.867205    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c69a781-1386-4047-979a-8eeb4e0709df-istio-envoy\") pod \"e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts\" (UID: \"4c69a781-1386-4047-979a-8eeb4e0709df\") " pod="openshift-ingress/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts"
Sep 11 14:43:43.867351 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:43.867253    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c69a781-1386-4047-979a-8eeb4e0709df-istio-data\") pod \"e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts\" (UID: \"4c69a781-1386-4047-979a-8eeb4e0709df\") " pod="openshift-ingress/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts"
Sep 11 14:43:43.867351 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:43.867296    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c69a781-1386-4047-979a-8eeb4e0709df-istio-podinfo\") pod \"e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts\" (UID: \"4c69a781-1386-4047-979a-8eeb4e0709df\") " pod="openshift-ingress/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts"
Sep 11 14:43:43.867579 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:43.867360    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c69a781-1386-4047-979a-8eeb4e0709df-istiod-ca-cert\") pod \"e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts\" (UID: \"4c69a781-1386-4047-979a-8eeb4e0709df\") " pod="openshift-ingress/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts"
Sep 11 14:43:43.867579 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:43.867412    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c69a781-1386-4047-979a-8eeb4e0709df-workload-certs\") pod \"e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts\" (UID: \"4c69a781-1386-4047-979a-8eeb4e0709df\") " pod="openshift-ingress/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts"
Sep 11 14:43:43.867579 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:43.867453    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c69a781-1386-4047-979a-8eeb4e0709df-credential-socket\") pod \"e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts\" (UID: \"4c69a781-1386-4047-979a-8eeb4e0709df\") " pod="openshift-ingress/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts"
Sep 11 14:43:43.867743 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:43.867573    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c69a781-1386-4047-979a-8eeb4e0709df-istio-token\") pod \"e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts\" (UID: \"4c69a781-1386-4047-979a-8eeb4e0709df\") " pod="openshift-ingress/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts"
Sep 11 14:43:43.970620 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:43.969167    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c69a781-1386-4047-979a-8eeb4e0709df-workload-certs\") pod \"e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts\" (UID: \"4c69a781-1386-4047-979a-8eeb4e0709df\") " pod="openshift-ingress/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts"
Sep 11 14:43:43.970620 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:43.969233    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c69a781-1386-4047-979a-8eeb4e0709df-credential-socket\") pod \"e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts\" (UID: \"4c69a781-1386-4047-979a-8eeb4e0709df\") " pod="openshift-ingress/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts"
Sep 11 14:43:43.970620 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:43.969291    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c69a781-1386-4047-979a-8eeb4e0709df-istio-token\") pod \"e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts\" (UID: \"4c69a781-1386-4047-979a-8eeb4e0709df\") " pod="openshift-ingress/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts"
Sep 11 14:43:43.970620 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:43.969377    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c69a781-1386-4047-979a-8eeb4e0709df-workload-socket\") pod \"e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts\" (UID: \"4c69a781-1386-4047-979a-8eeb4e0709df\") " pod="openshift-ingress/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts"
Sep 11 14:43:43.970620 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:43.969418    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-696jz\" (UniqueName: \"kubernetes.io/projected/4c69a781-1386-4047-979a-8eeb4e0709df-kube-api-access-696jz\") pod \"e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts\" (UID: \"4c69a781-1386-4047-979a-8eeb4e0709df\") " pod="openshift-ingress/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts"
Sep 11 14:43:43.970620 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:43.969448    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c69a781-1386-4047-979a-8eeb4e0709df-istio-envoy\") pod \"e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts\" (UID: \"4c69a781-1386-4047-979a-8eeb4e0709df\") " pod="openshift-ingress/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts"
Sep 11 14:43:43.970620 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:43.969470    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c69a781-1386-4047-979a-8eeb4e0709df-istio-data\") pod \"e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts\" (UID: \"4c69a781-1386-4047-979a-8eeb4e0709df\") " pod="openshift-ingress/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts"
Sep 11 14:43:43.970620 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:43.969496    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c69a781-1386-4047-979a-8eeb4e0709df-istio-podinfo\") pod \"e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts\" (UID: \"4c69a781-1386-4047-979a-8eeb4e0709df\") " pod="openshift-ingress/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts"
Sep 11 14:43:43.970620 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:43.969547    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c69a781-1386-4047-979a-8eeb4e0709df-istiod-ca-cert\") pod \"e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts\" (UID: \"4c69a781-1386-4047-979a-8eeb4e0709df\") " pod="openshift-ingress/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts"
Sep 11 14:43:43.970620 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:43.970517    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c69a781-1386-4047-979a-8eeb4e0709df-workload-certs\") pod \"e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts\" (UID: \"4c69a781-1386-4047-979a-8eeb4e0709df\") " pod="openshift-ingress/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts"
Sep 11 14:43:43.970620 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:43.970570    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c69a781-1386-4047-979a-8eeb4e0709df-istiod-ca-cert\") pod \"e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts\" (UID: \"4c69a781-1386-4047-979a-8eeb4e0709df\") " pod="openshift-ingress/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts"
Sep 11 14:43:43.971318 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:43.970859    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c69a781-1386-4047-979a-8eeb4e0709df-credential-socket\") pod \"e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts\" (UID: \"4c69a781-1386-4047-979a-8eeb4e0709df\") " pod="openshift-ingress/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts"
Sep 11 14:43:43.971318 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:43.970907    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c69a781-1386-4047-979a-8eeb4e0709df-workload-socket\") pod \"e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts\" (UID: \"4c69a781-1386-4047-979a-8eeb4e0709df\") " pod="openshift-ingress/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts"
Sep 11 14:43:43.971488 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:43.971439    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c69a781-1386-4047-979a-8eeb4e0709df-istio-data\") pod \"e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts\" (UID: \"4c69a781-1386-4047-979a-8eeb4e0709df\") " pod="openshift-ingress/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts"
Sep 11 14:43:43.974120 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:43.974076    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c69a781-1386-4047-979a-8eeb4e0709df-istio-envoy\") pod \"e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts\" (UID: \"4c69a781-1386-4047-979a-8eeb4e0709df\") " pod="openshift-ingress/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts"
Sep 11 14:43:43.974485 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:43.974257    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c69a781-1386-4047-979a-8eeb4e0709df-istio-podinfo\") pod \"e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts\" (UID: \"4c69a781-1386-4047-979a-8eeb4e0709df\") " pod="openshift-ingress/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts"
Sep 11 14:43:43.985039 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:43.985009    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-696jz\" (UniqueName: \"kubernetes.io/projected/4c69a781-1386-4047-979a-8eeb4e0709df-kube-api-access-696jz\") pod \"e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts\" (UID: \"4c69a781-1386-4047-979a-8eeb4e0709df\") " pod="openshift-ingress/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts"
Sep 11 14:43:43.993259 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:43.993232    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c69a781-1386-4047-979a-8eeb4e0709df-istio-token\") pod \"e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts\" (UID: \"4c69a781-1386-4047-979a-8eeb4e0709df\") " pod="openshift-ingress/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts"
Sep 11 14:43:44.032168 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:44.032132    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56b1b6-65zr5"]
Sep 11 14:43:44.036868 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:44.036844    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56b1b6-65zr5"
Sep 11 14:43:44.039519 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:44.039490    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 14:43:44.039661 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:44.039530    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hcm4g\""
Sep 11 14:43:44.039828 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:44.039811    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts"
Sep 11 14:43:44.046417 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:44.046347    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56b1b6-65zr5"]
Sep 11 14:43:44.171797 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:44.171755    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52ccb459-9e22-4ea3-9506-e81c3122f46f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-56b1b6-65zr5\" (UID: \"52ccb459-9e22-4ea3-9506-e81c3122f46f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56b1b6-65zr5"
Sep 11 14:43:44.272382 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:44.272348    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52ccb459-9e22-4ea3-9506-e81c3122f46f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-56b1b6-65zr5\" (UID: \"52ccb459-9e22-4ea3-9506-e81c3122f46f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56b1b6-65zr5"
Sep 11 14:43:44.272963 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:44.272944    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52ccb459-9e22-4ea3-9506-e81c3122f46f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-56b1b6-65zr5\" (UID: \"52ccb459-9e22-4ea3-9506-e81c3122f46f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56b1b6-65zr5"
Sep 11 14:43:44.345576 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:44.345552    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts"]
Sep 11 14:43:44.347144 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:43:44.347116    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c69a781_1386_4047_979a_8eeb4e0709df.slice/crio-e58ceaaace7741ff0a79b302a3c133aeec6121663623211c85652ab77fd92c79 WatchSource:0}: Error finding container e58ceaaace7741ff0a79b302a3c133aeec6121663623211c85652ab77fd92c79: Status 404 returned error can't find the container with id e58ceaaace7741ff0a79b302a3c133aeec6121663623211c85652ab77fd92c79
Sep 11 14:43:44.349982 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:44.349927    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:43:44.350087 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:44.350055    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:43:44.350157 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:44.350108    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:43:44.350462 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:44.350439    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56b1b6-65zr5"
Sep 11 14:43:44.663456 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:44.663431    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56b1b6-65zr5"]
Sep 11 14:43:44.665939 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:43:44.665901    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52ccb459_9e22_4ea3_9506_e81c3122f46f.slice/crio-5fed12c60dcd84a7629eaa04be177538035b16c70a0290f7b601c512235e735f WatchSource:0}: Error finding container 5fed12c60dcd84a7629eaa04be177538035b16c70a0290f7b601c512235e735f: Status 404 returned error can't find the container with id 5fed12c60dcd84a7629eaa04be177538035b16c70a0290f7b601c512235e735f
Sep 11 14:43:44.773635 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:44.773596    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts" event={"ID":"4c69a781-1386-4047-979a-8eeb4e0709df","Type":"ContainerStarted","Data":"f084d93fc3a38b69d3a233a9842f0bc9d305503c1446405c96fa0648cb5436bb"}
Sep 11 14:43:44.773836 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:44.773655    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts" event={"ID":"4c69a781-1386-4047-979a-8eeb4e0709df","Type":"ContainerStarted","Data":"e58ceaaace7741ff0a79b302a3c133aeec6121663623211c85652ab77fd92c79"}
Sep 11 14:43:44.775767 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:44.775739    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8" event={"ID":"8d6949fd-9353-442f-8feb-64d16d9eeb28","Type":"ContainerStarted","Data":"4ae800ac0af29cc938a05358918e6277a9987a93583acc0fa2a8fc887222559c"}
Sep 11 14:43:44.777101 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:44.777077    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56b1b6-65zr5" event={"ID":"52ccb459-9e22-4ea3-9506-e81c3122f46f","Type":"ContainerStarted","Data":"5fed12c60dcd84a7629eaa04be177538035b16c70a0290f7b601c512235e735f"}
Sep 11 14:43:44.819206 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:44.819148    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts" podStartSLOduration=1.8191294770000002 podStartE2EDuration="1.819129477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:43:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:43:44.8144798 +0000 UTC m=+1727.965029904" watchObservedRunningTime="2026-09-11 14:43:44.819129477 +0000 UTC m=+1727.969679541"
Sep 11 14:43:44.881170 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:44.881134    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8"
Sep 11 14:43:44.882343 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:44.882314    2581 patch_prober.go:28] interesting pod/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Sep 11 14:43:44.882469 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:44.882376    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8" podUID="8d6949fd-9353-442f-8feb-64d16d9eeb28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Sep 11 14:43:45.040445 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:45.040413    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts"
Sep 11 14:43:45.045523 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:45.045494    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts"
Sep 11 14:43:45.784491 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:45.784452    2581 generic.go:356] "Generic (PLEG): container finished" podID="52ccb459-9e22-4ea3-9506-e81c3122f46f" containerID="3ff4180a6d5283828230ea08d2fb870463c3f9a34505a576d878bf5467a96c38" exitCode=0
Sep 11 14:43:45.784898 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:45.784546    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56b1b6-65zr5" event={"ID":"52ccb459-9e22-4ea3-9506-e81c3122f46f","Type":"ContainerDied","Data":"3ff4180a6d5283828230ea08d2fb870463c3f9a34505a576d878bf5467a96c38"}
Sep 11 14:43:45.784898 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:45.784809    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts"
Sep 11 14:43:45.785862 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:45.785846    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts"
Sep 11 14:43:45.881495 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:45.881460    2581 patch_prober.go:28] interesting pod/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Sep 11 14:43:45.881642 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:45.881519    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8" podUID="8d6949fd-9353-442f-8feb-64d16d9eeb28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Sep 11 14:43:46.880683 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:46.880646    2581 patch_prober.go:28] interesting pod/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Sep 11 14:43:46.881096 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:46.880716    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8" podUID="8d6949fd-9353-442f-8feb-64d16d9eeb28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Sep 11 14:43:46.966274 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:46.966248    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56b1b6-65zr5"
Sep 11 14:43:47.118052 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:47.117927    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52ccb459-9e22-4ea3-9506-e81c3122f46f-maas-api-service-ca\") pod \"52ccb459-9e22-4ea3-9506-e81c3122f46f\" (UID: \"52ccb459-9e22-4ea3-9506-e81c3122f46f\") "
Sep 11 14:43:47.118302 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:47.118280    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52ccb459-9e22-4ea3-9506-e81c3122f46f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "52ccb459-9e22-4ea3-9506-e81c3122f46f" (UID: "52ccb459-9e22-4ea3-9506-e81c3122f46f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:43:47.219060 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:47.219007    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52ccb459-9e22-4ea3-9506-e81c3122f46f-maas-api-service-ca\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:43:47.615136 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:47.615087    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8"
Sep 11 14:43:47.801863 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:47.801763    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56b1b6-65zr5"
Sep 11 14:43:47.802579 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:47.802545    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56b1b6-65zr5" event={"ID":"52ccb459-9e22-4ea3-9506-e81c3122f46f","Type":"ContainerDied","Data":"5fed12c60dcd84a7629eaa04be177538035b16c70a0290f7b601c512235e735f"}
Sep 11 14:43:47.802759 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:47.802743    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fed12c60dcd84a7629eaa04be177538035b16c70a0290f7b601c512235e735f"
Sep 11 14:43:47.881589 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:47.881552    2581 patch_prober.go:28] interesting pod/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Sep 11 14:43:47.882030 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:47.881617    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8" podUID="8d6949fd-9353-442f-8feb-64d16d9eeb28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Sep 11 14:43:48.818453 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:48.818421    2581 generic.go:356] "Generic (PLEG): container finished" podID="1682346d-e312-4750-90f8-28db3c360404" containerID="b47637dc74e29506cd291d587e61a78266cec677b9231cc1b4b3580e9ec078e7" exitCode=0
Sep 11 14:43:48.818654 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:48.818478    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb" event={"ID":"1682346d-e312-4750-90f8-28db3c360404","Type":"ContainerDied","Data":"b47637dc74e29506cd291d587e61a78266cec677b9231cc1b4b3580e9ec078e7"}
Sep 11 14:43:48.818654 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:48.818532    2581 scope.go:117] "RemoveContainer" containerID="fb4ba3c19bc83157d8fbd75930c61ce3179e27697e09b1d9f6f990a043cac27b"
Sep 11 14:43:48.881219 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:48.881183    2581 patch_prober.go:28] interesting pod/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Sep 11 14:43:48.881416 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:48.881249    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8" podUID="8d6949fd-9353-442f-8feb-64d16d9eeb28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Sep 11 14:43:48.938459 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:48.938433    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb"
Sep 11 14:43:49.039886 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:49.039857    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1682346d-e312-4750-90f8-28db3c360404-workload-socket\") pod \"1682346d-e312-4750-90f8-28db3c360404\" (UID: \"1682346d-e312-4750-90f8-28db3c360404\") "
Sep 11 14:43:49.040105 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:49.039920    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1682346d-e312-4750-90f8-28db3c360404-istio-token\") pod \"1682346d-e312-4750-90f8-28db3c360404\" (UID: \"1682346d-e312-4750-90f8-28db3c360404\") "
Sep 11 14:43:49.040105 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:49.040052    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1682346d-e312-4750-90f8-28db3c360404-istiod-ca-cert\") pod \"1682346d-e312-4750-90f8-28db3c360404\" (UID: \"1682346d-e312-4750-90f8-28db3c360404\") "
Sep 11 14:43:49.040105 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:49.040074    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1682346d-e312-4750-90f8-28db3c360404-istio-podinfo\") pod \"1682346d-e312-4750-90f8-28db3c360404\" (UID: \"1682346d-e312-4750-90f8-28db3c360404\") "
Sep 11 14:43:49.040105 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:49.040093    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1682346d-e312-4750-90f8-28db3c360404-istio-envoy\") pod \"1682346d-e312-4750-90f8-28db3c360404\" (UID: \"1682346d-e312-4750-90f8-28db3c360404\") "
Sep 11 14:43:49.040336 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:49.040140    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1682346d-e312-4750-90f8-28db3c360404-workload-certs\") pod \"1682346d-e312-4750-90f8-28db3c360404\" (UID: \"1682346d-e312-4750-90f8-28db3c360404\") "
Sep 11 14:43:49.040336 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:49.040170    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1682346d-e312-4750-90f8-28db3c360404-istio-data\") pod \"1682346d-e312-4750-90f8-28db3c360404\" (UID: \"1682346d-e312-4750-90f8-28db3c360404\") "
Sep 11 14:43:49.040336 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:49.040198    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xzd2w\" (UniqueName: \"kubernetes.io/projected/1682346d-e312-4750-90f8-28db3c360404-kube-api-access-xzd2w\") pod \"1682346d-e312-4750-90f8-28db3c360404\" (UID: \"1682346d-e312-4750-90f8-28db3c360404\") "
Sep 11 14:43:49.040336 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:49.040226    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1682346d-e312-4750-90f8-28db3c360404-credential-socket\") pod \"1682346d-e312-4750-90f8-28db3c360404\" (UID: \"1682346d-e312-4750-90f8-28db3c360404\") "
Sep 11 14:43:49.040336 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:49.040132    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1682346d-e312-4750-90f8-28db3c360404-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1682346d-e312-4750-90f8-28db3c360404" (UID: "1682346d-e312-4750-90f8-28db3c360404"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:43:49.040607 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:49.040318    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1682346d-e312-4750-90f8-28db3c360404-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1682346d-e312-4750-90f8-28db3c360404" (UID: "1682346d-e312-4750-90f8-28db3c360404"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:43:49.040607 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:49.040427    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1682346d-e312-4750-90f8-28db3c360404-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1682346d-e312-4750-90f8-28db3c360404" (UID: "1682346d-e312-4750-90f8-28db3c360404"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:43:49.040607 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:49.040506    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1682346d-e312-4750-90f8-28db3c360404-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1682346d-e312-4750-90f8-28db3c360404" (UID: "1682346d-e312-4750-90f8-28db3c360404"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:43:49.040607 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:49.040600    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1682346d-e312-4750-90f8-28db3c360404-istiod-ca-cert\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:43:49.040767 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:49.040613    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1682346d-e312-4750-90f8-28db3c360404-workload-certs\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:43:49.040767 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:49.040621    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1682346d-e312-4750-90f8-28db3c360404-credential-socket\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:43:49.040767 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:49.040618    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1682346d-e312-4750-90f8-28db3c360404-istio-data" (OuterVolumeSpecName: "istio-data") pod "1682346d-e312-4750-90f8-28db3c360404" (UID: "1682346d-e312-4750-90f8-28db3c360404"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:43:49.040767 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:49.040635    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1682346d-e312-4750-90f8-28db3c360404-workload-socket\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:43:49.042272 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:49.042246    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1682346d-e312-4750-90f8-28db3c360404-istio-token" (OuterVolumeSpecName: "istio-token") pod "1682346d-e312-4750-90f8-28db3c360404" (UID: "1682346d-e312-4750-90f8-28db3c360404"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:43:49.042477 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:49.042457    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1682346d-e312-4750-90f8-28db3c360404-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1682346d-e312-4750-90f8-28db3c360404" (UID: "1682346d-e312-4750-90f8-28db3c360404"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:43:49.042477 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:49.042468    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1682346d-e312-4750-90f8-28db3c360404-kube-api-access-xzd2w" (OuterVolumeSpecName: "kube-api-access-xzd2w") pod "1682346d-e312-4750-90f8-28db3c360404" (UID: "1682346d-e312-4750-90f8-28db3c360404"). InnerVolumeSpecName "kube-api-access-xzd2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:43:49.042567 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:49.042475    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1682346d-e312-4750-90f8-28db3c360404-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1682346d-e312-4750-90f8-28db3c360404" (UID: "1682346d-e312-4750-90f8-28db3c360404"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 14:43:49.141253 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:49.141215    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1682346d-e312-4750-90f8-28db3c360404-istio-podinfo\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:43:49.141253 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:49.141249    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1682346d-e312-4750-90f8-28db3c360404-istio-envoy\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:43:49.141253 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:49.141258    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1682346d-e312-4750-90f8-28db3c360404-istio-data\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:43:49.141498 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:49.141266    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzd2w\" (UniqueName: \"kubernetes.io/projected/1682346d-e312-4750-90f8-28db3c360404-kube-api-access-xzd2w\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:43:49.141498 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:49.141274    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1682346d-e312-4750-90f8-28db3c360404-istio-token\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:43:49.825351 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:49.825264    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb"
Sep 11 14:43:49.825351 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:49.825310    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb" event={"ID":"1682346d-e312-4750-90f8-28db3c360404","Type":"ContainerDied","Data":"ef6931dbc1ba1809d722604615ed8f2ca95edc5fd7c048e43675a146f3904782"}
Sep 11 14:43:49.825600 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:49.825371    2581 scope.go:117] "RemoveContainer" containerID="b47637dc74e29506cd291d587e61a78266cec677b9231cc1b4b3580e9ec078e7"
Sep 11 14:43:49.857431 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:49.857397    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb"]
Sep 11 14:43:49.872173 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:49.872146    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-ef0ff5-openshift-default-788db8fb6d-8mwnb"]
Sep 11 14:43:49.881345 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:49.881318    2581 patch_prober.go:28] interesting pod/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Sep 11 14:43:49.881450 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:49.881367    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8" podUID="8d6949fd-9353-442f-8feb-64d16d9eeb28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Sep 11 14:43:50.885603 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:50.885572    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8"
Sep 11 14:43:50.886652 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:50.886631    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8"
Sep 11 14:43:51.499325 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:51.499288    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1682346d-e312-4750-90f8-28db3c360404" path="/var/lib/kubelet/pods/1682346d-e312-4750-90f8-28db3c360404/volumes"
Sep 11 14:43:53.599432 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:53.599393    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-40d3a2-bbbb765db-mp5mp"]
Sep 11 14:43:53.599848 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:53.599768    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1682346d-e312-4750-90f8-28db3c360404" containerName="istio-proxy"
Sep 11 14:43:53.599848 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:53.599779    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1682346d-e312-4750-90f8-28db3c360404" containerName="istio-proxy"
Sep 11 14:43:53.599848 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:53.599790    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52ccb459-9e22-4ea3-9506-e81c3122f46f" containerName="revoke-keys"
Sep 11 14:43:53.599848 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:53.599796    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="52ccb459-9e22-4ea3-9506-e81c3122f46f" containerName="revoke-keys"
Sep 11 14:43:53.600025 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:53.599866    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1682346d-e312-4750-90f8-28db3c360404" containerName="istio-proxy"
Sep 11 14:43:53.600025 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:53.599876    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="52ccb459-9e22-4ea3-9506-e81c3122f46f" containerName="revoke-keys"
Sep 11 14:43:53.600025 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:53.599883    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1682346d-e312-4750-90f8-28db3c360404" containerName="istio-proxy"
Sep 11 14:43:53.602831 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:53.602786    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-40d3a2-bbbb765db-mp5mp"
Sep 11 14:43:53.605470 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:53.605441    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-40d3a2\""
Sep 11 14:43:53.605621 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:53.605599    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-40d3a2-bdfd6f867-2jd4r"]
Sep 11 14:43:53.605956 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:53.605943    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1682346d-e312-4750-90f8-28db3c360404" containerName="istio-proxy"
Sep 11 14:43:53.606022 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:53.605957    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1682346d-e312-4750-90f8-28db3c360404" containerName="istio-proxy"
Sep 11 14:43:53.608954 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:53.608937    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-40d3a2-bdfd6f867-2jd4r"
Sep 11 14:43:53.611475 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:53.611457    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-40d3a2-dockercfg-jmfjj\""
Sep 11 14:43:53.615529 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:53.615508    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-40d3a2-bbbb765db-mp5mp"]
Sep 11 14:43:53.622310 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:53.622282    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-40d3a2-bdfd6f867-2jd4r"]
Sep 11 14:43:53.683433 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:53.683395    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwrks\" (UniqueName: \"kubernetes.io/projected/966ef1cb-e285-4d48-a36b-a2edcf3591ad-kube-api-access-mwrks\") pod \"payload-processing-e2e-ipp-b-40d3a2-bdfd6f867-2jd4r\" (UID: \"966ef1cb-e285-4d48-a36b-a2edcf3591ad\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-40d3a2-bdfd6f867-2jd4r"
Sep 11 14:43:53.683625 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:53.683447    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89e8c830-ad17-499b-9b8f-e7f8256e4402-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-40d3a2-bbbb765db-mp5mp\" (UID: \"89e8c830-ad17-499b-9b8f-e7f8256e4402\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-40d3a2-bbbb765db-mp5mp"
Sep 11 14:43:53.683625 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:53.683521    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/966ef1cb-e285-4d48-a36b-a2edcf3591ad-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-40d3a2-bdfd6f867-2jd4r\" (UID: \"966ef1cb-e285-4d48-a36b-a2edcf3591ad\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-40d3a2-bdfd6f867-2jd4r"
Sep 11 14:43:53.683625 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:53.683584    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpn2x\" (UniqueName: \"kubernetes.io/projected/89e8c830-ad17-499b-9b8f-e7f8256e4402-kube-api-access-dpn2x\") pod \"payload-pre-processing-e2e-ipp-b-40d3a2-bbbb765db-mp5mp\" (UID: \"89e8c830-ad17-499b-9b8f-e7f8256e4402\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-40d3a2-bbbb765db-mp5mp"
Sep 11 14:43:53.784357 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:53.784315    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwrks\" (UniqueName: \"kubernetes.io/projected/966ef1cb-e285-4d48-a36b-a2edcf3591ad-kube-api-access-mwrks\") pod \"payload-processing-e2e-ipp-b-40d3a2-bdfd6f867-2jd4r\" (UID: \"966ef1cb-e285-4d48-a36b-a2edcf3591ad\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-40d3a2-bdfd6f867-2jd4r"
Sep 11 14:43:53.784528 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:53.784369    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89e8c830-ad17-499b-9b8f-e7f8256e4402-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-40d3a2-bbbb765db-mp5mp\" (UID: \"89e8c830-ad17-499b-9b8f-e7f8256e4402\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-40d3a2-bbbb765db-mp5mp"
Sep 11 14:43:53.784528 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:53.784395    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/966ef1cb-e285-4d48-a36b-a2edcf3591ad-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-40d3a2-bdfd6f867-2jd4r\" (UID: \"966ef1cb-e285-4d48-a36b-a2edcf3591ad\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-40d3a2-bdfd6f867-2jd4r"
Sep 11 14:43:53.784528 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:53.784436    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpn2x\" (UniqueName: \"kubernetes.io/projected/89e8c830-ad17-499b-9b8f-e7f8256e4402-kube-api-access-dpn2x\") pod \"payload-pre-processing-e2e-ipp-b-40d3a2-bbbb765db-mp5mp\" (UID: \"89e8c830-ad17-499b-9b8f-e7f8256e4402\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-40d3a2-bbbb765db-mp5mp"
Sep 11 14:43:53.785239 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:53.785215    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/966ef1cb-e285-4d48-a36b-a2edcf3591ad-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-40d3a2-bdfd6f867-2jd4r\" (UID: \"966ef1cb-e285-4d48-a36b-a2edcf3591ad\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-40d3a2-bdfd6f867-2jd4r"
Sep 11 14:43:53.785341 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:53.785215    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89e8c830-ad17-499b-9b8f-e7f8256e4402-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-40d3a2-bbbb765db-mp5mp\" (UID: \"89e8c830-ad17-499b-9b8f-e7f8256e4402\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-40d3a2-bbbb765db-mp5mp"
Sep 11 14:43:53.794136 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:53.794114    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpn2x\" (UniqueName: \"kubernetes.io/projected/89e8c830-ad17-499b-9b8f-e7f8256e4402-kube-api-access-dpn2x\") pod \"payload-pre-processing-e2e-ipp-b-40d3a2-bbbb765db-mp5mp\" (UID: \"89e8c830-ad17-499b-9b8f-e7f8256e4402\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-40d3a2-bbbb765db-mp5mp"
Sep 11 14:43:53.794243 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:53.794115    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwrks\" (UniqueName: \"kubernetes.io/projected/966ef1cb-e285-4d48-a36b-a2edcf3591ad-kube-api-access-mwrks\") pod \"payload-processing-e2e-ipp-b-40d3a2-bdfd6f867-2jd4r\" (UID: \"966ef1cb-e285-4d48-a36b-a2edcf3591ad\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-40d3a2-bdfd6f867-2jd4r"
Sep 11 14:43:53.913537 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:53.913450    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-40d3a2-bbbb765db-mp5mp"
Sep 11 14:43:53.926238 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:53.925751    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-40d3a2-bdfd6f867-2jd4r"
Sep 11 14:43:54.197270 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:54.197244    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-40d3a2-bbbb765db-mp5mp"]
Sep 11 14:43:54.199108 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:43:54.199081    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89e8c830_ad17_499b_9b8f_e7f8256e4402.slice/crio-0e179427428a0b7f413e37d2cf65b3fd24c03f23a88ccf3d55bd46cd0b816748 WatchSource:0}: Error finding container 0e179427428a0b7f413e37d2cf65b3fd24c03f23a88ccf3d55bd46cd0b816748: Status 404 returned error can't find the container with id 0e179427428a0b7f413e37d2cf65b3fd24c03f23a88ccf3d55bd46cd0b816748
Sep 11 14:43:54.217480 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:54.217453    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-40d3a2-bdfd6f867-2jd4r"]
Sep 11 14:43:54.218808 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:43:54.218786    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod966ef1cb_e285_4d48_a36b_a2edcf3591ad.slice/crio-57c4b6ef5a2da6fcb69d8e236dc2b8ed728c9868d2912083217c0d7e80d34cb7 WatchSource:0}: Error finding container 57c4b6ef5a2da6fcb69d8e236dc2b8ed728c9868d2912083217c0d7e80d34cb7: Status 404 returned error can't find the container with id 57c4b6ef5a2da6fcb69d8e236dc2b8ed728c9868d2912083217c0d7e80d34cb7
Sep 11 14:43:54.853678 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:54.853638    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-40d3a2-bdfd6f867-2jd4r" event={"ID":"966ef1cb-e285-4d48-a36b-a2edcf3591ad","Type":"ContainerStarted","Data":"2a930c72f2051cefb6382dd5616a04f6ea3714d8ca83465b754084dadf3fd054"}
Sep 11 14:43:54.854156 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:54.853692    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-40d3a2-bdfd6f867-2jd4r" event={"ID":"966ef1cb-e285-4d48-a36b-a2edcf3591ad","Type":"ContainerStarted","Data":"57c4b6ef5a2da6fcb69d8e236dc2b8ed728c9868d2912083217c0d7e80d34cb7"}
Sep 11 14:43:54.854156 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:54.853728    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-40d3a2-bdfd6f867-2jd4r"
Sep 11 14:43:54.855318 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:54.855297    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-40d3a2-bbbb765db-mp5mp" event={"ID":"89e8c830-ad17-499b-9b8f-e7f8256e4402","Type":"ContainerStarted","Data":"22b65281056eec7dab7b48ec9d7368e2873e0b2bc1940222454fcdd1573989d0"}
Sep 11 14:43:54.855420 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:54.855332    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-40d3a2-bbbb765db-mp5mp" event={"ID":"89e8c830-ad17-499b-9b8f-e7f8256e4402","Type":"ContainerStarted","Data":"0e179427428a0b7f413e37d2cf65b3fd24c03f23a88ccf3d55bd46cd0b816748"}
Sep 11 14:43:54.855420 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:54.855360    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-40d3a2-bbbb765db-mp5mp"
Sep 11 14:43:54.875802 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:54.875753    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-40d3a2-bdfd6f867-2jd4r" podStartSLOduration=1.875739628 podStartE2EDuration="1.875739628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:43:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:43:54.874857782 +0000 UTC m=+1738.025407859" watchObservedRunningTime="2026-09-11 14:43:54.875739628 +0000 UTC m=+1738.026289712"
Sep 11 14:43:54.897192 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:43:54.897148    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-40d3a2-bbbb765db-mp5mp" podStartSLOduration=1.8971340479999999 podStartE2EDuration="1.897134048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:43:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:43:54.896622984 +0000 UTC m=+1738.047173052" watchObservedRunningTime="2026-09-11 14:43:54.897134048 +0000 UTC m=+1738.047684112"
Sep 11 14:44:02.280961 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:02.280922    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56b1b6-65zr5"]
Sep 11 14:44:02.283316 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:02.283291    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56b1b6-65zr5"]
Sep 11 14:44:03.503546 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:03.503496    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52ccb459-9e22-4ea3-9506-e81c3122f46f" path="/var/lib/kubelet/pods/52ccb459-9e22-4ea3-9506-e81c3122f46f/volumes"
Sep 11 14:44:05.862963 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:05.862929    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-40d3a2-bdfd6f867-2jd4r"
Sep 11 14:44:05.863522 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:05.863053    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-40d3a2-bbbb765db-mp5mp"
Sep 11 14:44:07.941401 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:07.941367    2581 generic.go:356] "Generic (PLEG): container finished" podID="4c69a781-1386-4047-979a-8eeb4e0709df" containerID="f084d93fc3a38b69d3a233a9842f0bc9d305503c1446405c96fa0648cb5436bb" exitCode=0
Sep 11 14:44:07.941848 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:07.941452    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts" event={"ID":"4c69a781-1386-4047-979a-8eeb4e0709df","Type":"ContainerDied","Data":"f084d93fc3a38b69d3a233a9842f0bc9d305503c1446405c96fa0648cb5436bb"}
Sep 11 14:44:07.941911 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:07.941877    2581 scope.go:117] "RemoveContainer" containerID="f084d93fc3a38b69d3a233a9842f0bc9d305503c1446405c96fa0648cb5436bb"
Sep 11 14:44:07.943014 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:07.942984    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:44:07.943111 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:07.943054    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:44:07.943111 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:07.943084    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 14:44:08.948900 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:08.948867    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts" event={"ID":"4c69a781-1386-4047-979a-8eeb4e0709df","Type":"ContainerStarted","Data":"f7540eea6a548d3df88c49282c79e9ca6450f2e5303a468285b4941a49815b62"}
Sep 11 14:44:09.040755 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:09.040699    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts"
Sep 11 14:44:09.041073 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:09.041045    2581 patch_prober.go:28] interesting pod/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Sep 11 14:44:09.041195 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:09.041118    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts" podUID="4c69a781-1386-4047-979a-8eeb4e0709df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Sep 11 14:44:10.040933 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:10.040895    2581 patch_prober.go:28] interesting pod/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Sep 11 14:44:10.041475 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:10.040951    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts" podUID="4c69a781-1386-4047-979a-8eeb4e0709df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Sep 11 14:44:11.040908 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:11.040868    2581 patch_prober.go:28] interesting pod/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Sep 11 14:44:11.041123 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:11.040926    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts" podUID="4c69a781-1386-4047-979a-8eeb4e0709df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Sep 11 14:44:11.667713 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:11.667675    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm"]
Sep 11 14:44:11.667957 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:11.667930    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm" podUID="7f05e099-7e7a-4c36-aeb8-6072904fdb5d" containerName="istio-proxy" containerID="cri-o://6eea61ef49275fcfcf291c1fb57e1e37f5bfa612c5ecc9e8af10b1c6aeb0a4da" gracePeriod=30
Sep 11 14:44:12.040728 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:12.040692    2581 patch_prober.go:28] interesting pod/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Sep 11 14:44:12.040924 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:12.040757    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts" podUID="4c69a781-1386-4047-979a-8eeb4e0709df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Sep 11 14:44:12.072553 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:12.072523    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-958398-p8mcj"]
Sep 11 14:44:12.074921 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:12.074903    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-958398-p8mcj"
Sep 11 14:44:12.077627 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:12.077604    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 14:44:12.077627 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:12.077608    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hcm4g\""
Sep 11 14:44:12.097911 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:12.097880    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-958398-p8mcj"]
Sep 11 14:44:12.172335 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:12.172291    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dda558e3-ff64-4c75-bcd4-0eedc88aa2d0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-958398-p8mcj\" (UID: \"dda558e3-ff64-4c75-bcd4-0eedc88aa2d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-958398-p8mcj"
Sep 11 14:44:12.273736 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:12.273695    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dda558e3-ff64-4c75-bcd4-0eedc88aa2d0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-958398-p8mcj\" (UID: \"dda558e3-ff64-4c75-bcd4-0eedc88aa2d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-958398-p8mcj"
Sep 11 14:44:12.274392 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:12.274372    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dda558e3-ff64-4c75-bcd4-0eedc88aa2d0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-958398-p8mcj\" (UID: \"dda558e3-ff64-4c75-bcd4-0eedc88aa2d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-958398-p8mcj"
Sep 11 14:44:12.385564 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:12.385463    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-958398-p8mcj"
Sep 11 14:44:12.687270 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:12.687179    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-958398-p8mcj"]
Sep 11 14:44:12.971240 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:12.971153    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-958398-p8mcj" event={"ID":"dda558e3-ff64-4c75-bcd4-0eedc88aa2d0","Type":"ContainerStarted","Data":"06ef7f3abf7a92893263d6dee3d30f0103358e0741840609c8924b90d90fe842"}
Sep 11 14:44:13.040666 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:13.040627    2581 patch_prober.go:28] interesting pod/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Sep 11 14:44:13.040839 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:13.040687    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts" podUID="4c69a781-1386-4047-979a-8eeb4e0709df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Sep 11 14:44:13.937629 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:13.937593    2581 patch_prober.go:28] interesting pod/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 14:44:13.938130 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:13.937651    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm" podUID="7f05e099-7e7a-4c36-aeb8-6072904fdb5d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 14:44:13.978668 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:13.978580    2581 generic.go:356] "Generic (PLEG): container finished" podID="dda558e3-ff64-4c75-bcd4-0eedc88aa2d0" containerID="f2cd78a44dfb8c4b28ea7e6d8481fa2b8b94561fb301db80dbd28249fc344660" exitCode=0
Sep 11 14:44:13.978668 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:13.978627    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-958398-p8mcj" event={"ID":"dda558e3-ff64-4c75-bcd4-0eedc88aa2d0","Type":"ContainerDied","Data":"f2cd78a44dfb8c4b28ea7e6d8481fa2b8b94561fb301db80dbd28249fc344660"}
Sep 11 14:44:14.226005 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:14.225949    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts"
Sep 11 14:44:14.226477 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:14.226449    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts"
Sep 11 14:44:14.228350 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:14.228330    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts"
Sep 11 14:44:15.112310 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:15.112287    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-958398-p8mcj"
Sep 11 14:44:15.206847 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:15.206810    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dda558e3-ff64-4c75-bcd4-0eedc88aa2d0-maas-api-service-ca\") pod \"dda558e3-ff64-4c75-bcd4-0eedc88aa2d0\" (UID: \"dda558e3-ff64-4c75-bcd4-0eedc88aa2d0\") "
Sep 11 14:44:15.207224 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:15.207203    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dda558e3-ff64-4c75-bcd4-0eedc88aa2d0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dda558e3-ff64-4c75-bcd4-0eedc88aa2d0" (UID: "dda558e3-ff64-4c75-bcd4-0eedc88aa2d0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:44:15.307593 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:15.307560    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dda558e3-ff64-4c75-bcd4-0eedc88aa2d0-maas-api-service-ca\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:44:15.990060 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:15.990026    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-958398-p8mcj" event={"ID":"dda558e3-ff64-4c75-bcd4-0eedc88aa2d0","Type":"ContainerDied","Data":"06ef7f3abf7a92893263d6dee3d30f0103358e0741840609c8924b90d90fe842"}
Sep 11 14:44:15.990060 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:15.990076    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06ef7f3abf7a92893263d6dee3d30f0103358e0741840609c8924b90d90fe842"
Sep 11 14:44:15.990287 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:15.990076    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-958398-p8mcj"
Sep 11 14:44:16.926149 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:16.926126    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm"
Sep 11 14:44:16.997045 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:16.997016    2581 generic.go:356] "Generic (PLEG): container finished" podID="7f05e099-7e7a-4c36-aeb8-6072904fdb5d" containerID="6eea61ef49275fcfcf291c1fb57e1e37f5bfa612c5ecc9e8af10b1c6aeb0a4da" exitCode=0
Sep 11 14:44:16.997201 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:16.997065    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm"
Sep 11 14:44:16.997201 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:16.997090    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm" event={"ID":"7f05e099-7e7a-4c36-aeb8-6072904fdb5d","Type":"ContainerDied","Data":"6eea61ef49275fcfcf291c1fb57e1e37f5bfa612c5ecc9e8af10b1c6aeb0a4da"}
Sep 11 14:44:16.997201 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:16.997133    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm" event={"ID":"7f05e099-7e7a-4c36-aeb8-6072904fdb5d","Type":"ContainerDied","Data":"fc0a5e0222e76de58154ce71cbd1ca2c7b0dbfc3f9dbde4ebcb599be4cf4e89c"}
Sep 11 14:44:16.997201 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:16.997161    2581 scope.go:117] "RemoveContainer" containerID="6eea61ef49275fcfcf291c1fb57e1e37f5bfa612c5ecc9e8af10b1c6aeb0a4da"
Sep 11 14:44:17.006274 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:17.006252    2581 scope.go:117] "RemoveContainer" containerID="5b68b97c7952729108589d41f1f0c7d99df0bd48ae9c2e1691066d995d3b4f57"
Sep 11 14:44:17.014344 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:17.014327    2581 scope.go:117] "RemoveContainer" containerID="6eea61ef49275fcfcf291c1fb57e1e37f5bfa612c5ecc9e8af10b1c6aeb0a4da"
Sep 11 14:44:17.014615 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:44:17.014596    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6eea61ef49275fcfcf291c1fb57e1e37f5bfa612c5ecc9e8af10b1c6aeb0a4da\": container with ID starting with 6eea61ef49275fcfcf291c1fb57e1e37f5bfa612c5ecc9e8af10b1c6aeb0a4da not found: ID does not exist" containerID="6eea61ef49275fcfcf291c1fb57e1e37f5bfa612c5ecc9e8af10b1c6aeb0a4da"
Sep 11 14:44:17.014675 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:17.014624    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eea61ef49275fcfcf291c1fb57e1e37f5bfa612c5ecc9e8af10b1c6aeb0a4da"} err="failed to get container status \"6eea61ef49275fcfcf291c1fb57e1e37f5bfa612c5ecc9e8af10b1c6aeb0a4da\": rpc error: code = NotFound desc = could not find container \"6eea61ef49275fcfcf291c1fb57e1e37f5bfa612c5ecc9e8af10b1c6aeb0a4da\": container with ID starting with 6eea61ef49275fcfcf291c1fb57e1e37f5bfa612c5ecc9e8af10b1c6aeb0a4da not found: ID does not exist"
Sep 11 14:44:17.014675 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:17.014642    2581 scope.go:117] "RemoveContainer" containerID="5b68b97c7952729108589d41f1f0c7d99df0bd48ae9c2e1691066d995d3b4f57"
Sep 11 14:44:17.014873 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:44:17.014856    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b68b97c7952729108589d41f1f0c7d99df0bd48ae9c2e1691066d995d3b4f57\": container with ID starting with 5b68b97c7952729108589d41f1f0c7d99df0bd48ae9c2e1691066d995d3b4f57 not found: ID does not exist" containerID="5b68b97c7952729108589d41f1f0c7d99df0bd48ae9c2e1691066d995d3b4f57"
Sep 11 14:44:17.014914 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:17.014879    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b68b97c7952729108589d41f1f0c7d99df0bd48ae9c2e1691066d995d3b4f57"} err="failed to get container status \"5b68b97c7952729108589d41f1f0c7d99df0bd48ae9c2e1691066d995d3b4f57\": rpc error: code = NotFound desc = could not find container \"5b68b97c7952729108589d41f1f0c7d99df0bd48ae9c2e1691066d995d3b4f57\": container with ID starting with 5b68b97c7952729108589d41f1f0c7d99df0bd48ae9c2e1691066d995d3b4f57 not found: ID does not exist"
Sep 11 14:44:17.025260 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:17.025238    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f05e099-7e7a-4c36-aeb8-6072904fdb5d-istio-podinfo\") pod \"7f05e099-7e7a-4c36-aeb8-6072904fdb5d\" (UID: \"7f05e099-7e7a-4c36-aeb8-6072904fdb5d\") "
Sep 11 14:44:17.025344 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:17.025273    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f05e099-7e7a-4c36-aeb8-6072904fdb5d-istio-token\") pod \"7f05e099-7e7a-4c36-aeb8-6072904fdb5d\" (UID: \"7f05e099-7e7a-4c36-aeb8-6072904fdb5d\") "
Sep 11 14:44:17.025344 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:17.025311    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f05e099-7e7a-4c36-aeb8-6072904fdb5d-istio-envoy\") pod \"7f05e099-7e7a-4c36-aeb8-6072904fdb5d\" (UID: \"7f05e099-7e7a-4c36-aeb8-6072904fdb5d\") "
Sep 11 14:44:17.025438 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:17.025362    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f05e099-7e7a-4c36-aeb8-6072904fdb5d-credential-socket\") pod \"7f05e099-7e7a-4c36-aeb8-6072904fdb5d\" (UID: \"7f05e099-7e7a-4c36-aeb8-6072904fdb5d\") "
Sep 11 14:44:17.025438 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:17.025405    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f05e099-7e7a-4c36-aeb8-6072904fdb5d-workload-certs\") pod \"7f05e099-7e7a-4c36-aeb8-6072904fdb5d\" (UID: \"7f05e099-7e7a-4c36-aeb8-6072904fdb5d\") "
Sep 11 14:44:17.025538 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:17.025437    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f05e099-7e7a-4c36-aeb8-6072904fdb5d-istio-data\") pod \"7f05e099-7e7a-4c36-aeb8-6072904fdb5d\" (UID: \"7f05e099-7e7a-4c36-aeb8-6072904fdb5d\") "
Sep 11 14:44:17.025538 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:17.025484    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f05e099-7e7a-4c36-aeb8-6072904fdb5d-istiod-ca-cert\") pod \"7f05e099-7e7a-4c36-aeb8-6072904fdb5d\" (UID: \"7f05e099-7e7a-4c36-aeb8-6072904fdb5d\") "
Sep 11 14:44:17.025538 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:17.025512    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f05e099-7e7a-4c36-aeb8-6072904fdb5d-workload-socket\") pod \"7f05e099-7e7a-4c36-aeb8-6072904fdb5d\" (UID: \"7f05e099-7e7a-4c36-aeb8-6072904fdb5d\") "
Sep 11 14:44:17.026135 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:17.026088    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f05e099-7e7a-4c36-aeb8-6072904fdb5d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7f05e099-7e7a-4c36-aeb8-6072904fdb5d" (UID: "7f05e099-7e7a-4c36-aeb8-6072904fdb5d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:44:17.026264 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:17.026226    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f05e099-7e7a-4c36-aeb8-6072904fdb5d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7f05e099-7e7a-4c36-aeb8-6072904fdb5d" (UID: "7f05e099-7e7a-4c36-aeb8-6072904fdb5d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:44:17.026397 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:17.026376    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gpjt\" (UniqueName: \"kubernetes.io/projected/7f05e099-7e7a-4c36-aeb8-6072904fdb5d-kube-api-access-7gpjt\") pod \"7f05e099-7e7a-4c36-aeb8-6072904fdb5d\" (UID: \"7f05e099-7e7a-4c36-aeb8-6072904fdb5d\") "
Sep 11 14:44:17.026488 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:17.026393    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f05e099-7e7a-4c36-aeb8-6072904fdb5d-istio-data" (OuterVolumeSpecName: "istio-data") pod "7f05e099-7e7a-4c36-aeb8-6072904fdb5d" (UID: "7f05e099-7e7a-4c36-aeb8-6072904fdb5d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:44:17.026547 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:17.026514    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f05e099-7e7a-4c36-aeb8-6072904fdb5d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7f05e099-7e7a-4c36-aeb8-6072904fdb5d" (UID: "7f05e099-7e7a-4c36-aeb8-6072904fdb5d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:44:17.034124 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:17.026822    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f05e099-7e7a-4c36-aeb8-6072904fdb5d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7f05e099-7e7a-4c36-aeb8-6072904fdb5d" (UID: "7f05e099-7e7a-4c36-aeb8-6072904fdb5d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:44:17.034124 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:17.028495    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f05e099-7e7a-4c36-aeb8-6072904fdb5d-istio-token" (OuterVolumeSpecName: "istio-token") pod "7f05e099-7e7a-4c36-aeb8-6072904fdb5d" (UID: "7f05e099-7e7a-4c36-aeb8-6072904fdb5d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:44:17.034124 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:17.028570    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f05e099-7e7a-4c36-aeb8-6072904fdb5d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7f05e099-7e7a-4c36-aeb8-6072904fdb5d" (UID: "7f05e099-7e7a-4c36-aeb8-6072904fdb5d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 14:44:17.034124 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:17.028924    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f05e099-7e7a-4c36-aeb8-6072904fdb5d-credential-socket\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:44:17.034124 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:17.028949    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f05e099-7e7a-4c36-aeb8-6072904fdb5d-workload-certs\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:44:17.034124 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:17.028993    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f05e099-7e7a-4c36-aeb8-6072904fdb5d-istio-data\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:44:17.034124 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:17.029006    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f05e099-7e7a-4c36-aeb8-6072904fdb5d-istiod-ca-cert\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:44:17.034124 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:17.029019    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f05e099-7e7a-4c36-aeb8-6072904fdb5d-workload-socket\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:44:17.035572 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:17.035548    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f05e099-7e7a-4c36-aeb8-6072904fdb5d-kube-api-access-7gpjt" (OuterVolumeSpecName: "kube-api-access-7gpjt") pod "7f05e099-7e7a-4c36-aeb8-6072904fdb5d" (UID: "7f05e099-7e7a-4c36-aeb8-6072904fdb5d"). InnerVolumeSpecName "kube-api-access-7gpjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:44:17.035689 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:17.035674    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f05e099-7e7a-4c36-aeb8-6072904fdb5d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7f05e099-7e7a-4c36-aeb8-6072904fdb5d" (UID: "7f05e099-7e7a-4c36-aeb8-6072904fdb5d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:44:17.130419 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:17.130376    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gpjt\" (UniqueName: \"kubernetes.io/projected/7f05e099-7e7a-4c36-aeb8-6072904fdb5d-kube-api-access-7gpjt\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:44:17.130419 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:17.130414    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f05e099-7e7a-4c36-aeb8-6072904fdb5d-istio-podinfo\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:44:17.130419 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:17.130425    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f05e099-7e7a-4c36-aeb8-6072904fdb5d-istio-token\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:44:17.130673 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:17.130434    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f05e099-7e7a-4c36-aeb8-6072904fdb5d-istio-envoy\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:44:17.330217 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:17.330182    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm"]
Sep 11 14:44:17.343354 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:17.343324    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-56b1b6-openshift-default-6d85d99798-bsmwm"]
Sep 11 14:44:17.501086 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:17.501051    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f05e099-7e7a-4c36-aeb8-6072904fdb5d" path="/var/lib/kubelet/pods/7f05e099-7e7a-4c36-aeb8-6072904fdb5d/volumes"
Sep 11 14:44:17.517188 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:17.517161    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-40d3a2-bdfd6f867-2jd4r_966ef1cb-e285-4d48-a36b-a2edcf3591ad/payload-processing/0.log"
Sep 11 14:44:19.828275 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:19.828229    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-f0ee6c/ipp-route-f0ee6c-kserve-5ddd6d97c5-5b4lt"]
Sep 11 14:44:19.828804 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:19.828783    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dda558e3-ff64-4c75-bcd4-0eedc88aa2d0" containerName="revoke-keys"
Sep 11 14:44:19.828892 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:19.828807    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="dda558e3-ff64-4c75-bcd4-0eedc88aa2d0" containerName="revoke-keys"
Sep 11 14:44:19.828892 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:19.828855    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f05e099-7e7a-4c36-aeb8-6072904fdb5d" containerName="istio-proxy"
Sep 11 14:44:19.828892 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:19.828865    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f05e099-7e7a-4c36-aeb8-6072904fdb5d" containerName="istio-proxy"
Sep 11 14:44:19.828892 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:19.828877    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f05e099-7e7a-4c36-aeb8-6072904fdb5d" containerName="istio-proxy"
Sep 11 14:44:19.828892 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:19.828886    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f05e099-7e7a-4c36-aeb8-6072904fdb5d" containerName="istio-proxy"
Sep 11 14:44:19.829172 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:19.829005    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="dda558e3-ff64-4c75-bcd4-0eedc88aa2d0" containerName="revoke-keys"
Sep 11 14:44:19.829172 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:19.829020    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f05e099-7e7a-4c36-aeb8-6072904fdb5d" containerName="istio-proxy"
Sep 11 14:44:19.829172 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:19.829030    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f05e099-7e7a-4c36-aeb8-6072904fdb5d" containerName="istio-proxy"
Sep 11 14:44:19.831899 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:19.831879    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f0ee6c/ipp-route-f0ee6c-kserve-5ddd6d97c5-5b4lt"
Sep 11 14:44:19.835471 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:19.835454    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-f0ee6c\"/\"kube-root-ca.crt\""
Sep 11 14:44:19.836790 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:19.836771    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-f0ee6c\"/\"ipp-route-f0ee6c-kserve-self-signed-certs\""
Sep 11 14:44:19.836903 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:19.836792    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-f0ee6c\"/\"openshift-service-ca.crt\""
Sep 11 14:44:19.836903 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:19.836815    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-f0ee6c\"/\"default-dockercfg-zntmz\""
Sep 11 14:44:19.845419 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:19.845396    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-f0ee6c/ipp-route-f0ee6c-kserve-5ddd6d97c5-5b4lt"]
Sep 11 14:44:19.954276 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:19.954242    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e633a8c-9e7b-48ac-99fc-ab6edabdb618-tls-certs\") pod \"ipp-route-f0ee6c-kserve-5ddd6d97c5-5b4lt\" (UID: \"0e633a8c-9e7b-48ac-99fc-ab6edabdb618\") " pod="ai-tenant-e2e-ipp-a-f0ee6c/ipp-route-f0ee6c-kserve-5ddd6d97c5-5b4lt"
Sep 11 14:44:19.954446 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:19.954283    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e633a8c-9e7b-48ac-99fc-ab6edabdb618-dshm\") pod \"ipp-route-f0ee6c-kserve-5ddd6d97c5-5b4lt\" (UID: \"0e633a8c-9e7b-48ac-99fc-ab6edabdb618\") " pod="ai-tenant-e2e-ipp-a-f0ee6c/ipp-route-f0ee6c-kserve-5ddd6d97c5-5b4lt"
Sep 11 14:44:19.954446 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:19.954306    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e633a8c-9e7b-48ac-99fc-ab6edabdb618-model-cache\") pod \"ipp-route-f0ee6c-kserve-5ddd6d97c5-5b4lt\" (UID: \"0e633a8c-9e7b-48ac-99fc-ab6edabdb618\") " pod="ai-tenant-e2e-ipp-a-f0ee6c/ipp-route-f0ee6c-kserve-5ddd6d97c5-5b4lt"
Sep 11 14:44:19.954446 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:19.954371    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5whr\" (UniqueName: \"kubernetes.io/projected/0e633a8c-9e7b-48ac-99fc-ab6edabdb618-kube-api-access-x5whr\") pod \"ipp-route-f0ee6c-kserve-5ddd6d97c5-5b4lt\" (UID: \"0e633a8c-9e7b-48ac-99fc-ab6edabdb618\") " pod="ai-tenant-e2e-ipp-a-f0ee6c/ipp-route-f0ee6c-kserve-5ddd6d97c5-5b4lt"
Sep 11 14:44:19.954446 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:19.954398    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e633a8c-9e7b-48ac-99fc-ab6edabdb618-tmp-dir\") pod \"ipp-route-f0ee6c-kserve-5ddd6d97c5-5b4lt\" (UID: \"0e633a8c-9e7b-48ac-99fc-ab6edabdb618\") " pod="ai-tenant-e2e-ipp-a-f0ee6c/ipp-route-f0ee6c-kserve-5ddd6d97c5-5b4lt"
Sep 11 14:44:19.954588 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:19.954504    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e633a8c-9e7b-48ac-99fc-ab6edabdb618-home\") pod \"ipp-route-f0ee6c-kserve-5ddd6d97c5-5b4lt\" (UID: \"0e633a8c-9e7b-48ac-99fc-ab6edabdb618\") " pod="ai-tenant-e2e-ipp-a-f0ee6c/ipp-route-f0ee6c-kserve-5ddd6d97c5-5b4lt"
Sep 11 14:44:20.055067 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:20.055033    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e633a8c-9e7b-48ac-99fc-ab6edabdb618-home\") pod \"ipp-route-f0ee6c-kserve-5ddd6d97c5-5b4lt\" (UID: \"0e633a8c-9e7b-48ac-99fc-ab6edabdb618\") " pod="ai-tenant-e2e-ipp-a-f0ee6c/ipp-route-f0ee6c-kserve-5ddd6d97c5-5b4lt"
Sep 11 14:44:20.055266 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:20.055083    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e633a8c-9e7b-48ac-99fc-ab6edabdb618-tls-certs\") pod \"ipp-route-f0ee6c-kserve-5ddd6d97c5-5b4lt\" (UID: \"0e633a8c-9e7b-48ac-99fc-ab6edabdb618\") " pod="ai-tenant-e2e-ipp-a-f0ee6c/ipp-route-f0ee6c-kserve-5ddd6d97c5-5b4lt"
Sep 11 14:44:20.055266 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:20.055112    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e633a8c-9e7b-48ac-99fc-ab6edabdb618-dshm\") pod \"ipp-route-f0ee6c-kserve-5ddd6d97c5-5b4lt\" (UID: \"0e633a8c-9e7b-48ac-99fc-ab6edabdb618\") " pod="ai-tenant-e2e-ipp-a-f0ee6c/ipp-route-f0ee6c-kserve-5ddd6d97c5-5b4lt"
Sep 11 14:44:20.055266 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:20.055137    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e633a8c-9e7b-48ac-99fc-ab6edabdb618-model-cache\") pod \"ipp-route-f0ee6c-kserve-5ddd6d97c5-5b4lt\" (UID: \"0e633a8c-9e7b-48ac-99fc-ab6edabdb618\") " pod="ai-tenant-e2e-ipp-a-f0ee6c/ipp-route-f0ee6c-kserve-5ddd6d97c5-5b4lt"
Sep 11 14:44:20.055266 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:20.055167    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5whr\" (UniqueName: \"kubernetes.io/projected/0e633a8c-9e7b-48ac-99fc-ab6edabdb618-kube-api-access-x5whr\") pod \"ipp-route-f0ee6c-kserve-5ddd6d97c5-5b4lt\" (UID: \"0e633a8c-9e7b-48ac-99fc-ab6edabdb618\") " pod="ai-tenant-e2e-ipp-a-f0ee6c/ipp-route-f0ee6c-kserve-5ddd6d97c5-5b4lt"
Sep 11 14:44:20.055266 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:20.055196    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e633a8c-9e7b-48ac-99fc-ab6edabdb618-tmp-dir\") pod \"ipp-route-f0ee6c-kserve-5ddd6d97c5-5b4lt\" (UID: \"0e633a8c-9e7b-48ac-99fc-ab6edabdb618\") " pod="ai-tenant-e2e-ipp-a-f0ee6c/ipp-route-f0ee6c-kserve-5ddd6d97c5-5b4lt"
Sep 11 14:44:20.055533 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:20.055493    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e633a8c-9e7b-48ac-99fc-ab6edabdb618-home\") pod \"ipp-route-f0ee6c-kserve-5ddd6d97c5-5b4lt\" (UID: \"0e633a8c-9e7b-48ac-99fc-ab6edabdb618\") " pod="ai-tenant-e2e-ipp-a-f0ee6c/ipp-route-f0ee6c-kserve-5ddd6d97c5-5b4lt"
Sep 11 14:44:20.055589 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:20.055541    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e633a8c-9e7b-48ac-99fc-ab6edabdb618-model-cache\") pod \"ipp-route-f0ee6c-kserve-5ddd6d97c5-5b4lt\" (UID: \"0e633a8c-9e7b-48ac-99fc-ab6edabdb618\") " pod="ai-tenant-e2e-ipp-a-f0ee6c/ipp-route-f0ee6c-kserve-5ddd6d97c5-5b4lt"
Sep 11 14:44:20.055648 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:20.055601    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e633a8c-9e7b-48ac-99fc-ab6edabdb618-tmp-dir\") pod \"ipp-route-f0ee6c-kserve-5ddd6d97c5-5b4lt\" (UID: \"0e633a8c-9e7b-48ac-99fc-ab6edabdb618\") " pod="ai-tenant-e2e-ipp-a-f0ee6c/ipp-route-f0ee6c-kserve-5ddd6d97c5-5b4lt"
Sep 11 14:44:20.057439 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:20.057415    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e633a8c-9e7b-48ac-99fc-ab6edabdb618-dshm\") pod \"ipp-route-f0ee6c-kserve-5ddd6d97c5-5b4lt\" (UID: \"0e633a8c-9e7b-48ac-99fc-ab6edabdb618\") " pod="ai-tenant-e2e-ipp-a-f0ee6c/ipp-route-f0ee6c-kserve-5ddd6d97c5-5b4lt"
Sep 11 14:44:20.057704 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:20.057687    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e633a8c-9e7b-48ac-99fc-ab6edabdb618-tls-certs\") pod \"ipp-route-f0ee6c-kserve-5ddd6d97c5-5b4lt\" (UID: \"0e633a8c-9e7b-48ac-99fc-ab6edabdb618\") " pod="ai-tenant-e2e-ipp-a-f0ee6c/ipp-route-f0ee6c-kserve-5ddd6d97c5-5b4lt"
Sep 11 14:44:20.072230 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:20.072207    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5whr\" (UniqueName: \"kubernetes.io/projected/0e633a8c-9e7b-48ac-99fc-ab6edabdb618-kube-api-access-x5whr\") pod \"ipp-route-f0ee6c-kserve-5ddd6d97c5-5b4lt\" (UID: \"0e633a8c-9e7b-48ac-99fc-ab6edabdb618\") " pod="ai-tenant-e2e-ipp-a-f0ee6c/ipp-route-f0ee6c-kserve-5ddd6d97c5-5b4lt"
Sep 11 14:44:20.141304 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:20.141198    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f0ee6c/ipp-route-f0ee6c-kserve-5ddd6d97c5-5b4lt"
Sep 11 14:44:20.432217 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:20.432184    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-f0ee6c/ipp-route-f0ee6c-kserve-5ddd6d97c5-5b4lt"]
Sep 11 14:44:20.434123 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:44:20.434095    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0e633a8c_9e7b_48ac_99fc_ab6edabdb618.slice/crio-874d0a60a87cac9048fa7c709144ed1320e2a698b70d13141b5396a88c93662e WatchSource:0}: Error finding container 874d0a60a87cac9048fa7c709144ed1320e2a698b70d13141b5396a88c93662e: Status 404 returned error can't find the container with id 874d0a60a87cac9048fa7c709144ed1320e2a698b70d13141b5396a88c93662e
Sep 11 14:44:21.023917 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:21.023879    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f0ee6c/ipp-route-f0ee6c-kserve-5ddd6d97c5-5b4lt" event={"ID":"0e633a8c-9e7b-48ac-99fc-ab6edabdb618","Type":"ContainerStarted","Data":"5e10198edaf82ada593d1cb4d553c4c29b003db29a7fceb902065d746954cccc"}
Sep 11 14:44:21.024345 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:21.023930    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f0ee6c/ipp-route-f0ee6c-kserve-5ddd6d97c5-5b4lt" event={"ID":"0e633a8c-9e7b-48ac-99fc-ab6edabdb618","Type":"ContainerStarted","Data":"874d0a60a87cac9048fa7c709144ed1320e2a698b70d13141b5396a88c93662e"}
Sep 11 14:44:21.069609 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:21.069555    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-f0ee6c/ipp-route-f0ee6c-kserve-5ddd6d97c5-5b4lt" podStartSLOduration=2.06953834 podStartE2EDuration="2.06953834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:44:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:44:21.069081396 +0000 UTC m=+1764.219631460" watchObservedRunningTime="2026-09-11 14:44:21.06953834 +0000 UTC m=+1764.220088460"
Sep 11 14:44:24.003473 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:24.003437    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-958398-p8mcj"]
Sep 11 14:44:24.010042 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:24.010012    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-958398-p8mcj"]
Sep 11 14:44:25.499297 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:25.499263    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dda558e3-ff64-4c75-bcd4-0eedc88aa2d0" path="/var/lib/kubelet/pods/dda558e3-ff64-4c75-bcd4-0eedc88aa2d0/volumes"
Sep 11 14:44:30.142319 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:30.142272    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-f0ee6c/ipp-route-f0ee6c-kserve-5ddd6d97c5-5b4lt"
Sep 11 14:44:30.142731 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:30.142341    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-f0ee6c/ipp-route-f0ee6c-kserve-5ddd6d97c5-5b4lt"
Sep 11 14:44:30.154964 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:30.154930    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-f0ee6c/ipp-route-f0ee6c-kserve-5ddd6d97c5-5b4lt"
Sep 11 14:44:31.098423 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:31.098390    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-f0ee6c/ipp-route-f0ee6c-kserve-5ddd6d97c5-5b4lt"
Sep 11 14:44:32.252042 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:32.252007    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-455f2e-xrttp"]
Sep 11 14:44:32.254482 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:32.254465    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-455f2e-xrttp"
Sep 11 14:44:32.257462 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:32.257441    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 14:44:32.257612 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:32.257580    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hcm4g\""
Sep 11 14:44:32.270818 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:32.270787    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df9efe80-6d26-43e0-9012-2fdd3fa1d30e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-455f2e-xrttp\" (UID: \"df9efe80-6d26-43e0-9012-2fdd3fa1d30e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-455f2e-xrttp"
Sep 11 14:44:32.273674 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:32.273651    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-455f2e-xrttp"]
Sep 11 14:44:32.371832 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:32.371799    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df9efe80-6d26-43e0-9012-2fdd3fa1d30e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-455f2e-xrttp\" (UID: \"df9efe80-6d26-43e0-9012-2fdd3fa1d30e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-455f2e-xrttp"
Sep 11 14:44:32.372466 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:32.372448    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df9efe80-6d26-43e0-9012-2fdd3fa1d30e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-455f2e-xrttp\" (UID: \"df9efe80-6d26-43e0-9012-2fdd3fa1d30e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-455f2e-xrttp"
Sep 11 14:44:32.565587 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:32.565498    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-455f2e-xrttp"
Sep 11 14:44:32.840459 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:32.840441    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-455f2e-xrttp"]
Sep 11 14:44:32.843198 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:44:32.843166    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf9efe80_6d26_43e0_9012_2fdd3fa1d30e.slice/crio-5a18a69d1fe597f92b1a095536a3c409e7f117d43dce44e2b1647ac310bbcc8f WatchSource:0}: Error finding container 5a18a69d1fe597f92b1a095536a3c409e7f117d43dce44e2b1647ac310bbcc8f: Status 404 returned error can't find the container with id 5a18a69d1fe597f92b1a095536a3c409e7f117d43dce44e2b1647ac310bbcc8f
Sep 11 14:44:33.100995 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:33.100949    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-455f2e-xrttp" event={"ID":"df9efe80-6d26-43e0-9012-2fdd3fa1d30e","Type":"ContainerStarted","Data":"5a18a69d1fe597f92b1a095536a3c409e7f117d43dce44e2b1647ac310bbcc8f"}
Sep 11 14:44:34.107653 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:34.107619    2581 generic.go:356] "Generic (PLEG): container finished" podID="df9efe80-6d26-43e0-9012-2fdd3fa1d30e" containerID="a051541d11f4b9b6f17463b1c1b6a497cad8c69f5479bf6491476da3676b81b1" exitCode=0
Sep 11 14:44:34.108064 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:34.107658    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-455f2e-xrttp" event={"ID":"df9efe80-6d26-43e0-9012-2fdd3fa1d30e","Type":"ContainerDied","Data":"a051541d11f4b9b6f17463b1c1b6a497cad8c69f5479bf6491476da3676b81b1"}
Sep 11 14:44:35.248673 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:35.248649    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-455f2e-xrttp"
Sep 11 14:44:35.298521 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:35.298492    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df9efe80-6d26-43e0-9012-2fdd3fa1d30e-maas-api-service-ca\") pod \"df9efe80-6d26-43e0-9012-2fdd3fa1d30e\" (UID: \"df9efe80-6d26-43e0-9012-2fdd3fa1d30e\") "
Sep 11 14:44:35.298822 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:35.298800    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df9efe80-6d26-43e0-9012-2fdd3fa1d30e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "df9efe80-6d26-43e0-9012-2fdd3fa1d30e" (UID: "df9efe80-6d26-43e0-9012-2fdd3fa1d30e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:44:35.400111 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:35.400026    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df9efe80-6d26-43e0-9012-2fdd3fa1d30e-maas-api-service-ca\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:44:36.119998 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:36.119947    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-455f2e-xrttp"
Sep 11 14:44:36.120321 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:36.120004    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-455f2e-xrttp" event={"ID":"df9efe80-6d26-43e0-9012-2fdd3fa1d30e","Type":"ContainerDied","Data":"5a18a69d1fe597f92b1a095536a3c409e7f117d43dce44e2b1647ac310bbcc8f"}
Sep 11 14:44:36.120321 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:36.120055    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a18a69d1fe597f92b1a095536a3c409e7f117d43dce44e2b1647ac310bbcc8f"
Sep 11 14:44:43.424550 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:43.424516    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-455f2e-85b7989885-2qzkr"]
Sep 11 14:44:43.424992 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:43.424781    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-455f2e-85b7989885-2qzkr" podUID="cb4f6189-03d0-4410-86f7-f5438a766546" containerName="payload-processing" containerID="cri-o://8d0490d6f6a432a667c90f78cf436971e9e75c01ff4bdce416ebe901b3496d94" gracePeriod=30
Sep 11 14:44:43.701113 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:43.701085    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-455f2e-85b7989885-2qzkr"
Sep 11 14:44:43.712827 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:43.712800    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-455f2e-xrttp"]
Sep 11 14:44:43.716612 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:43.716587    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-455f2e-xrttp"]
Sep 11 14:44:43.791843 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:43.791813    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb4f6189-03d0-4410-86f7-f5438a766546-plugins-config-volume\") pod \"cb4f6189-03d0-4410-86f7-f5438a766546\" (UID: \"cb4f6189-03d0-4410-86f7-f5438a766546\") "
Sep 11 14:44:43.792060 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:43.791926    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fz5bj\" (UniqueName: \"kubernetes.io/projected/cb4f6189-03d0-4410-86f7-f5438a766546-kube-api-access-fz5bj\") pod \"cb4f6189-03d0-4410-86f7-f5438a766546\" (UID: \"cb4f6189-03d0-4410-86f7-f5438a766546\") "
Sep 11 14:44:43.792211 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:43.792188    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb4f6189-03d0-4410-86f7-f5438a766546-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cb4f6189-03d0-4410-86f7-f5438a766546" (UID: "cb4f6189-03d0-4410-86f7-f5438a766546"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:44:43.793957 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:43.793935    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb4f6189-03d0-4410-86f7-f5438a766546-kube-api-access-fz5bj" (OuterVolumeSpecName: "kube-api-access-fz5bj") pod "cb4f6189-03d0-4410-86f7-f5438a766546" (UID: "cb4f6189-03d0-4410-86f7-f5438a766546"). InnerVolumeSpecName "kube-api-access-fz5bj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:44:43.893716 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:43.893676    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz5bj\" (UniqueName: \"kubernetes.io/projected/cb4f6189-03d0-4410-86f7-f5438a766546-kube-api-access-fz5bj\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:44:43.893716 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:43.893711    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb4f6189-03d0-4410-86f7-f5438a766546-plugins-config-volume\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:44:44.169000 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:44.168936    2581 generic.go:356] "Generic (PLEG): container finished" podID="cb4f6189-03d0-4410-86f7-f5438a766546" containerID="8d0490d6f6a432a667c90f78cf436971e9e75c01ff4bdce416ebe901b3496d94" exitCode=0
Sep 11 14:44:44.169208 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:44.169013    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-455f2e-85b7989885-2qzkr"
Sep 11 14:44:44.169208 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:44.169034    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-455f2e-85b7989885-2qzkr" event={"ID":"cb4f6189-03d0-4410-86f7-f5438a766546","Type":"ContainerDied","Data":"8d0490d6f6a432a667c90f78cf436971e9e75c01ff4bdce416ebe901b3496d94"}
Sep 11 14:44:44.169208 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:44.169091    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-455f2e-85b7989885-2qzkr" event={"ID":"cb4f6189-03d0-4410-86f7-f5438a766546","Type":"ContainerDied","Data":"4e5df498c931ec004127cd3797fe88f8808725acc9514f5479da58ffe6ffa4ef"}
Sep 11 14:44:44.169208 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:44.169119    2581 scope.go:117] "RemoveContainer" containerID="8d0490d6f6a432a667c90f78cf436971e9e75c01ff4bdce416ebe901b3496d94"
Sep 11 14:44:44.178148 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:44.178127    2581 scope.go:117] "RemoveContainer" containerID="8d0490d6f6a432a667c90f78cf436971e9e75c01ff4bdce416ebe901b3496d94"
Sep 11 14:44:44.178429 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:44:44.178407    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d0490d6f6a432a667c90f78cf436971e9e75c01ff4bdce416ebe901b3496d94\": container with ID starting with 8d0490d6f6a432a667c90f78cf436971e9e75c01ff4bdce416ebe901b3496d94 not found: ID does not exist" containerID="8d0490d6f6a432a667c90f78cf436971e9e75c01ff4bdce416ebe901b3496d94"
Sep 11 14:44:44.178498 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:44.178444    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d0490d6f6a432a667c90f78cf436971e9e75c01ff4bdce416ebe901b3496d94"} err="failed to get container status \"8d0490d6f6a432a667c90f78cf436971e9e75c01ff4bdce416ebe901b3496d94\": rpc error: code = NotFound desc = could not find container \"8d0490d6f6a432a667c90f78cf436971e9e75c01ff4bdce416ebe901b3496d94\": container with ID starting with 8d0490d6f6a432a667c90f78cf436971e9e75c01ff4bdce416ebe901b3496d94 not found: ID does not exist"
Sep 11 14:44:44.194038 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:44.194002    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-455f2e-85b7989885-2qzkr"]
Sep 11 14:44:44.196557 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:44.196531    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-455f2e-85b7989885-2qzkr"]
Sep 11 14:44:45.499839 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:45.499802    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb4f6189-03d0-4410-86f7-f5438a766546" path="/var/lib/kubelet/pods/cb4f6189-03d0-4410-86f7-f5438a766546/volumes"
Sep 11 14:44:45.500412 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:45.500388    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df9efe80-6d26-43e0-9012-2fdd3fa1d30e" path="/var/lib/kubelet/pods/df9efe80-6d26-43e0-9012-2fdd3fa1d30e/volumes"
Sep 11 14:44:56.302853 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:44:56.302823    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-40d3a2-bdfd6f867-2jd4r_966ef1cb-e285-4d48-a36b-a2edcf3591ad/payload-processing/0.log"
Sep 11 14:45:00.176204 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:00.176165    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818965-vtp8m"]
Sep 11 14:45:00.176618 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:00.176560    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb4f6189-03d0-4410-86f7-f5438a766546" containerName="payload-processing"
Sep 11 14:45:00.176618 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:00.176573    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb4f6189-03d0-4410-86f7-f5438a766546" containerName="payload-processing"
Sep 11 14:45:00.176618 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:00.176583    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df9efe80-6d26-43e0-9012-2fdd3fa1d30e" containerName="revoke-keys"
Sep 11 14:45:00.176618 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:00.176589    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="df9efe80-6d26-43e0-9012-2fdd3fa1d30e" containerName="revoke-keys"
Sep 11 14:45:00.176755 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:00.176648    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb4f6189-03d0-4410-86f7-f5438a766546" containerName="payload-processing"
Sep 11 14:45:00.176755 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:00.176659    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="df9efe80-6d26-43e0-9012-2fdd3fa1d30e" containerName="revoke-keys"
Sep 11 14:45:00.179566 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:00.179548    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-09b16c-29818965-mzt9j"]
Sep 11 14:45:00.179678 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:00.179658    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818965-vtp8m"
Sep 11 14:45:00.182835 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:00.182817    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-09b16c-29818965-mzt9j"
Sep 11 14:45:00.183139 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:00.183123    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hcm4g\""
Sep 11 14:45:00.190221 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:00.190203    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818965-vtp8m"
Sep 11 14:45:00.195217 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:00.195198    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1c284e-29818965-vfmgq"]
Sep 11 14:45:00.198362 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:00.198343    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-09b16c-29818965-mzt9j"
Sep 11 14:45:00.198713 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:00.198700    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1c284e-29818965-vfmgq"
Sep 11 14:45:00.207678 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:00.207605    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-09b16c-29818965-mzt9j"]
Sep 11 14:45:00.212223 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:00.212148    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818965-vtp8m"]
Sep 11 14:45:00.238890 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:00.238861    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1c284e-29818965-vfmgq"
Sep 11 14:45:00.274016 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:00.273984    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1c284e-29818965-vfmgq"]
Sep 11 14:45:00.499889 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:45:00.499858    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36ffa317_c41c_4b70_9bab_39865d4fd0e8.slice/crio-66e6ec230641c5c43fcbc364ea1af99193eb44a18f27c2a09494de330ee25cb7 WatchSource:0}: Error finding container 66e6ec230641c5c43fcbc364ea1af99193eb44a18f27c2a09494de330ee25cb7: Status 404 returned error can't find the container with id 66e6ec230641c5c43fcbc364ea1af99193eb44a18f27c2a09494de330ee25cb7
Sep 11 14:45:00.500190 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:00.500168    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818965-vtp8m"]
Sep 11 14:45:00.569343 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:00.569265    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1c284e-29818965-vfmgq"]
Sep 11 14:45:00.571596 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:45:00.571570    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e980be9_5d70_4705_b648_aa490a6b1aa9.slice/crio-03452a884094fc809451605c7d3bd6990b5bfb9df048fcff9bc5f04d3d816ca1 WatchSource:0}: Error finding container 03452a884094fc809451605c7d3bd6990b5bfb9df048fcff9bc5f04d3d816ca1: Status 404 returned error can't find the container with id 03452a884094fc809451605c7d3bd6990b5bfb9df048fcff9bc5f04d3d816ca1
Sep 11 14:45:00.585867 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:00.585831    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-09b16c-29818965-mzt9j"]
Sep 11 14:45:00.586963 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:45:00.586887    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb65d9252_52a0_47c0_80e9_4d5473a46754.slice/crio-2100922e61508e39c659e7582f14ba44c5c5d3048c1872a6cdeb37570ecf5f37 WatchSource:0}: Error finding container 2100922e61508e39c659e7582f14ba44c5c5d3048c1872a6cdeb37570ecf5f37: Status 404 returned error can't find the container with id 2100922e61508e39c659e7582f14ba44c5c5d3048c1872a6cdeb37570ecf5f37
Sep 11 14:45:01.277222 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:01.277178    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-09b16c-29818965-mzt9j" event={"ID":"b65d9252-52a0-47c0-80e9-4d5473a46754","Type":"ContainerStarted","Data":"5dd4e2b205bec3d8b687c6635369e48aaa2c8f4db12ab57715608e5c57646e49"}
Sep 11 14:45:01.277660 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:01.277247    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-09b16c-29818965-mzt9j" event={"ID":"b65d9252-52a0-47c0-80e9-4d5473a46754","Type":"ContainerStarted","Data":"2100922e61508e39c659e7582f14ba44c5c5d3048c1872a6cdeb37570ecf5f37"}
Sep 11 14:45:01.278785 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:01.278756    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1c284e-29818965-vfmgq" event={"ID":"8e980be9-5d70-4705-b648-aa490a6b1aa9","Type":"ContainerStarted","Data":"449e462b597d67b3326ed3d96d53d9585a6c2a476d3b000c2eb2cfaedda42583"}
Sep 11 14:45:01.278919 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:01.278813    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1c284e-29818965-vfmgq" event={"ID":"8e980be9-5d70-4705-b648-aa490a6b1aa9","Type":"ContainerStarted","Data":"03452a884094fc809451605c7d3bd6990b5bfb9df048fcff9bc5f04d3d816ca1"}
Sep 11 14:45:01.280237 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:01.280216    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818965-vtp8m" event={"ID":"36ffa317-c41c-4b70-9bab-39865d4fd0e8","Type":"ContainerStarted","Data":"c986b62131da4cd8748a0207bf374c3ab12e508cc768de6a6d69a37551c47e34"}
Sep 11 14:45:01.280307 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:01.280267    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818965-vtp8m" event={"ID":"36ffa317-c41c-4b70-9bab-39865d4fd0e8","Type":"ContainerStarted","Data":"66e6ec230641c5c43fcbc364ea1af99193eb44a18f27c2a09494de330ee25cb7"}
Sep 11 14:45:01.303325 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:01.303274    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-09b16c-29818965-mzt9j" podStartSLOduration=1.303259634 podStartE2EDuration="1.303259634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:45:01.301040971 +0000 UTC m=+1804.451591045" watchObservedRunningTime="2026-09-11 14:45:01.303259634 +0000 UTC m=+1804.453809738"
Sep 11 14:45:01.327473 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:01.327418    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818965-vtp8m" podStartSLOduration=1.327401208 podStartE2EDuration="1.327401208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:45:01.326668517 +0000 UTC m=+1804.477218593" watchObservedRunningTime="2026-09-11 14:45:01.327401208 +0000 UTC m=+1804.477951272"
Sep 11 14:45:01.348950 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:01.348899    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1c284e-29818965-vfmgq" podStartSLOduration=1.348881073 podStartE2EDuration="1.348881073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:45:01.347273614 +0000 UTC m=+1804.497823677" watchObservedRunningTime="2026-09-11 14:45:01.348881073 +0000 UTC m=+1804.499431137"
Sep 11 14:45:02.287381 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:02.287345    2581 generic.go:356] "Generic (PLEG): container finished" podID="b65d9252-52a0-47c0-80e9-4d5473a46754" containerID="5dd4e2b205bec3d8b687c6635369e48aaa2c8f4db12ab57715608e5c57646e49" exitCode=0
Sep 11 14:45:02.287871 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:02.287437    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-09b16c-29818965-mzt9j" event={"ID":"b65d9252-52a0-47c0-80e9-4d5473a46754","Type":"ContainerDied","Data":"5dd4e2b205bec3d8b687c6635369e48aaa2c8f4db12ab57715608e5c57646e49"}
Sep 11 14:45:02.289061 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:02.289036    2581 generic.go:356] "Generic (PLEG): container finished" podID="8e980be9-5d70-4705-b648-aa490a6b1aa9" containerID="449e462b597d67b3326ed3d96d53d9585a6c2a476d3b000c2eb2cfaedda42583" exitCode=0
Sep 11 14:45:02.289164 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:02.289117    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1c284e-29818965-vfmgq" event={"ID":"8e980be9-5d70-4705-b648-aa490a6b1aa9","Type":"ContainerDied","Data":"449e462b597d67b3326ed3d96d53d9585a6c2a476d3b000c2eb2cfaedda42583"}
Sep 11 14:45:02.290518 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:02.290499    2581 generic.go:356] "Generic (PLEG): container finished" podID="36ffa317-c41c-4b70-9bab-39865d4fd0e8" containerID="c986b62131da4cd8748a0207bf374c3ab12e508cc768de6a6d69a37551c47e34" exitCode=0
Sep 11 14:45:02.290575 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:02.290532    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818965-vtp8m" event={"ID":"36ffa317-c41c-4b70-9bab-39865d4fd0e8","Type":"ContainerDied","Data":"c986b62131da4cd8748a0207bf374c3ab12e508cc768de6a6d69a37551c47e34"}
Sep 11 14:45:03.460980 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:03.460932    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-09b16c-29818965-mzt9j"
Sep 11 14:45:03.501660 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:03.501633    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1c284e-29818965-vfmgq"
Sep 11 14:45:03.505703 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:03.505679    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818965-vtp8m"
Sep 11 14:45:04.301653 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:04.301623    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-09b16c-29818965-mzt9j"
Sep 11 14:45:04.301834 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:04.301665    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-09b16c-29818965-mzt9j" event={"ID":"b65d9252-52a0-47c0-80e9-4d5473a46754","Type":"ContainerDied","Data":"2100922e61508e39c659e7582f14ba44c5c5d3048c1872a6cdeb37570ecf5f37"}
Sep 11 14:45:04.301834 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:04.301723    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2100922e61508e39c659e7582f14ba44c5c5d3048c1872a6cdeb37570ecf5f37"
Sep 11 14:45:04.303106 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:04.303086    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1c284e-29818965-vfmgq"
Sep 11 14:45:04.303106 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:04.303092    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1c284e-29818965-vfmgq" event={"ID":"8e980be9-5d70-4705-b648-aa490a6b1aa9","Type":"ContainerDied","Data":"03452a884094fc809451605c7d3bd6990b5bfb9df048fcff9bc5f04d3d816ca1"}
Sep 11 14:45:04.303292 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:04.303133    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03452a884094fc809451605c7d3bd6990b5bfb9df048fcff9bc5f04d3d816ca1"
Sep 11 14:45:04.304571 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:04.304553    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818965-vtp8m" event={"ID":"36ffa317-c41c-4b70-9bab-39865d4fd0e8","Type":"ContainerDied","Data":"66e6ec230641c5c43fcbc364ea1af99193eb44a18f27c2a09494de330ee25cb7"}
Sep 11 14:45:04.304647 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:04.304585    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818965-vtp8m"
Sep 11 14:45:04.304695 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:04.304587    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66e6ec230641c5c43fcbc364ea1af99193eb44a18f27c2a09494de330ee25cb7"
Sep 11 14:45:31.102517 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:31.102480    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8f7967-qb9rm"]
Sep 11 14:45:31.103033 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:31.102857    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e980be9-5d70-4705-b648-aa490a6b1aa9" containerName="cleanup"
Sep 11 14:45:31.103033 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:31.102868    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e980be9-5d70-4705-b648-aa490a6b1aa9" containerName="cleanup"
Sep 11 14:45:31.103033 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:31.102880    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36ffa317-c41c-4b70-9bab-39865d4fd0e8" containerName="cleanup"
Sep 11 14:45:31.103033 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:31.102885    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="36ffa317-c41c-4b70-9bab-39865d4fd0e8" containerName="cleanup"
Sep 11 14:45:31.103033 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:31.102903    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b65d9252-52a0-47c0-80e9-4d5473a46754" containerName="cleanup"
Sep 11 14:45:31.103033 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:31.102908    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b65d9252-52a0-47c0-80e9-4d5473a46754" containerName="cleanup"
Sep 11 14:45:31.103033 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:31.103021    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e980be9-5d70-4705-b648-aa490a6b1aa9" containerName="cleanup"
Sep 11 14:45:31.103033 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:31.103032    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="36ffa317-c41c-4b70-9bab-39865d4fd0e8" containerName="cleanup"
Sep 11 14:45:31.103033 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:31.103039    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b65d9252-52a0-47c0-80e9-4d5473a46754" containerName="cleanup"
Sep 11 14:45:31.106234 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:31.106213    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8f7967-qb9rm"
Sep 11 14:45:31.117609 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:31.117584    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 14:45:31.119700 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:31.119681    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hcm4g\""
Sep 11 14:45:31.135903 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:31.135871    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8f7967-qb9rm"]
Sep 11 14:45:31.272048 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:31.272009    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee5805a0-7fae-4eee-a274-b63949aa7672-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-8f7967-qb9rm\" (UID: \"ee5805a0-7fae-4eee-a274-b63949aa7672\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8f7967-qb9rm"
Sep 11 14:45:31.372765 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:31.372669    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee5805a0-7fae-4eee-a274-b63949aa7672-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-8f7967-qb9rm\" (UID: \"ee5805a0-7fae-4eee-a274-b63949aa7672\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8f7967-qb9rm"
Sep 11 14:45:31.373348 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:31.373330    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee5805a0-7fae-4eee-a274-b63949aa7672-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-8f7967-qb9rm\" (UID: \"ee5805a0-7fae-4eee-a274-b63949aa7672\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8f7967-qb9rm"
Sep 11 14:45:31.416402 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:31.416360    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8f7967-qb9rm"
Sep 11 14:45:31.697476 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:31.697444    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8f7967-qb9rm"]
Sep 11 14:45:31.699756 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:45:31.699726    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee5805a0_7fae_4eee_a274_b63949aa7672.slice/crio-47fdb10a3d4031a7984c55cd66c8cade2da700450bb12cf7c9da577b4ff465d8 WatchSource:0}: Error finding container 47fdb10a3d4031a7984c55cd66c8cade2da700450bb12cf7c9da577b4ff465d8: Status 404 returned error can't find the container with id 47fdb10a3d4031a7984c55cd66c8cade2da700450bb12cf7c9da577b4ff465d8
Sep 11 14:45:32.463762 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:32.463728    2581 generic.go:356] "Generic (PLEG): container finished" podID="ee5805a0-7fae-4eee-a274-b63949aa7672" containerID="1da8fd484703f7a2f36fcfc5a95c6cf7b64479a48ba55efe0177d2e5d90ca292" exitCode=0
Sep 11 14:45:32.464202 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:32.463777    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8f7967-qb9rm" event={"ID":"ee5805a0-7fae-4eee-a274-b63949aa7672","Type":"ContainerDied","Data":"1da8fd484703f7a2f36fcfc5a95c6cf7b64479a48ba55efe0177d2e5d90ca292"}
Sep 11 14:45:32.464202 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:32.463818    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8f7967-qb9rm" event={"ID":"ee5805a0-7fae-4eee-a274-b63949aa7672","Type":"ContainerStarted","Data":"47fdb10a3d4031a7984c55cd66c8cade2da700450bb12cf7c9da577b4ff465d8"}
Sep 11 14:45:33.602252 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:33.602224    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8f7967-qb9rm"
Sep 11 14:45:33.697566 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:33.697531    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee5805a0-7fae-4eee-a274-b63949aa7672-maas-api-service-ca\") pod \"ee5805a0-7fae-4eee-a274-b63949aa7672\" (UID: \"ee5805a0-7fae-4eee-a274-b63949aa7672\") "
Sep 11 14:45:33.697901 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:33.697877    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee5805a0-7fae-4eee-a274-b63949aa7672-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ee5805a0-7fae-4eee-a274-b63949aa7672" (UID: "ee5805a0-7fae-4eee-a274-b63949aa7672"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:45:33.799141 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:33.799099    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee5805a0-7fae-4eee-a274-b63949aa7672-maas-api-service-ca\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:45:34.476141 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:34.476108    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8f7967-qb9rm"
Sep 11 14:45:34.476331 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:34.476135    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8f7967-qb9rm" event={"ID":"ee5805a0-7fae-4eee-a274-b63949aa7672","Type":"ContainerDied","Data":"47fdb10a3d4031a7984c55cd66c8cade2da700450bb12cf7c9da577b4ff465d8"}
Sep 11 14:45:34.476331 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:34.476188    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47fdb10a3d4031a7984c55cd66c8cade2da700450bb12cf7c9da577b4ff465d8"
Sep 11 14:45:43.509516 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:43.509481    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8f7967-qb9rm"]
Sep 11 14:45:43.511565 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:43.511541    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8f7967-qb9rm"]
Sep 11 14:45:45.499186 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:45.499153    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee5805a0-7fae-4eee-a274-b63949aa7672" path="/var/lib/kubelet/pods/ee5805a0-7fae-4eee-a274-b63949aa7672/volumes"
Sep 11 14:45:56.443603 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:56.443561    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0ee6c-g5gfx"]
Sep 11 14:45:56.444037 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:56.443992    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee5805a0-7fae-4eee-a274-b63949aa7672" containerName="revoke-keys"
Sep 11 14:45:56.444037 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:56.444007    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee5805a0-7fae-4eee-a274-b63949aa7672" containerName="revoke-keys"
Sep 11 14:45:56.444122 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:56.444096    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee5805a0-7fae-4eee-a274-b63949aa7672" containerName="revoke-keys"
Sep 11 14:45:56.447451 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:56.447434    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0ee6c-g5gfx"
Sep 11 14:45:56.453443 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:56.453418    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 14:45:56.454029 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:56.454005    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hcm4g\""
Sep 11 14:45:56.465057 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:56.465023    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0ee6c-g5gfx"]
Sep 11 14:45:56.532408 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:56.532377    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c9f9d8d-1224-472a-bcfc-0ded6303bba4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f0ee6c-g5gfx\" (UID: \"7c9f9d8d-1224-472a-bcfc-0ded6303bba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0ee6c-g5gfx"
Sep 11 14:45:56.633997 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:56.633930    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c9f9d8d-1224-472a-bcfc-0ded6303bba4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f0ee6c-g5gfx\" (UID: \"7c9f9d8d-1224-472a-bcfc-0ded6303bba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0ee6c-g5gfx"
Sep 11 14:45:56.634634 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:56.634612    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c9f9d8d-1224-472a-bcfc-0ded6303bba4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f0ee6c-g5gfx\" (UID: \"7c9f9d8d-1224-472a-bcfc-0ded6303bba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0ee6c-g5gfx"
Sep 11 14:45:56.757378 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:56.757333    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0ee6c-g5gfx"
Sep 11 14:45:57.038121 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:57.038078    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0ee6c-g5gfx"]
Sep 11 14:45:57.039298 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:45:57.039269    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c9f9d8d_1224_472a_bcfc_0ded6303bba4.slice/crio-f4c72eaf69b4c9b4fcd7ea16b80a01528424285b533f5c6c91380ba9c3b73767 WatchSource:0}: Error finding container f4c72eaf69b4c9b4fcd7ea16b80a01528424285b533f5c6c91380ba9c3b73767: Status 404 returned error can't find the container with id f4c72eaf69b4c9b4fcd7ea16b80a01528424285b533f5c6c91380ba9c3b73767
Sep 11 14:45:57.599198 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:57.599160    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0ee6c-g5gfx" event={"ID":"7c9f9d8d-1224-472a-bcfc-0ded6303bba4","Type":"ContainerStarted","Data":"ae268b2b9ecd63f37bc4598f0a8c6bbf09f3e8088b02bea1759002609c99c9ed"}
Sep 11 14:45:57.599599 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:57.599224    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0ee6c-g5gfx" event={"ID":"7c9f9d8d-1224-472a-bcfc-0ded6303bba4","Type":"ContainerStarted","Data":"f4c72eaf69b4c9b4fcd7ea16b80a01528424285b533f5c6c91380ba9c3b73767"}
Sep 11 14:45:57.685179 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:57.685145    2581 scope.go:117] "RemoveContainer" containerID="67f304e08cb6fe9c8c76f51198253c3690104442744021aec9400d4ddad4e963"
Sep 11 14:45:58.606353 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:58.606316    2581 generic.go:356] "Generic (PLEG): container finished" podID="7c9f9d8d-1224-472a-bcfc-0ded6303bba4" containerID="ae268b2b9ecd63f37bc4598f0a8c6bbf09f3e8088b02bea1759002609c99c9ed" exitCode=0
Sep 11 14:45:58.606722 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:58.606402    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0ee6c-g5gfx" event={"ID":"7c9f9d8d-1224-472a-bcfc-0ded6303bba4","Type":"ContainerDied","Data":"ae268b2b9ecd63f37bc4598f0a8c6bbf09f3e8088b02bea1759002609c99c9ed"}
Sep 11 14:45:59.741640 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:59.741614    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0ee6c-g5gfx"
Sep 11 14:45:59.868402 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:59.868311    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c9f9d8d-1224-472a-bcfc-0ded6303bba4-maas-api-service-ca\") pod \"7c9f9d8d-1224-472a-bcfc-0ded6303bba4\" (UID: \"7c9f9d8d-1224-472a-bcfc-0ded6303bba4\") "
Sep 11 14:45:59.868683 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:59.868660    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c9f9d8d-1224-472a-bcfc-0ded6303bba4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7c9f9d8d-1224-472a-bcfc-0ded6303bba4" (UID: "7c9f9d8d-1224-472a-bcfc-0ded6303bba4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:45:59.970027 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:45:59.969948    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c9f9d8d-1224-472a-bcfc-0ded6303bba4-maas-api-service-ca\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:46:00.619457 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:46:00.619430    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0ee6c-g5gfx"
Sep 11 14:46:00.619646 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:46:00.619468    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0ee6c-g5gfx" event={"ID":"7c9f9d8d-1224-472a-bcfc-0ded6303bba4","Type":"ContainerDied","Data":"f4c72eaf69b4c9b4fcd7ea16b80a01528424285b533f5c6c91380ba9c3b73767"}
Sep 11 14:46:00.619646 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:46:00.619513    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4c72eaf69b4c9b4fcd7ea16b80a01528424285b533f5c6c91380ba9c3b73767"
Sep 11 14:46:12.455489 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:46:12.455449    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0ee6c-g5gfx"]
Sep 11 14:46:12.458894 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:46:12.458868    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0ee6c-g5gfx"]
Sep 11 14:46:13.504434 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:46:13.504395    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c9f9d8d-1224-472a-bcfc-0ded6303bba4" path="/var/lib/kubelet/pods/7c9f9d8d-1224-472a-bcfc-0ded6303bba4/volumes"
Sep 11 14:46:17.862206 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:46:17.862169    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-f0ee6c/ipp-route-f0ee6c-kserve-5ddd6d97c5-5b4lt"]
Sep 11 14:46:32.881320 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:46:32.881196    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-f0ee6c/ipp-route-f0ee6c-kserve-5ddd6d97c5-5b4lt" podUID="0e633a8c-9e7b-48ac-99fc-ab6edabdb618" containerName="main" containerID="cri-o://5e10198edaf82ada593d1cb4d553c4c29b003db29a7fceb902065d746954cccc" gracePeriod=45
Sep 11 14:46:33.137782 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:46:33.137716    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f0ee6c/ipp-route-f0ee6c-kserve-5ddd6d97c5-5b4lt"
Sep 11 14:46:33.310255 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:46:33.310216    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e633a8c-9e7b-48ac-99fc-ab6edabdb618-dshm\") pod \"0e633a8c-9e7b-48ac-99fc-ab6edabdb618\" (UID: \"0e633a8c-9e7b-48ac-99fc-ab6edabdb618\") "
Sep 11 14:46:33.310255 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:46:33.310259    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e633a8c-9e7b-48ac-99fc-ab6edabdb618-home\") pod \"0e633a8c-9e7b-48ac-99fc-ab6edabdb618\" (UID: \"0e633a8c-9e7b-48ac-99fc-ab6edabdb618\") "
Sep 11 14:46:33.310502 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:46:33.310292    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e633a8c-9e7b-48ac-99fc-ab6edabdb618-model-cache\") pod \"0e633a8c-9e7b-48ac-99fc-ab6edabdb618\" (UID: \"0e633a8c-9e7b-48ac-99fc-ab6edabdb618\") "
Sep 11 14:46:33.310502 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:46:33.310324    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5whr\" (UniqueName: \"kubernetes.io/projected/0e633a8c-9e7b-48ac-99fc-ab6edabdb618-kube-api-access-x5whr\") pod \"0e633a8c-9e7b-48ac-99fc-ab6edabdb618\" (UID: \"0e633a8c-9e7b-48ac-99fc-ab6edabdb618\") "
Sep 11 14:46:33.310502 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:46:33.310345    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e633a8c-9e7b-48ac-99fc-ab6edabdb618-tls-certs\") pod \"0e633a8c-9e7b-48ac-99fc-ab6edabdb618\" (UID: \"0e633a8c-9e7b-48ac-99fc-ab6edabdb618\") "
Sep 11 14:46:33.310502 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:46:33.310409    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e633a8c-9e7b-48ac-99fc-ab6edabdb618-tmp-dir\") pod \"0e633a8c-9e7b-48ac-99fc-ab6edabdb618\" (UID: \"0e633a8c-9e7b-48ac-99fc-ab6edabdb618\") "
Sep 11 14:46:33.310726 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:46:33.310536    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e633a8c-9e7b-48ac-99fc-ab6edabdb618-home" (OuterVolumeSpecName: "home") pod "0e633a8c-9e7b-48ac-99fc-ab6edabdb618" (UID: "0e633a8c-9e7b-48ac-99fc-ab6edabdb618"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:46:33.310726 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:46:33.310595    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e633a8c-9e7b-48ac-99fc-ab6edabdb618-model-cache" (OuterVolumeSpecName: "model-cache") pod "0e633a8c-9e7b-48ac-99fc-ab6edabdb618" (UID: "0e633a8c-9e7b-48ac-99fc-ab6edabdb618"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:46:33.310877 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:46:33.310746    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e633a8c-9e7b-48ac-99fc-ab6edabdb618-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0e633a8c-9e7b-48ac-99fc-ab6edabdb618" (UID: "0e633a8c-9e7b-48ac-99fc-ab6edabdb618"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:46:33.310877 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:46:33.310834    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e633a8c-9e7b-48ac-99fc-ab6edabdb618-home\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:46:33.310877 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:46:33.310852    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e633a8c-9e7b-48ac-99fc-ab6edabdb618-model-cache\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:46:33.310877 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:46:33.310867    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e633a8c-9e7b-48ac-99fc-ab6edabdb618-tmp-dir\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:46:33.312640 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:46:33.312615    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e633a8c-9e7b-48ac-99fc-ab6edabdb618-dshm" (OuterVolumeSpecName: "dshm") pod "0e633a8c-9e7b-48ac-99fc-ab6edabdb618" (UID: "0e633a8c-9e7b-48ac-99fc-ab6edabdb618"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:46:33.312749 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:46:33.312686    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e633a8c-9e7b-48ac-99fc-ab6edabdb618-kube-api-access-x5whr" (OuterVolumeSpecName: "kube-api-access-x5whr") pod "0e633a8c-9e7b-48ac-99fc-ab6edabdb618" (UID: "0e633a8c-9e7b-48ac-99fc-ab6edabdb618"). InnerVolumeSpecName "kube-api-access-x5whr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:46:33.312749 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:46:33.312682    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e633a8c-9e7b-48ac-99fc-ab6edabdb618-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0e633a8c-9e7b-48ac-99fc-ab6edabdb618" (UID: "0e633a8c-9e7b-48ac-99fc-ab6edabdb618"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:46:33.412134 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:46:33.412047    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e633a8c-9e7b-48ac-99fc-ab6edabdb618-dshm\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:46:33.412134 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:46:33.412077    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5whr\" (UniqueName: \"kubernetes.io/projected/0e633a8c-9e7b-48ac-99fc-ab6edabdb618-kube-api-access-x5whr\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:46:33.412134 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:46:33.412087    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e633a8c-9e7b-48ac-99fc-ab6edabdb618-tls-certs\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:46:33.798163 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:46:33.798128    2581 generic.go:356] "Generic (PLEG): container finished" podID="0e633a8c-9e7b-48ac-99fc-ab6edabdb618" containerID="5e10198edaf82ada593d1cb4d553c4c29b003db29a7fceb902065d746954cccc" exitCode=0
Sep 11 14:46:33.798448 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:46:33.798172    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f0ee6c/ipp-route-f0ee6c-kserve-5ddd6d97c5-5b4lt"
Sep 11 14:46:33.798448 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:46:33.798210    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f0ee6c/ipp-route-f0ee6c-kserve-5ddd6d97c5-5b4lt" event={"ID":"0e633a8c-9e7b-48ac-99fc-ab6edabdb618","Type":"ContainerDied","Data":"5e10198edaf82ada593d1cb4d553c4c29b003db29a7fceb902065d746954cccc"}
Sep 11 14:46:33.798448 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:46:33.798254    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f0ee6c/ipp-route-f0ee6c-kserve-5ddd6d97c5-5b4lt" event={"ID":"0e633a8c-9e7b-48ac-99fc-ab6edabdb618","Type":"ContainerDied","Data":"874d0a60a87cac9048fa7c709144ed1320e2a698b70d13141b5396a88c93662e"}
Sep 11 14:46:33.798448 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:46:33.798271    2581 scope.go:117] "RemoveContainer" containerID="5e10198edaf82ada593d1cb4d553c4c29b003db29a7fceb902065d746954cccc"
Sep 11 14:46:33.807130 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:46:33.807110    2581 scope.go:117] "RemoveContainer" containerID="5e10198edaf82ada593d1cb4d553c4c29b003db29a7fceb902065d746954cccc"
Sep 11 14:46:33.807419 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:46:33.807398    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e10198edaf82ada593d1cb4d553c4c29b003db29a7fceb902065d746954cccc\": container with ID starting with 5e10198edaf82ada593d1cb4d553c4c29b003db29a7fceb902065d746954cccc not found: ID does not exist" containerID="5e10198edaf82ada593d1cb4d553c4c29b003db29a7fceb902065d746954cccc"
Sep 11 14:46:33.807535 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:46:33.807426    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e10198edaf82ada593d1cb4d553c4c29b003db29a7fceb902065d746954cccc"} err="failed to get container status \"5e10198edaf82ada593d1cb4d553c4c29b003db29a7fceb902065d746954cccc\": rpc error: code = NotFound desc = could not find container \"5e10198edaf82ada593d1cb4d553c4c29b003db29a7fceb902065d746954cccc\": container with ID starting with 5e10198edaf82ada593d1cb4d553c4c29b003db29a7fceb902065d746954cccc not found: ID does not exist"
Sep 11 14:46:33.826839 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:46:33.826814    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-f0ee6c/ipp-route-f0ee6c-kserve-5ddd6d97c5-5b4lt"]
Sep 11 14:46:33.838304 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:46:33.838280    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-f0ee6c/ipp-route-f0ee6c-kserve-5ddd6d97c5-5b4lt"]
Sep 11 14:46:35.506160 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:46:35.506125    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e633a8c-9e7b-48ac-99fc-ab6edabdb618" path="/var/lib/kubelet/pods/0e633a8c-9e7b-48ac-99fc-ab6edabdb618/volumes"
Sep 11 14:47:03.249354 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:03.249317    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8"]
Sep 11 14:47:03.249924 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:03.249657    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8" podUID="8d6949fd-9353-442f-8feb-64d16d9eeb28" containerName="istio-proxy" containerID="cri-o://4ae800ac0af29cc938a05358918e6277a9987a93583acc0fa2a8fc887222559c" gracePeriod=30
Sep 11 14:47:03.695661 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:03.695570    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-40d3a2-ldgv8"]
Sep 11 14:47:03.696066 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:03.696046    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c9f9d8d-1224-472a-bcfc-0ded6303bba4" containerName="revoke-keys"
Sep 11 14:47:03.696168 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:03.696067    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c9f9d8d-1224-472a-bcfc-0ded6303bba4" containerName="revoke-keys"
Sep 11 14:47:03.696168 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:03.696085    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e633a8c-9e7b-48ac-99fc-ab6edabdb618" containerName="main"
Sep 11 14:47:03.696168 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:03.696105    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e633a8c-9e7b-48ac-99fc-ab6edabdb618" containerName="main"
Sep 11 14:47:03.696329 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:03.696231    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e633a8c-9e7b-48ac-99fc-ab6edabdb618" containerName="main"
Sep 11 14:47:03.696329 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:03.696258    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c9f9d8d-1224-472a-bcfc-0ded6303bba4" containerName="revoke-keys"
Sep 11 14:47:03.699308 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:03.699288    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-40d3a2-ldgv8"
Sep 11 14:47:03.705434 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:03.705400    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 14:47:03.705576 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:03.705443    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hcm4g\""
Sep 11 14:47:03.717123 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:03.717092    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-40d3a2-ldgv8"]
Sep 11 14:47:03.806259 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:03.806223    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c4b017d-3dd2-445f-9499-6d5f5b018630-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-40d3a2-ldgv8\" (UID: \"1c4b017d-3dd2-445f-9499-6d5f5b018630\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-40d3a2-ldgv8"
Sep 11 14:47:03.907792 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:03.907751    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c4b017d-3dd2-445f-9499-6d5f5b018630-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-40d3a2-ldgv8\" (UID: \"1c4b017d-3dd2-445f-9499-6d5f5b018630\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-40d3a2-ldgv8"
Sep 11 14:47:03.908415 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:03.908394    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c4b017d-3dd2-445f-9499-6d5f5b018630-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-40d3a2-ldgv8\" (UID: \"1c4b017d-3dd2-445f-9499-6d5f5b018630\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-40d3a2-ldgv8"
Sep 11 14:47:04.008810 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:04.008772    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-40d3a2-ldgv8"
Sep 11 14:47:04.288467 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:04.288394    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-40d3a2-ldgv8"]
Sep 11 14:47:04.291049 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:47:04.291020    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c4b017d_3dd2_445f_9499_6d5f5b018630.slice/crio-c5406045c50a5379722f7fb906536cc956a0c0da71be60adbf3740319c925f13 WatchSource:0}: Error finding container c5406045c50a5379722f7fb906536cc956a0c0da71be60adbf3740319c925f13: Status 404 returned error can't find the container with id c5406045c50a5379722f7fb906536cc956a0c0da71be60adbf3740319c925f13
Sep 11 14:47:04.973230 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:04.973193    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-40d3a2-ldgv8" event={"ID":"1c4b017d-3dd2-445f-9499-6d5f5b018630","Type":"ContainerStarted","Data":"a6b550f26678a67b7d1395ecf07d554cf6d63e4e11c6be35a7c7d294c54ec3d6"}
Sep 11 14:47:04.973463 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:04.973254    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-40d3a2-ldgv8" event={"ID":"1c4b017d-3dd2-445f-9499-6d5f5b018630","Type":"ContainerStarted","Data":"c5406045c50a5379722f7fb906536cc956a0c0da71be60adbf3740319c925f13"}
Sep 11 14:47:05.000713 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:05.000661    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-40d3a2-ldgv8" podStartSLOduration=2.000642702 podStartE2EDuration="2.000642702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:47:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:47:04.999231347 +0000 UTC m=+1928.149781413" watchObservedRunningTime="2026-09-11 14:47:05.000642702 +0000 UTC m=+1928.151192766"
Sep 11 14:47:05.888812 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:05.888774    2581 patch_prober.go:28] interesting pod/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 14:47:05.889304 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:05.888838    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8" podUID="8d6949fd-9353-442f-8feb-64d16d9eeb28" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 14:47:05.980196 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:05.980154    2581 generic.go:356] "Generic (PLEG): container finished" podID="1c4b017d-3dd2-445f-9499-6d5f5b018630" containerID="a6b550f26678a67b7d1395ecf07d554cf6d63e4e11c6be35a7c7d294c54ec3d6" exitCode=0
Sep 11 14:47:05.980196 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:05.980197    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-40d3a2-ldgv8" event={"ID":"1c4b017d-3dd2-445f-9499-6d5f5b018630","Type":"ContainerDied","Data":"a6b550f26678a67b7d1395ecf07d554cf6d63e4e11c6be35a7c7d294c54ec3d6"}
Sep 11 14:47:07.118644 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:07.118617    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-40d3a2-ldgv8"
Sep 11 14:47:07.240720 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:07.240692    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c4b017d-3dd2-445f-9499-6d5f5b018630-maas-api-service-ca\") pod \"1c4b017d-3dd2-445f-9499-6d5f5b018630\" (UID: \"1c4b017d-3dd2-445f-9499-6d5f5b018630\") "
Sep 11 14:47:07.241103 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:07.241080    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c4b017d-3dd2-445f-9499-6d5f5b018630-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1c4b017d-3dd2-445f-9499-6d5f5b018630" (UID: "1c4b017d-3dd2-445f-9499-6d5f5b018630"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:47:07.341837 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:07.341799    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c4b017d-3dd2-445f-9499-6d5f5b018630-maas-api-service-ca\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:47:07.991584 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:07.991545    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-40d3a2-ldgv8" event={"ID":"1c4b017d-3dd2-445f-9499-6d5f5b018630","Type":"ContainerDied","Data":"c5406045c50a5379722f7fb906536cc956a0c0da71be60adbf3740319c925f13"}
Sep 11 14:47:07.991584 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:07.991577    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-40d3a2-ldgv8"
Sep 11 14:47:07.991787 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:07.991607    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5406045c50a5379722f7fb906536cc956a0c0da71be60adbf3740319c925f13"
Sep 11 14:47:08.498891 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:08.498868    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8"
Sep 11 14:47:08.652997 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:08.652883    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d6949fd-9353-442f-8feb-64d16d9eeb28-credential-socket\") pod \"8d6949fd-9353-442f-8feb-64d16d9eeb28\" (UID: \"8d6949fd-9353-442f-8feb-64d16d9eeb28\") "
Sep 11 14:47:08.652997 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:08.652956    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d6949fd-9353-442f-8feb-64d16d9eeb28-istio-token\") pod \"8d6949fd-9353-442f-8feb-64d16d9eeb28\" (UID: \"8d6949fd-9353-442f-8feb-64d16d9eeb28\") "
Sep 11 14:47:08.652997 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:08.653003    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d6949fd-9353-442f-8feb-64d16d9eeb28-istiod-ca-cert\") pod \"8d6949fd-9353-442f-8feb-64d16d9eeb28\" (UID: \"8d6949fd-9353-442f-8feb-64d16d9eeb28\") "
Sep 11 14:47:08.653258 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:08.653021    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d6949fd-9353-442f-8feb-64d16d9eeb28-istio-podinfo\") pod \"8d6949fd-9353-442f-8feb-64d16d9eeb28\" (UID: \"8d6949fd-9353-442f-8feb-64d16d9eeb28\") "
Sep 11 14:47:08.653258 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:08.653043    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d6949fd-9353-442f-8feb-64d16d9eeb28-istio-data\") pod \"8d6949fd-9353-442f-8feb-64d16d9eeb28\" (UID: \"8d6949fd-9353-442f-8feb-64d16d9eeb28\") "
Sep 11 14:47:08.653258 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:08.653084    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d6949fd-9353-442f-8feb-64d16d9eeb28-workload-socket\") pod \"8d6949fd-9353-442f-8feb-64d16d9eeb28\" (UID: \"8d6949fd-9353-442f-8feb-64d16d9eeb28\") "
Sep 11 14:47:08.653258 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:08.653102    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d6949fd-9353-442f-8feb-64d16d9eeb28-istio-envoy\") pod \"8d6949fd-9353-442f-8feb-64d16d9eeb28\" (UID: \"8d6949fd-9353-442f-8feb-64d16d9eeb28\") "
Sep 11 14:47:08.653258 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:08.653157    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d6949fd-9353-442f-8feb-64d16d9eeb28-workload-certs\") pod \"8d6949fd-9353-442f-8feb-64d16d9eeb28\" (UID: \"8d6949fd-9353-442f-8feb-64d16d9eeb28\") "
Sep 11 14:47:08.653258 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:08.653234    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d6949fd-9353-442f-8feb-64d16d9eeb28-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8d6949fd-9353-442f-8feb-64d16d9eeb28" (UID: "8d6949fd-9353-442f-8feb-64d16d9eeb28"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:47:08.653258 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:08.653249    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46n9v\" (UniqueName: \"kubernetes.io/projected/8d6949fd-9353-442f-8feb-64d16d9eeb28-kube-api-access-46n9v\") pod \"8d6949fd-9353-442f-8feb-64d16d9eeb28\" (UID: \"8d6949fd-9353-442f-8feb-64d16d9eeb28\") "
Sep 11 14:47:08.653560 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:08.653364    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d6949fd-9353-442f-8feb-64d16d9eeb28-istio-data" (OuterVolumeSpecName: "istio-data") pod "8d6949fd-9353-442f-8feb-64d16d9eeb28" (UID: "8d6949fd-9353-442f-8feb-64d16d9eeb28"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:47:08.653560 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:08.653375    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d6949fd-9353-442f-8feb-64d16d9eeb28-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8d6949fd-9353-442f-8feb-64d16d9eeb28" (UID: "8d6949fd-9353-442f-8feb-64d16d9eeb28"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:47:08.653560 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:08.653465    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d6949fd-9353-442f-8feb-64d16d9eeb28-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8d6949fd-9353-442f-8feb-64d16d9eeb28" (UID: "8d6949fd-9353-442f-8feb-64d16d9eeb28"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:47:08.653721 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:08.653581    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d6949fd-9353-442f-8feb-64d16d9eeb28-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8d6949fd-9353-442f-8feb-64d16d9eeb28" (UID: "8d6949fd-9353-442f-8feb-64d16d9eeb28"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:47:08.653721 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:08.653711    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d6949fd-9353-442f-8feb-64d16d9eeb28-credential-socket\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:47:08.653828 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:08.653734    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d6949fd-9353-442f-8feb-64d16d9eeb28-istiod-ca-cert\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:47:08.653828 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:08.653749    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d6949fd-9353-442f-8feb-64d16d9eeb28-istio-data\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:47:08.653828 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:08.653761    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d6949fd-9353-442f-8feb-64d16d9eeb28-workload-socket\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:47:08.653828 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:08.653774    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d6949fd-9353-442f-8feb-64d16d9eeb28-workload-certs\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:47:08.655321 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:08.655287    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d6949fd-9353-442f-8feb-64d16d9eeb28-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8d6949fd-9353-442f-8feb-64d16d9eeb28" (UID: "8d6949fd-9353-442f-8feb-64d16d9eeb28"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:47:08.655453 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:08.655370    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d6949fd-9353-442f-8feb-64d16d9eeb28-kube-api-access-46n9v" (OuterVolumeSpecName: "kube-api-access-46n9v") pod "8d6949fd-9353-442f-8feb-64d16d9eeb28" (UID: "8d6949fd-9353-442f-8feb-64d16d9eeb28"). InnerVolumeSpecName "kube-api-access-46n9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:47:08.655453 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:08.655394    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d6949fd-9353-442f-8feb-64d16d9eeb28-istio-token" (OuterVolumeSpecName: "istio-token") pod "8d6949fd-9353-442f-8feb-64d16d9eeb28" (UID: "8d6949fd-9353-442f-8feb-64d16d9eeb28"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:47:08.655546 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:08.655520    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d6949fd-9353-442f-8feb-64d16d9eeb28-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8d6949fd-9353-442f-8feb-64d16d9eeb28" (UID: "8d6949fd-9353-442f-8feb-64d16d9eeb28"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 14:47:08.755077 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:08.755043    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d6949fd-9353-442f-8feb-64d16d9eeb28-istio-podinfo\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:47:08.755077 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:08.755070    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d6949fd-9353-442f-8feb-64d16d9eeb28-istio-envoy\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:47:08.755077 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:08.755079    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46n9v\" (UniqueName: \"kubernetes.io/projected/8d6949fd-9353-442f-8feb-64d16d9eeb28-kube-api-access-46n9v\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:47:08.755320 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:08.755089    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d6949fd-9353-442f-8feb-64d16d9eeb28-istio-token\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:47:08.998852 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:08.998818    2581 generic.go:356] "Generic (PLEG): container finished" podID="8d6949fd-9353-442f-8feb-64d16d9eeb28" containerID="4ae800ac0af29cc938a05358918e6277a9987a93583acc0fa2a8fc887222559c" exitCode=0
Sep 11 14:47:08.999053 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:08.998868    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8"
Sep 11 14:47:08.999053 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:08.998899    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8" event={"ID":"8d6949fd-9353-442f-8feb-64d16d9eeb28","Type":"ContainerDied","Data":"4ae800ac0af29cc938a05358918e6277a9987a93583acc0fa2a8fc887222559c"}
Sep 11 14:47:08.999053 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:08.998952    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8" event={"ID":"8d6949fd-9353-442f-8feb-64d16d9eeb28","Type":"ContainerDied","Data":"b235669eb5e591f14e6b31a880ea2c646639f7ed0145b3fb3756b1a1d6bd29b1"}
Sep 11 14:47:08.999053 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:08.999001    2581 scope.go:117] "RemoveContainer" containerID="4ae800ac0af29cc938a05358918e6277a9987a93583acc0fa2a8fc887222559c"
Sep 11 14:47:09.008653 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:09.008631    2581 scope.go:117] "RemoveContainer" containerID="b25863dc77d6e57c5bb3c4362d0c1730fb2df3474767d2cec0845b2dbc31bdb1"
Sep 11 14:47:09.017000 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:09.016960    2581 scope.go:117] "RemoveContainer" containerID="4ae800ac0af29cc938a05358918e6277a9987a93583acc0fa2a8fc887222559c"
Sep 11 14:47:09.017270 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:47:09.017252    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ae800ac0af29cc938a05358918e6277a9987a93583acc0fa2a8fc887222559c\": container with ID starting with 4ae800ac0af29cc938a05358918e6277a9987a93583acc0fa2a8fc887222559c not found: ID does not exist" containerID="4ae800ac0af29cc938a05358918e6277a9987a93583acc0fa2a8fc887222559c"
Sep 11 14:47:09.017323 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:09.017279    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ae800ac0af29cc938a05358918e6277a9987a93583acc0fa2a8fc887222559c"} err="failed to get container status \"4ae800ac0af29cc938a05358918e6277a9987a93583acc0fa2a8fc887222559c\": rpc error: code = NotFound desc = could not find container \"4ae800ac0af29cc938a05358918e6277a9987a93583acc0fa2a8fc887222559c\": container with ID starting with 4ae800ac0af29cc938a05358918e6277a9987a93583acc0fa2a8fc887222559c not found: ID does not exist"
Sep 11 14:47:09.017323 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:09.017297    2581 scope.go:117] "RemoveContainer" containerID="b25863dc77d6e57c5bb3c4362d0c1730fb2df3474767d2cec0845b2dbc31bdb1"
Sep 11 14:47:09.017554 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:47:09.017534    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b25863dc77d6e57c5bb3c4362d0c1730fb2df3474767d2cec0845b2dbc31bdb1\": container with ID starting with b25863dc77d6e57c5bb3c4362d0c1730fb2df3474767d2cec0845b2dbc31bdb1 not found: ID does not exist" containerID="b25863dc77d6e57c5bb3c4362d0c1730fb2df3474767d2cec0845b2dbc31bdb1"
Sep 11 14:47:09.017599 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:09.017560    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b25863dc77d6e57c5bb3c4362d0c1730fb2df3474767d2cec0845b2dbc31bdb1"} err="failed to get container status \"b25863dc77d6e57c5bb3c4362d0c1730fb2df3474767d2cec0845b2dbc31bdb1\": rpc error: code = NotFound desc = could not find container \"b25863dc77d6e57c5bb3c4362d0c1730fb2df3474767d2cec0845b2dbc31bdb1\": container with ID starting with b25863dc77d6e57c5bb3c4362d0c1730fb2df3474767d2cec0845b2dbc31bdb1 not found: ID does not exist"
Sep 11 14:47:09.038782 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:09.038756    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8"]
Sep 11 14:47:09.056512 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:09.056486    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f0ee6c-openshift-default-699c9457c6-bwpc8"]
Sep 11 14:47:09.499581 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:09.499546    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d6949fd-9353-442f-8feb-64d16d9eeb28" path="/var/lib/kubelet/pods/8d6949fd-9353-442f-8feb-64d16d9eeb28/volumes"
Sep 11 14:47:14.792710 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:14.792675    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-40d3a2-bdfd6f867-2jd4r"]
Sep 11 14:47:14.793116 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:14.792892    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-40d3a2-bdfd6f867-2jd4r" podUID="966ef1cb-e285-4d48-a36b-a2edcf3591ad" containerName="payload-processing" containerID="cri-o://2a930c72f2051cefb6382dd5616a04f6ea3714d8ca83465b754084dadf3fd054" gracePeriod=30
Sep 11 14:47:14.809721 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:14.809666    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-40d3a2-bbbb765db-mp5mp"]
Sep 11 14:47:14.810022 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:14.809948    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-40d3a2-bbbb765db-mp5mp" podUID="89e8c830-ad17-499b-9b8f-e7f8256e4402" containerName="payload-pre-processing" containerID="cri-o://22b65281056eec7dab7b48ec9d7368e2873e0b2bc1940222454fcdd1573989d0" gracePeriod=30
Sep 11 14:47:15.039716 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:15.039669    2581 generic.go:356] "Generic (PLEG): container finished" podID="966ef1cb-e285-4d48-a36b-a2edcf3591ad" containerID="2a930c72f2051cefb6382dd5616a04f6ea3714d8ca83465b754084dadf3fd054" exitCode=0
Sep 11 14:47:15.039895 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:15.039746    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-40d3a2-bdfd6f867-2jd4r" event={"ID":"966ef1cb-e285-4d48-a36b-a2edcf3591ad","Type":"ContainerDied","Data":"2a930c72f2051cefb6382dd5616a04f6ea3714d8ca83465b754084dadf3fd054"}
Sep 11 14:47:15.041851 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:15.041824    2581 generic.go:356] "Generic (PLEG): container finished" podID="89e8c830-ad17-499b-9b8f-e7f8256e4402" containerID="22b65281056eec7dab7b48ec9d7368e2873e0b2bc1940222454fcdd1573989d0" exitCode=0
Sep 11 14:47:15.041997 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:15.041899    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-40d3a2-bbbb765db-mp5mp" event={"ID":"89e8c830-ad17-499b-9b8f-e7f8256e4402","Type":"ContainerDied","Data":"22b65281056eec7dab7b48ec9d7368e2873e0b2bc1940222454fcdd1573989d0"}
Sep 11 14:47:15.055315 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:15.055125    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-40d3a2-ldgv8"]
Sep 11 14:47:15.057788 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:15.057768    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-40d3a2-ldgv8"]
Sep 11 14:47:15.088436 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:15.088417    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-40d3a2-bdfd6f867-2jd4r"
Sep 11 14:47:15.091485 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:15.091459    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-40d3a2-bbbb765db-mp5mp"
Sep 11 14:47:15.213888 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:15.213850    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89e8c830-ad17-499b-9b8f-e7f8256e4402-plugins-config-volume\") pod \"89e8c830-ad17-499b-9b8f-e7f8256e4402\" (UID: \"89e8c830-ad17-499b-9b8f-e7f8256e4402\") "
Sep 11 14:47:15.214129 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:15.213955    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwrks\" (UniqueName: \"kubernetes.io/projected/966ef1cb-e285-4d48-a36b-a2edcf3591ad-kube-api-access-mwrks\") pod \"966ef1cb-e285-4d48-a36b-a2edcf3591ad\" (UID: \"966ef1cb-e285-4d48-a36b-a2edcf3591ad\") "
Sep 11 14:47:15.214129 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:15.214028    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/966ef1cb-e285-4d48-a36b-a2edcf3591ad-plugins-config-volume\") pod \"966ef1cb-e285-4d48-a36b-a2edcf3591ad\" (UID: \"966ef1cb-e285-4d48-a36b-a2edcf3591ad\") "
Sep 11 14:47:15.214129 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:15.214046    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dpn2x\" (UniqueName: \"kubernetes.io/projected/89e8c830-ad17-499b-9b8f-e7f8256e4402-kube-api-access-dpn2x\") pod \"89e8c830-ad17-499b-9b8f-e7f8256e4402\" (UID: \"89e8c830-ad17-499b-9b8f-e7f8256e4402\") "
Sep 11 14:47:15.214355 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:15.214328    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89e8c830-ad17-499b-9b8f-e7f8256e4402-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "89e8c830-ad17-499b-9b8f-e7f8256e4402" (UID: "89e8c830-ad17-499b-9b8f-e7f8256e4402"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:47:15.214439 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:15.214411    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/966ef1cb-e285-4d48-a36b-a2edcf3591ad-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "966ef1cb-e285-4d48-a36b-a2edcf3591ad" (UID: "966ef1cb-e285-4d48-a36b-a2edcf3591ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:47:15.216125 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:15.216106    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/966ef1cb-e285-4d48-a36b-a2edcf3591ad-kube-api-access-mwrks" (OuterVolumeSpecName: "kube-api-access-mwrks") pod "966ef1cb-e285-4d48-a36b-a2edcf3591ad" (UID: "966ef1cb-e285-4d48-a36b-a2edcf3591ad"). InnerVolumeSpecName "kube-api-access-mwrks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:47:15.216230 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:15.216210    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89e8c830-ad17-499b-9b8f-e7f8256e4402-kube-api-access-dpn2x" (OuterVolumeSpecName: "kube-api-access-dpn2x") pod "89e8c830-ad17-499b-9b8f-e7f8256e4402" (UID: "89e8c830-ad17-499b-9b8f-e7f8256e4402"). InnerVolumeSpecName "kube-api-access-dpn2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:47:15.315708 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:15.315625    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89e8c830-ad17-499b-9b8f-e7f8256e4402-plugins-config-volume\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:47:15.315708 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:15.315655    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwrks\" (UniqueName: \"kubernetes.io/projected/966ef1cb-e285-4d48-a36b-a2edcf3591ad-kube-api-access-mwrks\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:47:15.315708 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:15.315664    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/966ef1cb-e285-4d48-a36b-a2edcf3591ad-plugins-config-volume\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:47:15.315708 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:15.315673    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpn2x\" (UniqueName: \"kubernetes.io/projected/89e8c830-ad17-499b-9b8f-e7f8256e4402-kube-api-access-dpn2x\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:47:15.502389 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:15.502351    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c4b017d-3dd2-445f-9499-6d5f5b018630" path="/var/lib/kubelet/pods/1c4b017d-3dd2-445f-9499-6d5f5b018630/volumes"
Sep 11 14:47:16.049987 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:16.049945    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-40d3a2-bdfd6f867-2jd4r"
Sep 11 14:47:16.049987 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:16.049980    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-40d3a2-bdfd6f867-2jd4r" event={"ID":"966ef1cb-e285-4d48-a36b-a2edcf3591ad","Type":"ContainerDied","Data":"57c4b6ef5a2da6fcb69d8e236dc2b8ed728c9868d2912083217c0d7e80d34cb7"}
Sep 11 14:47:16.050527 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:16.050048    2581 scope.go:117] "RemoveContainer" containerID="2a930c72f2051cefb6382dd5616a04f6ea3714d8ca83465b754084dadf3fd054"
Sep 11 14:47:16.051535 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:16.051510    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-40d3a2-bbbb765db-mp5mp" event={"ID":"89e8c830-ad17-499b-9b8f-e7f8256e4402","Type":"ContainerDied","Data":"0e179427428a0b7f413e37d2cf65b3fd24c03f23a88ccf3d55bd46cd0b816748"}
Sep 11 14:47:16.051620 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:16.051535    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-40d3a2-bbbb765db-mp5mp"
Sep 11 14:47:16.059011 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:16.058994    2581 scope.go:117] "RemoveContainer" containerID="22b65281056eec7dab7b48ec9d7368e2873e0b2bc1940222454fcdd1573989d0"
Sep 11 14:47:16.086677 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:16.086646    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-40d3a2-bbbb765db-mp5mp"]
Sep 11 14:47:16.088810 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:16.088786    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-40d3a2-bbbb765db-mp5mp"]
Sep 11 14:47:16.107444 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:16.107413    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-40d3a2-bdfd6f867-2jd4r"]
Sep 11 14:47:16.114777 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:16.114753    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-40d3a2-bdfd6f867-2jd4r"]
Sep 11 14:47:17.499797 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:17.499754    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89e8c830-ad17-499b-9b8f-e7f8256e4402" path="/var/lib/kubelet/pods/89e8c830-ad17-499b-9b8f-e7f8256e4402/volumes"
Sep 11 14:47:17.500388 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:17.500366    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="966ef1cb-e285-4d48-a36b-a2edcf3591ad" path="/var/lib/kubelet/pods/966ef1cb-e285-4d48-a36b-a2edcf3591ad/volumes"
Sep 11 14:47:22.865356 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:22.865319    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts"]
Sep 11 14:47:22.865790 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:22.865522    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts" podUID="4c69a781-1386-4047-979a-8eeb4e0709df" containerName="istio-proxy" containerID="cri-o://f7540eea6a548d3df88c49282c79e9ca6450f2e5303a468285b4941a49815b62" gracePeriod=30
Sep 11 14:47:23.315540 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:23.315503    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-09b16c-wrkrh"]
Sep 11 14:47:23.315879 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:23.315867    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d6949fd-9353-442f-8feb-64d16d9eeb28" containerName="istio-proxy"
Sep 11 14:47:23.315929 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:23.315886    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d6949fd-9353-442f-8feb-64d16d9eeb28" containerName="istio-proxy"
Sep 11 14:47:23.315929 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:23.315894    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89e8c830-ad17-499b-9b8f-e7f8256e4402" containerName="payload-pre-processing"
Sep 11 14:47:23.315929 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:23.315899    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="89e8c830-ad17-499b-9b8f-e7f8256e4402" containerName="payload-pre-processing"
Sep 11 14:47:23.315929 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:23.315909    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="966ef1cb-e285-4d48-a36b-a2edcf3591ad" containerName="payload-processing"
Sep 11 14:47:23.315929 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:23.315914    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="966ef1cb-e285-4d48-a36b-a2edcf3591ad" containerName="payload-processing"
Sep 11 14:47:23.315929 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:23.315923    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c4b017d-3dd2-445f-9499-6d5f5b018630" containerName="revoke-keys"
Sep 11 14:47:23.315929 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:23.315929    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c4b017d-3dd2-445f-9499-6d5f5b018630" containerName="revoke-keys"
Sep 11 14:47:23.316185 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:23.316008    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d6949fd-9353-442f-8feb-64d16d9eeb28" containerName="istio-proxy"
Sep 11 14:47:23.316185 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:23.316018    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="89e8c830-ad17-499b-9b8f-e7f8256e4402" containerName="payload-pre-processing"
Sep 11 14:47:23.316185 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:23.316027    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="966ef1cb-e285-4d48-a36b-a2edcf3591ad" containerName="payload-processing"
Sep 11 14:47:23.316185 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:23.316033    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c4b017d-3dd2-445f-9499-6d5f5b018630" containerName="revoke-keys"
Sep 11 14:47:23.320640 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:23.320620    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-09b16c-wrkrh"
Sep 11 14:47:23.323693 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:23.323671    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 14:47:23.323787 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:23.323671    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hcm4g\""
Sep 11 14:47:23.330643 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:23.330607    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-09b16c-wrkrh"]
Sep 11 14:47:23.393767 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:23.393728    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e9d53ad-e6b1-4afa-ac08-4d0097ab628e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-09b16c-wrkrh\" (UID: \"2e9d53ad-e6b1-4afa-ac08-4d0097ab628e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-09b16c-wrkrh"
Sep 11 14:47:23.495195 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:23.495151    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e9d53ad-e6b1-4afa-ac08-4d0097ab628e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-09b16c-wrkrh\" (UID: \"2e9d53ad-e6b1-4afa-ac08-4d0097ab628e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-09b16c-wrkrh"
Sep 11 14:47:23.495793 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:23.495771    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e9d53ad-e6b1-4afa-ac08-4d0097ab628e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-09b16c-wrkrh\" (UID: \"2e9d53ad-e6b1-4afa-ac08-4d0097ab628e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-09b16c-wrkrh"
Sep 11 14:47:23.631381 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:23.631294    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-09b16c-wrkrh"
Sep 11 14:47:23.909593 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:23.909414    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-09b16c-wrkrh"]
Sep 11 14:47:23.912187 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:47:23.912156    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e9d53ad_e6b1_4afa_ac08_4d0097ab628e.slice/crio-994be21b62d272eba3cd4f344cba41047ae9d385654d3d9366a8685634cbff22 WatchSource:0}: Error finding container 994be21b62d272eba3cd4f344cba41047ae9d385654d3d9366a8685634cbff22: Status 404 returned error can't find the container with id 994be21b62d272eba3cd4f344cba41047ae9d385654d3d9366a8685634cbff22
Sep 11 14:47:24.095665 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:24.095633    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-09b16c-wrkrh" event={"ID":"2e9d53ad-e6b1-4afa-ac08-4d0097ab628e","Type":"ContainerStarted","Data":"994be21b62d272eba3cd4f344cba41047ae9d385654d3d9366a8685634cbff22"}
Sep 11 14:47:25.102300 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:25.102264    2581 generic.go:356] "Generic (PLEG): container finished" podID="2e9d53ad-e6b1-4afa-ac08-4d0097ab628e" containerID="fd3c1a39752dca882af405f658acf36d09afce852b1080102f2e729567ce72c8" exitCode=0
Sep 11 14:47:25.102755 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:25.102357    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-09b16c-wrkrh" event={"ID":"2e9d53ad-e6b1-4afa-ac08-4d0097ab628e","Type":"ContainerDied","Data":"fd3c1a39752dca882af405f658acf36d09afce852b1080102f2e729567ce72c8"}
Sep 11 14:47:26.239483 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:26.239458    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-09b16c-wrkrh"
Sep 11 14:47:26.320761 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:26.320724    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e9d53ad-e6b1-4afa-ac08-4d0097ab628e-maas-api-service-ca\") pod \"2e9d53ad-e6b1-4afa-ac08-4d0097ab628e\" (UID: \"2e9d53ad-e6b1-4afa-ac08-4d0097ab628e\") "
Sep 11 14:47:26.321114 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:26.321091    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e9d53ad-e6b1-4afa-ac08-4d0097ab628e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2e9d53ad-e6b1-4afa-ac08-4d0097ab628e" (UID: "2e9d53ad-e6b1-4afa-ac08-4d0097ab628e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:47:26.422028 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:26.421910    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e9d53ad-e6b1-4afa-ac08-4d0097ab628e-maas-api-service-ca\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:47:27.115015 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:27.114955    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-09b16c-wrkrh"
Sep 11 14:47:27.115015 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:27.114954    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-09b16c-wrkrh" event={"ID":"2e9d53ad-e6b1-4afa-ac08-4d0097ab628e","Type":"ContainerDied","Data":"994be21b62d272eba3cd4f344cba41047ae9d385654d3d9366a8685634cbff22"}
Sep 11 14:47:27.115252 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:27.115042    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="994be21b62d272eba3cd4f344cba41047ae9d385654d3d9366a8685634cbff22"
Sep 11 14:47:28.115011 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:28.114984    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts"
Sep 11 14:47:28.128580 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:28.128496    2581 generic.go:356] "Generic (PLEG): container finished" podID="4c69a781-1386-4047-979a-8eeb4e0709df" containerID="f7540eea6a548d3df88c49282c79e9ca6450f2e5303a468285b4941a49815b62" exitCode=0
Sep 11 14:47:28.128580 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:28.128552    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts"
Sep 11 14:47:28.128770 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:28.128581    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts" event={"ID":"4c69a781-1386-4047-979a-8eeb4e0709df","Type":"ContainerDied","Data":"f7540eea6a548d3df88c49282c79e9ca6450f2e5303a468285b4941a49815b62"}
Sep 11 14:47:28.128770 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:28.128675    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts" event={"ID":"4c69a781-1386-4047-979a-8eeb4e0709df","Type":"ContainerDied","Data":"e58ceaaace7741ff0a79b302a3c133aeec6121663623211c85652ab77fd92c79"}
Sep 11 14:47:28.128770 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:28.128707    2581 scope.go:117] "RemoveContainer" containerID="f7540eea6a548d3df88c49282c79e9ca6450f2e5303a468285b4941a49815b62"
Sep 11 14:47:28.139453 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:28.139427    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c69a781-1386-4047-979a-8eeb4e0709df-istio-data\") pod \"4c69a781-1386-4047-979a-8eeb4e0709df\" (UID: \"4c69a781-1386-4047-979a-8eeb4e0709df\") "
Sep 11 14:47:28.139599 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:28.139490    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c69a781-1386-4047-979a-8eeb4e0709df-workload-certs\") pod \"4c69a781-1386-4047-979a-8eeb4e0709df\" (UID: \"4c69a781-1386-4047-979a-8eeb4e0709df\") "
Sep 11 14:47:28.139599 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:28.139535    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-696jz\" (UniqueName: \"kubernetes.io/projected/4c69a781-1386-4047-979a-8eeb4e0709df-kube-api-access-696jz\") pod \"4c69a781-1386-4047-979a-8eeb4e0709df\" (UID: \"4c69a781-1386-4047-979a-8eeb4e0709df\") "
Sep 11 14:47:28.139599 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:28.139568    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c69a781-1386-4047-979a-8eeb4e0709df-istio-envoy\") pod \"4c69a781-1386-4047-979a-8eeb4e0709df\" (UID: \"4c69a781-1386-4047-979a-8eeb4e0709df\") "
Sep 11 14:47:28.139778 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:28.139603    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c69a781-1386-4047-979a-8eeb4e0709df-workload-socket\") pod \"4c69a781-1386-4047-979a-8eeb4e0709df\" (UID: \"4c69a781-1386-4047-979a-8eeb4e0709df\") "
Sep 11 14:47:28.139778 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:28.139634    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c69a781-1386-4047-979a-8eeb4e0709df-istio-podinfo\") pod \"4c69a781-1386-4047-979a-8eeb4e0709df\" (UID: \"4c69a781-1386-4047-979a-8eeb4e0709df\") "
Sep 11 14:47:28.139778 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:28.139725    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c69a781-1386-4047-979a-8eeb4e0709df-istio-data" (OuterVolumeSpecName: "istio-data") pod "4c69a781-1386-4047-979a-8eeb4e0709df" (UID: "4c69a781-1386-4047-979a-8eeb4e0709df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:47:28.139778 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:28.139735    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c69a781-1386-4047-979a-8eeb4e0709df-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4c69a781-1386-4047-979a-8eeb4e0709df" (UID: "4c69a781-1386-4047-979a-8eeb4e0709df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:47:28.140015 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:28.139785    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c69a781-1386-4047-979a-8eeb4e0709df-credential-socket\") pod \"4c69a781-1386-4047-979a-8eeb4e0709df\" (UID: \"4c69a781-1386-4047-979a-8eeb4e0709df\") "
Sep 11 14:47:28.140015 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:28.139826    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c69a781-1386-4047-979a-8eeb4e0709df-istiod-ca-cert\") pod \"4c69a781-1386-4047-979a-8eeb4e0709df\" (UID: \"4c69a781-1386-4047-979a-8eeb4e0709df\") "
Sep 11 14:47:28.140015 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:28.139858    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c69a781-1386-4047-979a-8eeb4e0709df-istio-token\") pod \"4c69a781-1386-4047-979a-8eeb4e0709df\" (UID: \"4c69a781-1386-4047-979a-8eeb4e0709df\") "
Sep 11 14:47:28.140015 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:28.139964    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c69a781-1386-4047-979a-8eeb4e0709df-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4c69a781-1386-4047-979a-8eeb4e0709df" (UID: "4c69a781-1386-4047-979a-8eeb4e0709df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:47:28.140250 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:28.140186    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c69a781-1386-4047-979a-8eeb4e0709df-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4c69a781-1386-4047-979a-8eeb4e0709df" (UID: "4c69a781-1386-4047-979a-8eeb4e0709df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:47:28.140332 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:28.140297    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c69a781-1386-4047-979a-8eeb4e0709df-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4c69a781-1386-4047-979a-8eeb4e0709df" (UID: "4c69a781-1386-4047-979a-8eeb4e0709df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:47:28.140399 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:28.140305    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c69a781-1386-4047-979a-8eeb4e0709df-istio-data\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:47:28.140399 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:28.140358    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c69a781-1386-4047-979a-8eeb4e0709df-workload-certs\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:47:28.140399 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:28.140375    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c69a781-1386-4047-979a-8eeb4e0709df-workload-socket\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:47:28.140399 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:28.140388    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c69a781-1386-4047-979a-8eeb4e0709df-credential-socket\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:47:28.141150 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:28.141124    2581 scope.go:117] "RemoveContainer" containerID="f084d93fc3a38b69d3a233a9842f0bc9d305503c1446405c96fa0648cb5436bb"
Sep 11 14:47:28.142020 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:28.141995    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c69a781-1386-4047-979a-8eeb4e0709df-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4c69a781-1386-4047-979a-8eeb4e0709df" (UID: "4c69a781-1386-4047-979a-8eeb4e0709df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:47:28.142117 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:28.142096    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c69a781-1386-4047-979a-8eeb4e0709df-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4c69a781-1386-4047-979a-8eeb4e0709df" (UID: "4c69a781-1386-4047-979a-8eeb4e0709df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 14:47:28.142189 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:28.142112    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c69a781-1386-4047-979a-8eeb4e0709df-kube-api-access-696jz" (OuterVolumeSpecName: "kube-api-access-696jz") pod "4c69a781-1386-4047-979a-8eeb4e0709df" (UID: "4c69a781-1386-4047-979a-8eeb4e0709df"). InnerVolumeSpecName "kube-api-access-696jz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:47:28.142367 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:28.142348    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c69a781-1386-4047-979a-8eeb4e0709df-istio-token" (OuterVolumeSpecName: "istio-token") pod "4c69a781-1386-4047-979a-8eeb4e0709df" (UID: "4c69a781-1386-4047-979a-8eeb4e0709df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:47:28.162002 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:28.161963    2581 scope.go:117] "RemoveContainer" containerID="f7540eea6a548d3df88c49282c79e9ca6450f2e5303a468285b4941a49815b62"
Sep 11 14:47:28.162357 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:47:28.162332    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7540eea6a548d3df88c49282c79e9ca6450f2e5303a468285b4941a49815b62\": container with ID starting with f7540eea6a548d3df88c49282c79e9ca6450f2e5303a468285b4941a49815b62 not found: ID does not exist" containerID="f7540eea6a548d3df88c49282c79e9ca6450f2e5303a468285b4941a49815b62"
Sep 11 14:47:28.162426 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:28.162369    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7540eea6a548d3df88c49282c79e9ca6450f2e5303a468285b4941a49815b62"} err="failed to get container status \"f7540eea6a548d3df88c49282c79e9ca6450f2e5303a468285b4941a49815b62\": rpc error: code = NotFound desc = could not find container \"f7540eea6a548d3df88c49282c79e9ca6450f2e5303a468285b4941a49815b62\": container with ID starting with f7540eea6a548d3df88c49282c79e9ca6450f2e5303a468285b4941a49815b62 not found: ID does not exist"
Sep 11 14:47:28.162426 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:28.162394    2581 scope.go:117] "RemoveContainer" containerID="f084d93fc3a38b69d3a233a9842f0bc9d305503c1446405c96fa0648cb5436bb"
Sep 11 14:47:28.162710 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:47:28.162685    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f084d93fc3a38b69d3a233a9842f0bc9d305503c1446405c96fa0648cb5436bb\": container with ID starting with f084d93fc3a38b69d3a233a9842f0bc9d305503c1446405c96fa0648cb5436bb not found: ID does not exist" containerID="f084d93fc3a38b69d3a233a9842f0bc9d305503c1446405c96fa0648cb5436bb"
Sep 11 14:47:28.162808 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:28.162715    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f084d93fc3a38b69d3a233a9842f0bc9d305503c1446405c96fa0648cb5436bb"} err="failed to get container status \"f084d93fc3a38b69d3a233a9842f0bc9d305503c1446405c96fa0648cb5436bb\": rpc error: code = NotFound desc = could not find container \"f084d93fc3a38b69d3a233a9842f0bc9d305503c1446405c96fa0648cb5436bb\": container with ID starting with f084d93fc3a38b69d3a233a9842f0bc9d305503c1446405c96fa0648cb5436bb not found: ID does not exist"
Sep 11 14:47:28.241671 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:28.241638    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c69a781-1386-4047-979a-8eeb4e0709df-istio-envoy\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:47:28.241671 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:28.241667    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c69a781-1386-4047-979a-8eeb4e0709df-istio-podinfo\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:47:28.241671 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:28.241677    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c69a781-1386-4047-979a-8eeb4e0709df-istiod-ca-cert\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:47:28.241913 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:28.241686    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c69a781-1386-4047-979a-8eeb4e0709df-istio-token\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:47:28.241913 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:28.241694    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-696jz\" (UniqueName: \"kubernetes.io/projected/4c69a781-1386-4047-979a-8eeb4e0709df-kube-api-access-696jz\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:47:28.460703 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:28.460669    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts"]
Sep 11 14:47:28.466705 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:28.466680    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-40d3a2-openshift-default-575fbdd968-fppts"]
Sep 11 14:47:29.499787 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:29.499753    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c69a781-1386-4047-979a-8eeb4e0709df" path="/var/lib/kubelet/pods/4c69a781-1386-4047-979a-8eeb4e0709df/volumes"
Sep 11 14:47:34.125473 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:34.125424    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-09b16c-29818965-mzt9j"]
Sep 11 14:47:34.129897 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:34.129870    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-09b16c-29818965-mzt9j"]
Sep 11 14:47:34.157227 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:34.157195    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-09b16c-57c884f8fc-8pm84"]
Sep 11 14:47:34.157801 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:34.157753    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-09b16c-57c884f8fc-8pm84" podUID="984b5347-e783-4113-96c3-49552ac4bdcc" containerName="payload-processing" containerID="cri-o://ff94a58c1cbd23839f2e247ffa4fa156a3eaf70d7edfd10c3b869f5bf354a920" gracePeriod=30
Sep 11 14:47:34.251980 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:34.251937    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-09b16c-57c884f8fc-8pm84" podUID="984b5347-e783-4113-96c3-49552ac4bdcc" containerName="payload-processing" probeResult="failure" output="dial tcp 10.132.0.53:9004: connect: connection refused"
Sep 11 14:47:34.408769 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:34.408745    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-09b16c-57c884f8fc-8pm84"
Sep 11 14:47:34.432117 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:34.432084    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-09b16c-wrkrh"]
Sep 11 14:47:34.442471 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:34.442446    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-09b16c-wrkrh"]
Sep 11 14:47:34.499951 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:34.499918    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwmx2\" (UniqueName: \"kubernetes.io/projected/984b5347-e783-4113-96c3-49552ac4bdcc-kube-api-access-rwmx2\") pod \"984b5347-e783-4113-96c3-49552ac4bdcc\" (UID: \"984b5347-e783-4113-96c3-49552ac4bdcc\") "
Sep 11 14:47:34.500170 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:34.499988    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/984b5347-e783-4113-96c3-49552ac4bdcc-plugins-config-volume\") pod \"984b5347-e783-4113-96c3-49552ac4bdcc\" (UID: \"984b5347-e783-4113-96c3-49552ac4bdcc\") "
Sep 11 14:47:34.500470 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:34.500441    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/984b5347-e783-4113-96c3-49552ac4bdcc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "984b5347-e783-4113-96c3-49552ac4bdcc" (UID: "984b5347-e783-4113-96c3-49552ac4bdcc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:47:34.502000 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:34.501964    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/984b5347-e783-4113-96c3-49552ac4bdcc-kube-api-access-rwmx2" (OuterVolumeSpecName: "kube-api-access-rwmx2") pod "984b5347-e783-4113-96c3-49552ac4bdcc" (UID: "984b5347-e783-4113-96c3-49552ac4bdcc"). InnerVolumeSpecName "kube-api-access-rwmx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:47:34.601381 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:34.601345    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/984b5347-e783-4113-96c3-49552ac4bdcc-plugins-config-volume\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:47:34.601381 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:34.601374    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwmx2\" (UniqueName: \"kubernetes.io/projected/984b5347-e783-4113-96c3-49552ac4bdcc-kube-api-access-rwmx2\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:47:35.184279 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:35.184241    2581 generic.go:356] "Generic (PLEG): container finished" podID="984b5347-e783-4113-96c3-49552ac4bdcc" containerID="ff94a58c1cbd23839f2e247ffa4fa156a3eaf70d7edfd10c3b869f5bf354a920" exitCode=0
Sep 11 14:47:35.184747 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:35.184289    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-09b16c-57c884f8fc-8pm84"
Sep 11 14:47:35.184747 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:35.184344    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-09b16c-57c884f8fc-8pm84" event={"ID":"984b5347-e783-4113-96c3-49552ac4bdcc","Type":"ContainerDied","Data":"ff94a58c1cbd23839f2e247ffa4fa156a3eaf70d7edfd10c3b869f5bf354a920"}
Sep 11 14:47:35.184747 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:35.184412    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-09b16c-57c884f8fc-8pm84" event={"ID":"984b5347-e783-4113-96c3-49552ac4bdcc","Type":"ContainerDied","Data":"7ca4cbe5a5d1f5f3154eb56374d65febc05e21b1a13285758f0e6fdb59863196"}
Sep 11 14:47:35.184747 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:35.184439    2581 scope.go:117] "RemoveContainer" containerID="ff94a58c1cbd23839f2e247ffa4fa156a3eaf70d7edfd10c3b869f5bf354a920"
Sep 11 14:47:35.193578 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:35.193557    2581 scope.go:117] "RemoveContainer" containerID="ff94a58c1cbd23839f2e247ffa4fa156a3eaf70d7edfd10c3b869f5bf354a920"
Sep 11 14:47:35.193849 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:47:35.193830    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff94a58c1cbd23839f2e247ffa4fa156a3eaf70d7edfd10c3b869f5bf354a920\": container with ID starting with ff94a58c1cbd23839f2e247ffa4fa156a3eaf70d7edfd10c3b869f5bf354a920 not found: ID does not exist" containerID="ff94a58c1cbd23839f2e247ffa4fa156a3eaf70d7edfd10c3b869f5bf354a920"
Sep 11 14:47:35.193894 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:35.193858    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff94a58c1cbd23839f2e247ffa4fa156a3eaf70d7edfd10c3b869f5bf354a920"} err="failed to get container status \"ff94a58c1cbd23839f2e247ffa4fa156a3eaf70d7edfd10c3b869f5bf354a920\": rpc error: code = NotFound desc = could not find container \"ff94a58c1cbd23839f2e247ffa4fa156a3eaf70d7edfd10c3b869f5bf354a920\": container with ID starting with ff94a58c1cbd23839f2e247ffa4fa156a3eaf70d7edfd10c3b869f5bf354a920 not found: ID does not exist"
Sep 11 14:47:35.213193 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:35.213165    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-09b16c-57c884f8fc-8pm84"]
Sep 11 14:47:35.227090 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:35.227061    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-09b16c-57c884f8fc-8pm84"]
Sep 11 14:47:35.499045 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:35.499006    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e9d53ad-e6b1-4afa-ac08-4d0097ab628e" path="/var/lib/kubelet/pods/2e9d53ad-e6b1-4afa-ac08-4d0097ab628e/volumes"
Sep 11 14:47:35.499486 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:35.499471    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="984b5347-e783-4113-96c3-49552ac4bdcc" path="/var/lib/kubelet/pods/984b5347-e783-4113-96c3-49552ac4bdcc/volumes"
Sep 11 14:47:35.499724 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:35.499712    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b65d9252-52a0-47c0-80e9-4d5473a46754" path="/var/lib/kubelet/pods/b65d9252-52a0-47c0-80e9-4d5473a46754/volumes"
Sep 11 14:47:41.951437 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:41.951402    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh"]
Sep 11 14:47:41.951851 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:41.951611    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh" podUID="88197d9f-d033-4403-aef8-dd65332427a9" containerName="istio-proxy" containerID="cri-o://a59c51c6a043bc3b5b9752833b854ffa4da631470ce2a213163173e114111445" gracePeriod=30
Sep 11 14:47:42.360721 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:42.360664    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1c284e-b4nlj"]
Sep 11 14:47:42.361051 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:42.361037    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e9d53ad-e6b1-4afa-ac08-4d0097ab628e" containerName="revoke-keys"
Sep 11 14:47:42.361101 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:42.361054    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e9d53ad-e6b1-4afa-ac08-4d0097ab628e" containerName="revoke-keys"
Sep 11 14:47:42.361101 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:42.361075    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="984b5347-e783-4113-96c3-49552ac4bdcc" containerName="payload-processing"
Sep 11 14:47:42.361101 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:42.361081    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="984b5347-e783-4113-96c3-49552ac4bdcc" containerName="payload-processing"
Sep 11 14:47:42.361101 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:42.361094    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c69a781-1386-4047-979a-8eeb4e0709df" containerName="istio-proxy"
Sep 11 14:47:42.361101 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:42.361100    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c69a781-1386-4047-979a-8eeb4e0709df" containerName="istio-proxy"
Sep 11 14:47:42.361256 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:42.361107    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d6949fd-9353-442f-8feb-64d16d9eeb28" containerName="istio-proxy"
Sep 11 14:47:42.361256 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:42.361112    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d6949fd-9353-442f-8feb-64d16d9eeb28" containerName="istio-proxy"
Sep 11 14:47:42.361256 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:42.361119    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c69a781-1386-4047-979a-8eeb4e0709df" containerName="istio-proxy"
Sep 11 14:47:42.361256 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:42.361125    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c69a781-1386-4047-979a-8eeb4e0709df" containerName="istio-proxy"
Sep 11 14:47:42.361256 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:42.361197    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d6949fd-9353-442f-8feb-64d16d9eeb28" containerName="istio-proxy"
Sep 11 14:47:42.361256 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:42.361204    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c69a781-1386-4047-979a-8eeb4e0709df" containerName="istio-proxy"
Sep 11 14:47:42.361256 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:42.361214    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c69a781-1386-4047-979a-8eeb4e0709df" containerName="istio-proxy"
Sep 11 14:47:42.361256 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:42.361224    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="984b5347-e783-4113-96c3-49552ac4bdcc" containerName="payload-processing"
Sep 11 14:47:42.361256 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:42.361234    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e9d53ad-e6b1-4afa-ac08-4d0097ab628e" containerName="revoke-keys"
Sep 11 14:47:42.365556 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:42.365537    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1c284e-b4nlj"
Sep 11 14:47:42.370324 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:42.370296    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 14:47:42.370324 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:42.370312    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hcm4g\""
Sep 11 14:47:42.380099 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:42.380068    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1c284e-b4nlj"]
Sep 11 14:47:42.478289 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:42.478249    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eee1dd32-5c4b-4b88-9656-78c1c2b93e2d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-1c284e-b4nlj\" (UID: \"eee1dd32-5c4b-4b88-9656-78c1c2b93e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1c284e-b4nlj"
Sep 11 14:47:42.578984 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:42.578938    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eee1dd32-5c4b-4b88-9656-78c1c2b93e2d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-1c284e-b4nlj\" (UID: \"eee1dd32-5c4b-4b88-9656-78c1c2b93e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1c284e-b4nlj"
Sep 11 14:47:42.579598 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:42.579572    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eee1dd32-5c4b-4b88-9656-78c1c2b93e2d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-1c284e-b4nlj\" (UID: \"eee1dd32-5c4b-4b88-9656-78c1c2b93e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1c284e-b4nlj"
Sep 11 14:47:42.675453 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:42.675355    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1c284e-b4nlj"
Sep 11 14:47:42.965372 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:42.965272    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1c284e-b4nlj"]
Sep 11 14:47:42.968525 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:47:42.968501    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeee1dd32_5c4b_4b88_9656_78c1c2b93e2d.slice/crio-a2566f210ae9441dd0a9ed0019adf9afa2aebcb2947a6cdf60e7ce2c8eb624a0 WatchSource:0}: Error finding container a2566f210ae9441dd0a9ed0019adf9afa2aebcb2947a6cdf60e7ce2c8eb624a0: Status 404 returned error can't find the container with id a2566f210ae9441dd0a9ed0019adf9afa2aebcb2947a6cdf60e7ce2c8eb624a0
Sep 11 14:47:43.230241 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:43.230213    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1c284e-b4nlj" event={"ID":"eee1dd32-5c4b-4b88-9656-78c1c2b93e2d","Type":"ContainerStarted","Data":"a2566f210ae9441dd0a9ed0019adf9afa2aebcb2947a6cdf60e7ce2c8eb624a0"}
Sep 11 14:47:44.237171 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:44.237134    2581 generic.go:356] "Generic (PLEG): container finished" podID="eee1dd32-5c4b-4b88-9656-78c1c2b93e2d" containerID="ae5679e4c1c00dc5d2be51890d214a340ba9366e755f288f33a5dfa8c32575c7" exitCode=0
Sep 11 14:47:44.237567 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:44.237218    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1c284e-b4nlj" event={"ID":"eee1dd32-5c4b-4b88-9656-78c1c2b93e2d","Type":"ContainerDied","Data":"ae5679e4c1c00dc5d2be51890d214a340ba9366e755f288f33a5dfa8c32575c7"}
Sep 11 14:47:45.370544 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:45.370520    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1c284e-b4nlj"
Sep 11 14:47:45.506875 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:45.506800    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eee1dd32-5c4b-4b88-9656-78c1c2b93e2d-maas-api-service-ca\") pod \"eee1dd32-5c4b-4b88-9656-78c1c2b93e2d\" (UID: \"eee1dd32-5c4b-4b88-9656-78c1c2b93e2d\") "
Sep 11 14:47:45.507132 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:45.507110    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eee1dd32-5c4b-4b88-9656-78c1c2b93e2d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "eee1dd32-5c4b-4b88-9656-78c1c2b93e2d" (UID: "eee1dd32-5c4b-4b88-9656-78c1c2b93e2d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:47:45.608293 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:45.608253    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eee1dd32-5c4b-4b88-9656-78c1c2b93e2d-maas-api-service-ca\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:47:46.249684 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:46.249655    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1c284e-b4nlj"
Sep 11 14:47:46.249889 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:46.249653    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1c284e-b4nlj" event={"ID":"eee1dd32-5c4b-4b88-9656-78c1c2b93e2d","Type":"ContainerDied","Data":"a2566f210ae9441dd0a9ed0019adf9afa2aebcb2947a6cdf60e7ce2c8eb624a0"}
Sep 11 14:47:46.249889 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:46.249784    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2566f210ae9441dd0a9ed0019adf9afa2aebcb2947a6cdf60e7ce2c8eb624a0"
Sep 11 14:47:47.201476 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:47.201449    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh"
Sep 11 14:47:47.256529 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:47.256491    2581 generic.go:356] "Generic (PLEG): container finished" podID="88197d9f-d033-4403-aef8-dd65332427a9" containerID="a59c51c6a043bc3b5b9752833b854ffa4da631470ce2a213163173e114111445" exitCode=0
Sep 11 14:47:47.256736 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:47.256543    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh"
Sep 11 14:47:47.256736 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:47.256585    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh" event={"ID":"88197d9f-d033-4403-aef8-dd65332427a9","Type":"ContainerDied","Data":"a59c51c6a043bc3b5b9752833b854ffa4da631470ce2a213163173e114111445"}
Sep 11 14:47:47.256736 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:47.256643    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh" event={"ID":"88197d9f-d033-4403-aef8-dd65332427a9","Type":"ContainerDied","Data":"fedb7e7769d81d9928802051994f0c0e3f9b3fcc4dcd7e576aa229cf3f29780b"}
Sep 11 14:47:47.256736 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:47.256666    2581 scope.go:117] "RemoveContainer" containerID="a59c51c6a043bc3b5b9752833b854ffa4da631470ce2a213163173e114111445"
Sep 11 14:47:47.265344 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:47.265325    2581 scope.go:117] "RemoveContainer" containerID="f8e71261cd31bb95cee7abb65ad46a5893bb12f629044fc8971098b24a223e42"
Sep 11 14:47:47.273601 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:47.273583    2581 scope.go:117] "RemoveContainer" containerID="a59c51c6a043bc3b5b9752833b854ffa4da631470ce2a213163173e114111445"
Sep 11 14:47:47.273912 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:47:47.273888    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a59c51c6a043bc3b5b9752833b854ffa4da631470ce2a213163173e114111445\": container with ID starting with a59c51c6a043bc3b5b9752833b854ffa4da631470ce2a213163173e114111445 not found: ID does not exist" containerID="a59c51c6a043bc3b5b9752833b854ffa4da631470ce2a213163173e114111445"
Sep 11 14:47:47.274024 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:47.273926    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a59c51c6a043bc3b5b9752833b854ffa4da631470ce2a213163173e114111445"} err="failed to get container status \"a59c51c6a043bc3b5b9752833b854ffa4da631470ce2a213163173e114111445\": rpc error: code = NotFound desc = could not find container \"a59c51c6a043bc3b5b9752833b854ffa4da631470ce2a213163173e114111445\": container with ID starting with a59c51c6a043bc3b5b9752833b854ffa4da631470ce2a213163173e114111445 not found: ID does not exist"
Sep 11 14:47:47.274024 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:47.273950    2581 scope.go:117] "RemoveContainer" containerID="f8e71261cd31bb95cee7abb65ad46a5893bb12f629044fc8971098b24a223e42"
Sep 11 14:47:47.274246 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:47:47.274228    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8e71261cd31bb95cee7abb65ad46a5893bb12f629044fc8971098b24a223e42\": container with ID starting with f8e71261cd31bb95cee7abb65ad46a5893bb12f629044fc8971098b24a223e42 not found: ID does not exist" containerID="f8e71261cd31bb95cee7abb65ad46a5893bb12f629044fc8971098b24a223e42"
Sep 11 14:47:47.274286 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:47.274252    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8e71261cd31bb95cee7abb65ad46a5893bb12f629044fc8971098b24a223e42"} err="failed to get container status \"f8e71261cd31bb95cee7abb65ad46a5893bb12f629044fc8971098b24a223e42\": rpc error: code = NotFound desc = could not find container \"f8e71261cd31bb95cee7abb65ad46a5893bb12f629044fc8971098b24a223e42\": container with ID starting with f8e71261cd31bb95cee7abb65ad46a5893bb12f629044fc8971098b24a223e42 not found: ID does not exist"
Sep 11 14:47:47.323402 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:47.323368    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88197d9f-d033-4403-aef8-dd65332427a9-credential-socket\") pod \"88197d9f-d033-4403-aef8-dd65332427a9\" (UID: \"88197d9f-d033-4403-aef8-dd65332427a9\") "
Sep 11 14:47:47.323402 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:47.323408    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88197d9f-d033-4403-aef8-dd65332427a9-workload-certs\") pod \"88197d9f-d033-4403-aef8-dd65332427a9\" (UID: \"88197d9f-d033-4403-aef8-dd65332427a9\") "
Sep 11 14:47:47.323650 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:47.323446    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88197d9f-d033-4403-aef8-dd65332427a9-istio-podinfo\") pod \"88197d9f-d033-4403-aef8-dd65332427a9\" (UID: \"88197d9f-d033-4403-aef8-dd65332427a9\") "
Sep 11 14:47:47.323650 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:47.323462    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88197d9f-d033-4403-aef8-dd65332427a9-istiod-ca-cert\") pod \"88197d9f-d033-4403-aef8-dd65332427a9\" (UID: \"88197d9f-d033-4403-aef8-dd65332427a9\") "
Sep 11 14:47:47.323650 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:47.323490    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88197d9f-d033-4403-aef8-dd65332427a9-workload-socket\") pod \"88197d9f-d033-4403-aef8-dd65332427a9\" (UID: \"88197d9f-d033-4403-aef8-dd65332427a9\") "
Sep 11 14:47:47.323650 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:47.323564    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88197d9f-d033-4403-aef8-dd65332427a9-istio-envoy\") pod \"88197d9f-d033-4403-aef8-dd65332427a9\" (UID: \"88197d9f-d033-4403-aef8-dd65332427a9\") "
Sep 11 14:47:47.323650 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:47.323621    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88197d9f-d033-4403-aef8-dd65332427a9-istio-token\") pod \"88197d9f-d033-4403-aef8-dd65332427a9\" (UID: \"88197d9f-d033-4403-aef8-dd65332427a9\") "
Sep 11 14:47:47.323650 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:47.323643    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88197d9f-d033-4403-aef8-dd65332427a9-istio-data\") pod \"88197d9f-d033-4403-aef8-dd65332427a9\" (UID: \"88197d9f-d033-4403-aef8-dd65332427a9\") "
Sep 11 14:47:47.323952 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:47.323670    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6z5tg\" (UniqueName: \"kubernetes.io/projected/88197d9f-d033-4403-aef8-dd65332427a9-kube-api-access-6z5tg\") pod \"88197d9f-d033-4403-aef8-dd65332427a9\" (UID: \"88197d9f-d033-4403-aef8-dd65332427a9\") "
Sep 11 14:47:47.323952 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:47.323722    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88197d9f-d033-4403-aef8-dd65332427a9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "88197d9f-d033-4403-aef8-dd65332427a9" (UID: "88197d9f-d033-4403-aef8-dd65332427a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:47:47.323952 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:47.323742    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88197d9f-d033-4403-aef8-dd65332427a9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "88197d9f-d033-4403-aef8-dd65332427a9" (UID: "88197d9f-d033-4403-aef8-dd65332427a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:47:47.323952 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:47.323933    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88197d9f-d033-4403-aef8-dd65332427a9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "88197d9f-d033-4403-aef8-dd65332427a9" (UID: "88197d9f-d033-4403-aef8-dd65332427a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:47:47.324200 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:47.324073    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88197d9f-d033-4403-aef8-dd65332427a9-credential-socket\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:47:47.324200 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:47.324093    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88197d9f-d033-4403-aef8-dd65332427a9-workload-certs\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:47:47.324200 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:47.324081    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88197d9f-d033-4403-aef8-dd65332427a9-istio-data" (OuterVolumeSpecName: "istio-data") pod "88197d9f-d033-4403-aef8-dd65332427a9" (UID: "88197d9f-d033-4403-aef8-dd65332427a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:47:47.324200 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:47.324109    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88197d9f-d033-4403-aef8-dd65332427a9-workload-socket\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:47:47.324416 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:47.324209    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88197d9f-d033-4403-aef8-dd65332427a9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "88197d9f-d033-4403-aef8-dd65332427a9" (UID: "88197d9f-d033-4403-aef8-dd65332427a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:47:47.325764 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:47.325742    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88197d9f-d033-4403-aef8-dd65332427a9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "88197d9f-d033-4403-aef8-dd65332427a9" (UID: "88197d9f-d033-4403-aef8-dd65332427a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 14:47:47.325864 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:47.325838    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88197d9f-d033-4403-aef8-dd65332427a9-istio-token" (OuterVolumeSpecName: "istio-token") pod "88197d9f-d033-4403-aef8-dd65332427a9" (UID: "88197d9f-d033-4403-aef8-dd65332427a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:47:47.326151 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:47.326125    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88197d9f-d033-4403-aef8-dd65332427a9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "88197d9f-d033-4403-aef8-dd65332427a9" (UID: "88197d9f-d033-4403-aef8-dd65332427a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:47:47.326198 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:47.326169    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88197d9f-d033-4403-aef8-dd65332427a9-kube-api-access-6z5tg" (OuterVolumeSpecName: "kube-api-access-6z5tg") pod "88197d9f-d033-4403-aef8-dd65332427a9" (UID: "88197d9f-d033-4403-aef8-dd65332427a9"). InnerVolumeSpecName "kube-api-access-6z5tg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:47:47.424603 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:47.424566    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z5tg\" (UniqueName: \"kubernetes.io/projected/88197d9f-d033-4403-aef8-dd65332427a9-kube-api-access-6z5tg\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:47:47.424603 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:47.424598    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88197d9f-d033-4403-aef8-dd65332427a9-istio-podinfo\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:47:47.424603 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:47.424610    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88197d9f-d033-4403-aef8-dd65332427a9-istiod-ca-cert\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:47:47.424852 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:47.424620    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88197d9f-d033-4403-aef8-dd65332427a9-istio-envoy\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:47:47.424852 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:47.424628    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88197d9f-d033-4403-aef8-dd65332427a9-istio-token\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:47:47.424852 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:47.424636    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88197d9f-d033-4403-aef8-dd65332427a9-istio-data\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:47:47.576012 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:47.575961    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh"]
Sep 11 14:47:47.581509 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:47.581484    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-09b16c-openshift-default-6466f7f6d6-q2skh"]
Sep 11 14:47:49.499368 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:49.499292    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88197d9f-d033-4403-aef8-dd65332427a9" path="/var/lib/kubelet/pods/88197d9f-d033-4403-aef8-dd65332427a9/volumes"
Sep 11 14:47:53.287392 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:53.287350    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1c284e-29818965-vfmgq"]
Sep 11 14:47:53.289667 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:53.289639    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-1c284e-29818965-vfmgq"]
Sep 11 14:47:53.501129 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:53.501038    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e980be9-5d70-4705-b648-aa490a6b1aa9" path="/var/lib/kubelet/pods/8e980be9-5d70-4705-b648-aa490a6b1aa9/volumes"
Sep 11 14:47:53.667069 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:53.666985    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1c284e-b4nlj"]
Sep 11 14:47:53.675587 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:53.675560    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1c284e-b4nlj"]
Sep 11 14:47:55.499576 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:47:55.499539    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eee1dd32-5c4b-4b88-9656-78c1c2b93e2d" path="/var/lib/kubelet/pods/eee1dd32-5c4b-4b88-9656-78c1c2b93e2d/volumes"
Sep 11 14:48:01.294510 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:48:01.294473    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8"]
Sep 11 14:48:01.295027 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:48:01.294709    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8" podUID="1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887" containerName="istio-proxy" containerID="cri-o://b611a569df4ff66649b38e0234e5dde907fe5e0154f96bbb0afda33b83a8b80a" gracePeriod=30
Sep 11 14:48:05.403060 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:48:05.403020    2581 patch_prober.go:28] interesting pod/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 14:48:05.403515 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:48:05.403117    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8" podUID="1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 14:48:06.553116 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:48:06.553089    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8"
Sep 11 14:48:06.699564 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:48:06.699469    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-trltp\" (UniqueName: \"kubernetes.io/projected/1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887-kube-api-access-trltp\") pod \"1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887\" (UID: \"1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887\") "
Sep 11 14:48:06.699564 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:48:06.699521    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887-istio-podinfo\") pod \"1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887\" (UID: \"1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887\") "
Sep 11 14:48:06.699807 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:48:06.699584    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887-istio-token\") pod \"1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887\" (UID: \"1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887\") "
Sep 11 14:48:06.699807 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:48:06.699624    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887-istiod-ca-cert\") pod \"1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887\" (UID: \"1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887\") "
Sep 11 14:48:06.699807 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:48:06.699659    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887-istio-envoy\") pod \"1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887\" (UID: \"1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887\") "
Sep 11 14:48:06.699807 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:48:06.699700    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887-credential-socket\") pod \"1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887\" (UID: \"1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887\") "
Sep 11 14:48:06.699807 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:48:06.699733    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887-istio-data\") pod \"1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887\" (UID: \"1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887\") "
Sep 11 14:48:06.700073 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:48:06.699919    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887-workload-certs\") pod \"1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887\" (UID: \"1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887\") "
Sep 11 14:48:06.700073 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:48:06.700008    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887-workload-socket\") pod \"1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887\" (UID: \"1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887\") "
Sep 11 14:48:06.700367 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:48:06.700194    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887" (UID: "1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 14:48:06.700525 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:48:06.700455    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887" (UID: "1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:48:06.700525 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:48:06.700473    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887-istiod-ca-cert\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:48:06.700525 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:48:06.700498    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887-istio-data" (OuterVolumeSpecName: "istio-data") pod "1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887" (UID: "1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:48:06.700728 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:48:06.700674    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887" (UID: "1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:48:06.701346 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:48:06.701121    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887" (UID: "1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:48:06.702491 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:48:06.702450    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887-kube-api-access-trltp" (OuterVolumeSpecName: "kube-api-access-trltp") pod "1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887" (UID: "1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887"). InnerVolumeSpecName "kube-api-access-trltp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:48:06.702853 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:48:06.702818    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887-istio-token" (OuterVolumeSpecName: "istio-token") pod "1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887" (UID: "1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:48:06.702853 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:48:06.702838    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887" (UID: "1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:48:06.703611 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:48:06.703588    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887" (UID: "1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 14:48:06.801141 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:48:06.801095    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887-istio-token\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:48:06.801141 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:48:06.801140    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887-istio-envoy\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:48:06.801356 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:48:06.801156    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887-credential-socket\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:48:06.801356 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:48:06.801171    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887-istio-data\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:48:06.801356 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:48:06.801185    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887-workload-certs\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:48:06.801356 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:48:06.801198    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887-workload-socket\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:48:06.801356 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:48:06.801213    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trltp\" (UniqueName: \"kubernetes.io/projected/1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887-kube-api-access-trltp\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:48:06.801356 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:48:06.801226    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887-istio-podinfo\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:48:07.365381 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:48:07.365343    2581 generic.go:356] "Generic (PLEG): container finished" podID="1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887" containerID="b611a569df4ff66649b38e0234e5dde907fe5e0154f96bbb0afda33b83a8b80a" exitCode=0
Sep 11 14:48:07.365562 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:48:07.365400    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8"
Sep 11 14:48:07.365562 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:48:07.365421    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8" event={"ID":"1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887","Type":"ContainerDied","Data":"b611a569df4ff66649b38e0234e5dde907fe5e0154f96bbb0afda33b83a8b80a"}
Sep 11 14:48:07.365562 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:48:07.365489    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8" event={"ID":"1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887","Type":"ContainerDied","Data":"e120b8b5759224652e388193d05c9c5d0db753bb8053429df224d584263e1b3f"}
Sep 11 14:48:07.365562 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:48:07.365520    2581 scope.go:117] "RemoveContainer" containerID="b611a569df4ff66649b38e0234e5dde907fe5e0154f96bbb0afda33b83a8b80a"
Sep 11 14:48:07.375286 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:48:07.375267    2581 scope.go:117] "RemoveContainer" containerID="07ed191a516fa6ba406ef6c649c57c30e676d7f97c33a1a4328f78259aef0bc8"
Sep 11 14:48:07.383624 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:48:07.383602    2581 scope.go:117] "RemoveContainer" containerID="b611a569df4ff66649b38e0234e5dde907fe5e0154f96bbb0afda33b83a8b80a"
Sep 11 14:48:07.383915 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:48:07.383894    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b611a569df4ff66649b38e0234e5dde907fe5e0154f96bbb0afda33b83a8b80a\": container with ID starting with b611a569df4ff66649b38e0234e5dde907fe5e0154f96bbb0afda33b83a8b80a not found: ID does not exist" containerID="b611a569df4ff66649b38e0234e5dde907fe5e0154f96bbb0afda33b83a8b80a"
Sep 11 14:48:07.384042 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:48:07.383923    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b611a569df4ff66649b38e0234e5dde907fe5e0154f96bbb0afda33b83a8b80a"} err="failed to get container status \"b611a569df4ff66649b38e0234e5dde907fe5e0154f96bbb0afda33b83a8b80a\": rpc error: code = NotFound desc = could not find container \"b611a569df4ff66649b38e0234e5dde907fe5e0154f96bbb0afda33b83a8b80a\": container with ID starting with b611a569df4ff66649b38e0234e5dde907fe5e0154f96bbb0afda33b83a8b80a not found: ID does not exist"
Sep 11 14:48:07.384042 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:48:07.383942    2581 scope.go:117] "RemoveContainer" containerID="07ed191a516fa6ba406ef6c649c57c30e676d7f97c33a1a4328f78259aef0bc8"
Sep 11 14:48:07.384241 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:48:07.384215    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07ed191a516fa6ba406ef6c649c57c30e676d7f97c33a1a4328f78259aef0bc8\": container with ID starting with 07ed191a516fa6ba406ef6c649c57c30e676d7f97c33a1a4328f78259aef0bc8 not found: ID does not exist" containerID="07ed191a516fa6ba406ef6c649c57c30e676d7f97c33a1a4328f78259aef0bc8"
Sep 11 14:48:07.384295 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:48:07.384246    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07ed191a516fa6ba406ef6c649c57c30e676d7f97c33a1a4328f78259aef0bc8"} err="failed to get container status \"07ed191a516fa6ba406ef6c649c57c30e676d7f97c33a1a4328f78259aef0bc8\": rpc error: code = NotFound desc = could not find container \"07ed191a516fa6ba406ef6c649c57c30e676d7f97c33a1a4328f78259aef0bc8\": container with ID starting with 07ed191a516fa6ba406ef6c649c57c30e676d7f97c33a1a4328f78259aef0bc8 not found: ID does not exist"
Sep 11 14:48:07.392692 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:48:07.392667    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8"]
Sep 11 14:48:07.398084 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:48:07.398062    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-1c284e-openshift-default-58db8476f6-rfhl8"]
Sep 11 14:48:07.499081 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:48:07.499049    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887" path="/var/lib/kubelet/pods/1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887/volumes"
Sep 11 14:49:57.772863 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:49:57.772826    2581 scope.go:117] "RemoveContainer" containerID="3ff4180a6d5283828230ea08d2fb870463c3f9a34505a576d878bf5467a96c38"
Sep 11 14:49:57.781582 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:49:57.781560    2581 scope.go:117] "RemoveContainer" containerID="ddb4fa2f8d3acd5b5b7075639e30b3ad6921c4e96a4ec4d199d93d54dd27e0f8"
Sep 11 14:50:57.803407 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:50:57.803266    2581 scope.go:117] "RemoveContainer" containerID="f2cd78a44dfb8c4b28ea7e6d8481fa2b8b94561fb301db80dbd28249fc344660"
Sep 11 14:50:57.811718 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:50:57.811698    2581 scope.go:117] "RemoveContainer" containerID="a051541d11f4b9b6f17463b1c1b6a497cad8c69f5479bf6491476da3676b81b1"
Sep 11 14:51:57.834843 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:51:57.834808    2581 scope.go:117] "RemoveContainer" containerID="ae268b2b9ecd63f37bc4598f0a8c6bbf09f3e8088b02bea1759002609c99c9ed"
Sep 11 14:51:57.843441 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:51:57.843420    2581 scope.go:117] "RemoveContainer" containerID="5dd4e2b205bec3d8b687c6635369e48aaa2c8f4db12ab57715608e5c57646e49"
Sep 11 14:51:57.851006 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:51:57.850987    2581 scope.go:117] "RemoveContainer" containerID="1da8fd484703f7a2f36fcfc5a95c6cf7b64479a48ba55efe0177d2e5d90ca292"
Sep 11 14:51:57.858134 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:51:57.858115    2581 scope.go:117] "RemoveContainer" containerID="449e462b597d67b3326ed3d96d53d9585a6c2a476d3b000c2eb2cfaedda42583"
Sep 11 14:52:30.825531 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:30.825445    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-bf44e879-kserve-67dc6b47c9-m4pmh"]
Sep 11 14:52:30.826036 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:30.825756    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eee1dd32-5c4b-4b88-9656-78c1c2b93e2d" containerName="revoke-keys"
Sep 11 14:52:30.826036 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:30.825767    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="eee1dd32-5c4b-4b88-9656-78c1c2b93e2d" containerName="revoke-keys"
Sep 11 14:52:30.826036 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:30.825775    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887" containerName="istio-proxy"
Sep 11 14:52:30.826036 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:30.825781    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887" containerName="istio-proxy"
Sep 11 14:52:30.826036 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:30.825787    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88197d9f-d033-4403-aef8-dd65332427a9" containerName="istio-proxy"
Sep 11 14:52:30.826036 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:30.825792    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="88197d9f-d033-4403-aef8-dd65332427a9" containerName="istio-proxy"
Sep 11 14:52:30.826036 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:30.825804    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887" containerName="istio-proxy"
Sep 11 14:52:30.826036 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:30.825809    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887" containerName="istio-proxy"
Sep 11 14:52:30.826036 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:30.825872    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="88197d9f-d033-4403-aef8-dd65332427a9" containerName="istio-proxy"
Sep 11 14:52:30.826036 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:30.825880    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="88197d9f-d033-4403-aef8-dd65332427a9" containerName="istio-proxy"
Sep 11 14:52:30.826036 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:30.825886    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="eee1dd32-5c4b-4b88-9656-78c1c2b93e2d" containerName="revoke-keys"
Sep 11 14:52:30.826036 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:30.825891    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887" containerName="istio-proxy"
Sep 11 14:52:30.826036 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:30.825897    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d0b8dcf-6ab3-40ac-8b31-3eca2d6b2887" containerName="istio-proxy"
Sep 11 14:52:30.828504 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:30.828485    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-bf44e879-kserve-67dc6b47c9-m4pmh"
Sep 11 14:52:30.831560 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:30.831534    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-bf44e879-kserve-self-signed-certs\""
Sep 11 14:52:30.839944 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:30.839922    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-bf44e879-kserve-67dc6b47c9-m4pmh"]
Sep 11 14:52:30.985078 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:30.985045    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45c6d388-acf7-4db2-bac6-b03afc3a58b0-tmp-dir\") pod \"e2e-same-id-a-bf44e879-kserve-67dc6b47c9-m4pmh\" (UID: \"45c6d388-acf7-4db2-bac6-b03afc3a58b0\") " pod="llm/e2e-same-id-a-bf44e879-kserve-67dc6b47c9-m4pmh"
Sep 11 14:52:30.985258 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:30.985095    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45c6d388-acf7-4db2-bac6-b03afc3a58b0-tls-certs\") pod \"e2e-same-id-a-bf44e879-kserve-67dc6b47c9-m4pmh\" (UID: \"45c6d388-acf7-4db2-bac6-b03afc3a58b0\") " pod="llm/e2e-same-id-a-bf44e879-kserve-67dc6b47c9-m4pmh"
Sep 11 14:52:30.985258 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:30.985179    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45c6d388-acf7-4db2-bac6-b03afc3a58b0-model-cache\") pod \"e2e-same-id-a-bf44e879-kserve-67dc6b47c9-m4pmh\" (UID: \"45c6d388-acf7-4db2-bac6-b03afc3a58b0\") " pod="llm/e2e-same-id-a-bf44e879-kserve-67dc6b47c9-m4pmh"
Sep 11 14:52:30.985258 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:30.985239    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sssxw\" (UniqueName: \"kubernetes.io/projected/45c6d388-acf7-4db2-bac6-b03afc3a58b0-kube-api-access-sssxw\") pod \"e2e-same-id-a-bf44e879-kserve-67dc6b47c9-m4pmh\" (UID: \"45c6d388-acf7-4db2-bac6-b03afc3a58b0\") " pod="llm/e2e-same-id-a-bf44e879-kserve-67dc6b47c9-m4pmh"
Sep 11 14:52:30.985382 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:30.985283    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45c6d388-acf7-4db2-bac6-b03afc3a58b0-home\") pod \"e2e-same-id-a-bf44e879-kserve-67dc6b47c9-m4pmh\" (UID: \"45c6d388-acf7-4db2-bac6-b03afc3a58b0\") " pod="llm/e2e-same-id-a-bf44e879-kserve-67dc6b47c9-m4pmh"
Sep 11 14:52:30.985382 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:30.985300    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45c6d388-acf7-4db2-bac6-b03afc3a58b0-dshm\") pod \"e2e-same-id-a-bf44e879-kserve-67dc6b47c9-m4pmh\" (UID: \"45c6d388-acf7-4db2-bac6-b03afc3a58b0\") " pod="llm/e2e-same-id-a-bf44e879-kserve-67dc6b47c9-m4pmh"
Sep 11 14:52:31.086135 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:31.086038    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45c6d388-acf7-4db2-bac6-b03afc3a58b0-tls-certs\") pod \"e2e-same-id-a-bf44e879-kserve-67dc6b47c9-m4pmh\" (UID: \"45c6d388-acf7-4db2-bac6-b03afc3a58b0\") " pod="llm/e2e-same-id-a-bf44e879-kserve-67dc6b47c9-m4pmh"
Sep 11 14:52:31.086135 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:31.086094    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45c6d388-acf7-4db2-bac6-b03afc3a58b0-model-cache\") pod \"e2e-same-id-a-bf44e879-kserve-67dc6b47c9-m4pmh\" (UID: \"45c6d388-acf7-4db2-bac6-b03afc3a58b0\") " pod="llm/e2e-same-id-a-bf44e879-kserve-67dc6b47c9-m4pmh"
Sep 11 14:52:31.086370 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:31.086137    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sssxw\" (UniqueName: \"kubernetes.io/projected/45c6d388-acf7-4db2-bac6-b03afc3a58b0-kube-api-access-sssxw\") pod \"e2e-same-id-a-bf44e879-kserve-67dc6b47c9-m4pmh\" (UID: \"45c6d388-acf7-4db2-bac6-b03afc3a58b0\") " pod="llm/e2e-same-id-a-bf44e879-kserve-67dc6b47c9-m4pmh"
Sep 11 14:52:31.086370 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:31.086278    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45c6d388-acf7-4db2-bac6-b03afc3a58b0-home\") pod \"e2e-same-id-a-bf44e879-kserve-67dc6b47c9-m4pmh\" (UID: \"45c6d388-acf7-4db2-bac6-b03afc3a58b0\") " pod="llm/e2e-same-id-a-bf44e879-kserve-67dc6b47c9-m4pmh"
Sep 11 14:52:31.086370 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:31.086314    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45c6d388-acf7-4db2-bac6-b03afc3a58b0-dshm\") pod \"e2e-same-id-a-bf44e879-kserve-67dc6b47c9-m4pmh\" (UID: \"45c6d388-acf7-4db2-bac6-b03afc3a58b0\") " pod="llm/e2e-same-id-a-bf44e879-kserve-67dc6b47c9-m4pmh"
Sep 11 14:52:31.086370 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:31.086346    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45c6d388-acf7-4db2-bac6-b03afc3a58b0-tmp-dir\") pod \"e2e-same-id-a-bf44e879-kserve-67dc6b47c9-m4pmh\" (UID: \"45c6d388-acf7-4db2-bac6-b03afc3a58b0\") " pod="llm/e2e-same-id-a-bf44e879-kserve-67dc6b47c9-m4pmh"
Sep 11 14:52:31.086599 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:31.086576    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45c6d388-acf7-4db2-bac6-b03afc3a58b0-model-cache\") pod \"e2e-same-id-a-bf44e879-kserve-67dc6b47c9-m4pmh\" (UID: \"45c6d388-acf7-4db2-bac6-b03afc3a58b0\") " pod="llm/e2e-same-id-a-bf44e879-kserve-67dc6b47c9-m4pmh"
Sep 11 14:52:31.086599 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:31.086585    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45c6d388-acf7-4db2-bac6-b03afc3a58b0-home\") pod \"e2e-same-id-a-bf44e879-kserve-67dc6b47c9-m4pmh\" (UID: \"45c6d388-acf7-4db2-bac6-b03afc3a58b0\") " pod="llm/e2e-same-id-a-bf44e879-kserve-67dc6b47c9-m4pmh"
Sep 11 14:52:31.086667 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:31.086647    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45c6d388-acf7-4db2-bac6-b03afc3a58b0-tmp-dir\") pod \"e2e-same-id-a-bf44e879-kserve-67dc6b47c9-m4pmh\" (UID: \"45c6d388-acf7-4db2-bac6-b03afc3a58b0\") " pod="llm/e2e-same-id-a-bf44e879-kserve-67dc6b47c9-m4pmh"
Sep 11 14:52:31.088411 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:31.088379    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45c6d388-acf7-4db2-bac6-b03afc3a58b0-dshm\") pod \"e2e-same-id-a-bf44e879-kserve-67dc6b47c9-m4pmh\" (UID: \"45c6d388-acf7-4db2-bac6-b03afc3a58b0\") " pod="llm/e2e-same-id-a-bf44e879-kserve-67dc6b47c9-m4pmh"
Sep 11 14:52:31.088815 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:31.088792    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45c6d388-acf7-4db2-bac6-b03afc3a58b0-tls-certs\") pod \"e2e-same-id-a-bf44e879-kserve-67dc6b47c9-m4pmh\" (UID: \"45c6d388-acf7-4db2-bac6-b03afc3a58b0\") " pod="llm/e2e-same-id-a-bf44e879-kserve-67dc6b47c9-m4pmh"
Sep 11 14:52:31.106225 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:31.106196    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sssxw\" (UniqueName: \"kubernetes.io/projected/45c6d388-acf7-4db2-bac6-b03afc3a58b0-kube-api-access-sssxw\") pod \"e2e-same-id-a-bf44e879-kserve-67dc6b47c9-m4pmh\" (UID: \"45c6d388-acf7-4db2-bac6-b03afc3a58b0\") " pod="llm/e2e-same-id-a-bf44e879-kserve-67dc6b47c9-m4pmh"
Sep 11 14:52:31.137563 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:31.137530    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-bf44e879-kserve-67dc6b47c9-m4pmh"
Sep 11 14:52:31.423296 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:31.423213    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-bf44e879-kserve-67dc6b47c9-m4pmh"]
Sep 11 14:52:31.425929 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:52:31.425902    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod45c6d388_acf7_4db2_bac6_b03afc3a58b0.slice/crio-62f312f6f4dcd60845443e6ef771263affa49c15873fe814f79c9664ddc4af2d WatchSource:0}: Error finding container 62f312f6f4dcd60845443e6ef771263affa49c15873fe814f79c9664ddc4af2d: Status 404 returned error can't find the container with id 62f312f6f4dcd60845443e6ef771263affa49c15873fe814f79c9664ddc4af2d
Sep 11 14:52:31.698080 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:31.697983    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-bf44e879-kserve-67dc6b47c9-m4pmh" event={"ID":"45c6d388-acf7-4db2-bac6-b03afc3a58b0","Type":"ContainerStarted","Data":"87c0ebe3f28d41df964e69fb72ea8291185ac206b1ecdfbb1263efc4744c1dff"}
Sep 11 14:52:31.698080 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:31.698026    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-bf44e879-kserve-67dc6b47c9-m4pmh" event={"ID":"45c6d388-acf7-4db2-bac6-b03afc3a58b0","Type":"ContainerStarted","Data":"62f312f6f4dcd60845443e6ef771263affa49c15873fe814f79c9664ddc4af2d"}
Sep 11 14:52:31.724133 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:31.724081    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-bf44e879-kserve-67dc6b47c9-m4pmh" podStartSLOduration=1.7240671810000001 podStartE2EDuration="1.724067181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:52:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:52:31.722376857 +0000 UTC m=+2254.872926924" watchObservedRunningTime="2026-09-11 14:52:31.724067181 +0000 UTC m=+2254.874617273"
Sep 11 14:52:41.138639 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:41.138594    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-bf44e879-kserve-67dc6b47c9-m4pmh"
Sep 11 14:52:41.139165 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:41.138692    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-bf44e879-kserve-67dc6b47c9-m4pmh"
Sep 11 14:52:41.151479 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:41.151456    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-bf44e879-kserve-67dc6b47c9-m4pmh"
Sep 11 14:52:41.757206 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:41.757174    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-bf44e879-kserve-67dc6b47c9-m4pmh"
Sep 11 14:52:43.531822 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:43.531788    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-bf44e879-kserve-586f486895-292qh"]
Sep 11 14:52:43.532262 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:43.532159    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88197d9f-d033-4403-aef8-dd65332427a9" containerName="istio-proxy"
Sep 11 14:52:43.532262 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:43.532174    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="88197d9f-d033-4403-aef8-dd65332427a9" containerName="istio-proxy"
Sep 11 14:52:43.535123 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:43.535107    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-bf44e879-kserve-586f486895-292qh"
Sep 11 14:52:43.538256 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:43.538233    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-bf44e879-kserve-self-signed-certs\""
Sep 11 14:52:43.545737 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:43.545712    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-bf44e879-kserve-586f486895-292qh"]
Sep 11 14:52:43.596616 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:43.596587    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/046c2c1b-7022-4e06-9730-cebaebc98014-dshm\") pod \"e2e-same-id-b-bf44e879-kserve-586f486895-292qh\" (UID: \"046c2c1b-7022-4e06-9730-cebaebc98014\") " pod="llm/e2e-same-id-b-bf44e879-kserve-586f486895-292qh"
Sep 11 14:52:43.596805 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:43.596631    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/046c2c1b-7022-4e06-9730-cebaebc98014-home\") pod \"e2e-same-id-b-bf44e879-kserve-586f486895-292qh\" (UID: \"046c2c1b-7022-4e06-9730-cebaebc98014\") " pod="llm/e2e-same-id-b-bf44e879-kserve-586f486895-292qh"
Sep 11 14:52:43.596805 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:43.596688    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/046c2c1b-7022-4e06-9730-cebaebc98014-model-cache\") pod \"e2e-same-id-b-bf44e879-kserve-586f486895-292qh\" (UID: \"046c2c1b-7022-4e06-9730-cebaebc98014\") " pod="llm/e2e-same-id-b-bf44e879-kserve-586f486895-292qh"
Sep 11 14:52:43.596805 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:43.596716    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69kvk\" (UniqueName: \"kubernetes.io/projected/046c2c1b-7022-4e06-9730-cebaebc98014-kube-api-access-69kvk\") pod \"e2e-same-id-b-bf44e879-kserve-586f486895-292qh\" (UID: \"046c2c1b-7022-4e06-9730-cebaebc98014\") " pod="llm/e2e-same-id-b-bf44e879-kserve-586f486895-292qh"
Sep 11 14:52:43.596805 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:43.596778    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/046c2c1b-7022-4e06-9730-cebaebc98014-tmp-dir\") pod \"e2e-same-id-b-bf44e879-kserve-586f486895-292qh\" (UID: \"046c2c1b-7022-4e06-9730-cebaebc98014\") " pod="llm/e2e-same-id-b-bf44e879-kserve-586f486895-292qh"
Sep 11 14:52:43.597039 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:43.596876    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/046c2c1b-7022-4e06-9730-cebaebc98014-tls-certs\") pod \"e2e-same-id-b-bf44e879-kserve-586f486895-292qh\" (UID: \"046c2c1b-7022-4e06-9730-cebaebc98014\") " pod="llm/e2e-same-id-b-bf44e879-kserve-586f486895-292qh"
Sep 11 14:52:43.697583 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:43.697548    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/046c2c1b-7022-4e06-9730-cebaebc98014-tmp-dir\") pod \"e2e-same-id-b-bf44e879-kserve-586f486895-292qh\" (UID: \"046c2c1b-7022-4e06-9730-cebaebc98014\") " pod="llm/e2e-same-id-b-bf44e879-kserve-586f486895-292qh"
Sep 11 14:52:43.697758 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:43.697620    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/046c2c1b-7022-4e06-9730-cebaebc98014-tls-certs\") pod \"e2e-same-id-b-bf44e879-kserve-586f486895-292qh\" (UID: \"046c2c1b-7022-4e06-9730-cebaebc98014\") " pod="llm/e2e-same-id-b-bf44e879-kserve-586f486895-292qh"
Sep 11 14:52:43.697758 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:43.697640    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/046c2c1b-7022-4e06-9730-cebaebc98014-dshm\") pod \"e2e-same-id-b-bf44e879-kserve-586f486895-292qh\" (UID: \"046c2c1b-7022-4e06-9730-cebaebc98014\") " pod="llm/e2e-same-id-b-bf44e879-kserve-586f486895-292qh"
Sep 11 14:52:43.697758 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:43.697661    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/046c2c1b-7022-4e06-9730-cebaebc98014-home\") pod \"e2e-same-id-b-bf44e879-kserve-586f486895-292qh\" (UID: \"046c2c1b-7022-4e06-9730-cebaebc98014\") " pod="llm/e2e-same-id-b-bf44e879-kserve-586f486895-292qh"
Sep 11 14:52:43.697914 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:43.697787    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/046c2c1b-7022-4e06-9730-cebaebc98014-model-cache\") pod \"e2e-same-id-b-bf44e879-kserve-586f486895-292qh\" (UID: \"046c2c1b-7022-4e06-9730-cebaebc98014\") " pod="llm/e2e-same-id-b-bf44e879-kserve-586f486895-292qh"
Sep 11 14:52:43.697914 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:43.697833    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69kvk\" (UniqueName: \"kubernetes.io/projected/046c2c1b-7022-4e06-9730-cebaebc98014-kube-api-access-69kvk\") pod \"e2e-same-id-b-bf44e879-kserve-586f486895-292qh\" (UID: \"046c2c1b-7022-4e06-9730-cebaebc98014\") " pod="llm/e2e-same-id-b-bf44e879-kserve-586f486895-292qh"
Sep 11 14:52:43.698058 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:43.698014    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/046c2c1b-7022-4e06-9730-cebaebc98014-tmp-dir\") pod \"e2e-same-id-b-bf44e879-kserve-586f486895-292qh\" (UID: \"046c2c1b-7022-4e06-9730-cebaebc98014\") " pod="llm/e2e-same-id-b-bf44e879-kserve-586f486895-292qh"
Sep 11 14:52:43.698123 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:43.698078    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/046c2c1b-7022-4e06-9730-cebaebc98014-home\") pod \"e2e-same-id-b-bf44e879-kserve-586f486895-292qh\" (UID: \"046c2c1b-7022-4e06-9730-cebaebc98014\") " pod="llm/e2e-same-id-b-bf44e879-kserve-586f486895-292qh"
Sep 11 14:52:43.698186 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:43.698157    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/046c2c1b-7022-4e06-9730-cebaebc98014-model-cache\") pod \"e2e-same-id-b-bf44e879-kserve-586f486895-292qh\" (UID: \"046c2c1b-7022-4e06-9730-cebaebc98014\") " pod="llm/e2e-same-id-b-bf44e879-kserve-586f486895-292qh"
Sep 11 14:52:43.699912 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:43.699887    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/046c2c1b-7022-4e06-9730-cebaebc98014-dshm\") pod \"e2e-same-id-b-bf44e879-kserve-586f486895-292qh\" (UID: \"046c2c1b-7022-4e06-9730-cebaebc98014\") " pod="llm/e2e-same-id-b-bf44e879-kserve-586f486895-292qh"
Sep 11 14:52:43.700045 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:43.700019    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/046c2c1b-7022-4e06-9730-cebaebc98014-tls-certs\") pod \"e2e-same-id-b-bf44e879-kserve-586f486895-292qh\" (UID: \"046c2c1b-7022-4e06-9730-cebaebc98014\") " pod="llm/e2e-same-id-b-bf44e879-kserve-586f486895-292qh"
Sep 11 14:52:43.710159 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:43.710141    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69kvk\" (UniqueName: \"kubernetes.io/projected/046c2c1b-7022-4e06-9730-cebaebc98014-kube-api-access-69kvk\") pod \"e2e-same-id-b-bf44e879-kserve-586f486895-292qh\" (UID: \"046c2c1b-7022-4e06-9730-cebaebc98014\") " pod="llm/e2e-same-id-b-bf44e879-kserve-586f486895-292qh"
Sep 11 14:52:43.844800 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:43.844706    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-bf44e879-kserve-586f486895-292qh"
Sep 11 14:52:44.122546 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:44.122522    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-bf44e879-kserve-586f486895-292qh"]
Sep 11 14:52:44.124594 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:52:44.124567    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod046c2c1b_7022_4e06_9730_cebaebc98014.slice/crio-8d9c5ecdbecbaec1391b130853ae0cd37ab87924946788110fa321d261540d7a WatchSource:0}: Error finding container 8d9c5ecdbecbaec1391b130853ae0cd37ab87924946788110fa321d261540d7a: Status 404 returned error can't find the container with id 8d9c5ecdbecbaec1391b130853ae0cd37ab87924946788110fa321d261540d7a
Sep 11 14:52:44.763529 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:44.763491    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-bf44e879-kserve-586f486895-292qh" event={"ID":"046c2c1b-7022-4e06-9730-cebaebc98014","Type":"ContainerStarted","Data":"6f6700caa463007c7e70a4f236ffb3b3250747ad27d88a01107ebbd1dc6a975d"}
Sep 11 14:52:44.763945 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:44.763540    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-bf44e879-kserve-586f486895-292qh" event={"ID":"046c2c1b-7022-4e06-9730-cebaebc98014","Type":"ContainerStarted","Data":"8d9c5ecdbecbaec1391b130853ae0cd37ab87924946788110fa321d261540d7a"}
Sep 11 14:52:44.811712 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:44.811660    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-bf44e879-kserve-586f486895-292qh" podStartSLOduration=1.811646674 podStartE2EDuration="1.811646674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:52:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:52:44.810800395 +0000 UTC m=+2267.961350459" watchObservedRunningTime="2026-09-11 14:52:44.811646674 +0000 UTC m=+2267.962196738"
Sep 11 14:52:53.845172 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:53.845122    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-bf44e879-kserve-586f486895-292qh"
Sep 11 14:52:53.845660 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:53.845194    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-bf44e879-kserve-586f486895-292qh"
Sep 11 14:52:53.857789 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:53.857761    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-bf44e879-kserve-586f486895-292qh"
Sep 11 14:52:54.822544 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:52:54.822513    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-bf44e879-kserve-586f486895-292qh"
Sep 11 14:53:09.326283 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:09.326250    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-bf44e879-kserve-67dc6b47c9-m4pmh"]
Sep 11 14:53:11.517876 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:11.517845    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-bf44e879-kserve-586f486895-292qh"]
Sep 11 14:53:24.344946 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:24.344898    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-bf44e879-kserve-67dc6b47c9-m4pmh" podUID="45c6d388-acf7-4db2-bac6-b03afc3a58b0" containerName="main" containerID="cri-o://87c0ebe3f28d41df964e69fb72ea8291185ac206b1ecdfbb1263efc4744c1dff" gracePeriod=45
Sep 11 14:53:24.597193 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:24.597126    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-bf44e879-kserve-67dc6b47c9-m4pmh"
Sep 11 14:53:24.671405 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:24.671370    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45c6d388-acf7-4db2-bac6-b03afc3a58b0-tls-certs\") pod \"45c6d388-acf7-4db2-bac6-b03afc3a58b0\" (UID: \"45c6d388-acf7-4db2-bac6-b03afc3a58b0\") "
Sep 11 14:53:24.671682 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:24.671426    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45c6d388-acf7-4db2-bac6-b03afc3a58b0-home\") pod \"45c6d388-acf7-4db2-bac6-b03afc3a58b0\" (UID: \"45c6d388-acf7-4db2-bac6-b03afc3a58b0\") "
Sep 11 14:53:24.671682 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:24.671468    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45c6d388-acf7-4db2-bac6-b03afc3a58b0-model-cache\") pod \"45c6d388-acf7-4db2-bac6-b03afc3a58b0\" (UID: \"45c6d388-acf7-4db2-bac6-b03afc3a58b0\") "
Sep 11 14:53:24.671682 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:24.671507    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sssxw\" (UniqueName: \"kubernetes.io/projected/45c6d388-acf7-4db2-bac6-b03afc3a58b0-kube-api-access-sssxw\") pod \"45c6d388-acf7-4db2-bac6-b03afc3a58b0\" (UID: \"45c6d388-acf7-4db2-bac6-b03afc3a58b0\") "
Sep 11 14:53:24.671682 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:24.671537    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45c6d388-acf7-4db2-bac6-b03afc3a58b0-tmp-dir\") pod \"45c6d388-acf7-4db2-bac6-b03afc3a58b0\" (UID: \"45c6d388-acf7-4db2-bac6-b03afc3a58b0\") "
Sep 11 14:53:24.671682 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:24.671596    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45c6d388-acf7-4db2-bac6-b03afc3a58b0-dshm\") pod \"45c6d388-acf7-4db2-bac6-b03afc3a58b0\" (UID: \"45c6d388-acf7-4db2-bac6-b03afc3a58b0\") "
Sep 11 14:53:24.672003 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:24.671804    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45c6d388-acf7-4db2-bac6-b03afc3a58b0-home" (OuterVolumeSpecName: "home") pod "45c6d388-acf7-4db2-bac6-b03afc3a58b0" (UID: "45c6d388-acf7-4db2-bac6-b03afc3a58b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:53:24.672003 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:24.671867    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45c6d388-acf7-4db2-bac6-b03afc3a58b0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "45c6d388-acf7-4db2-bac6-b03afc3a58b0" (UID: "45c6d388-acf7-4db2-bac6-b03afc3a58b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:53:24.672003 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:24.671887    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45c6d388-acf7-4db2-bac6-b03afc3a58b0-model-cache" (OuterVolumeSpecName: "model-cache") pod "45c6d388-acf7-4db2-bac6-b03afc3a58b0" (UID: "45c6d388-acf7-4db2-bac6-b03afc3a58b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:53:24.672172 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:24.672013    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45c6d388-acf7-4db2-bac6-b03afc3a58b0-model-cache\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:53:24.672172 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:24.672031    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45c6d388-acf7-4db2-bac6-b03afc3a58b0-tmp-dir\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:53:24.672172 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:24.672043    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45c6d388-acf7-4db2-bac6-b03afc3a58b0-home\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:53:24.673779 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:24.673743    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45c6d388-acf7-4db2-bac6-b03afc3a58b0-dshm" (OuterVolumeSpecName: "dshm") pod "45c6d388-acf7-4db2-bac6-b03afc3a58b0" (UID: "45c6d388-acf7-4db2-bac6-b03afc3a58b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:53:24.673884 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:24.673825    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45c6d388-acf7-4db2-bac6-b03afc3a58b0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "45c6d388-acf7-4db2-bac6-b03afc3a58b0" (UID: "45c6d388-acf7-4db2-bac6-b03afc3a58b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:53:24.673884 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:24.673830    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45c6d388-acf7-4db2-bac6-b03afc3a58b0-kube-api-access-sssxw" (OuterVolumeSpecName: "kube-api-access-sssxw") pod "45c6d388-acf7-4db2-bac6-b03afc3a58b0" (UID: "45c6d388-acf7-4db2-bac6-b03afc3a58b0"). InnerVolumeSpecName "kube-api-access-sssxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:53:24.773405 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:24.773366    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45c6d388-acf7-4db2-bac6-b03afc3a58b0-tls-certs\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:53:24.773405 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:24.773401    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sssxw\" (UniqueName: \"kubernetes.io/projected/45c6d388-acf7-4db2-bac6-b03afc3a58b0-kube-api-access-sssxw\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:53:24.773405 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:24.773411    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45c6d388-acf7-4db2-bac6-b03afc3a58b0-dshm\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:53:24.964666 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:24.964577    2581 generic.go:356] "Generic (PLEG): container finished" podID="45c6d388-acf7-4db2-bac6-b03afc3a58b0" containerID="87c0ebe3f28d41df964e69fb72ea8291185ac206b1ecdfbb1263efc4744c1dff" exitCode=0
Sep 11 14:53:24.964666 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:24.964621    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-bf44e879-kserve-67dc6b47c9-m4pmh"
Sep 11 14:53:24.964666 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:24.964652    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-bf44e879-kserve-67dc6b47c9-m4pmh" event={"ID":"45c6d388-acf7-4db2-bac6-b03afc3a58b0","Type":"ContainerDied","Data":"87c0ebe3f28d41df964e69fb72ea8291185ac206b1ecdfbb1263efc4744c1dff"}
Sep 11 14:53:24.964926 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:24.964696    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-bf44e879-kserve-67dc6b47c9-m4pmh" event={"ID":"45c6d388-acf7-4db2-bac6-b03afc3a58b0","Type":"ContainerDied","Data":"62f312f6f4dcd60845443e6ef771263affa49c15873fe814f79c9664ddc4af2d"}
Sep 11 14:53:24.964926 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:24.964717    2581 scope.go:117] "RemoveContainer" containerID="87c0ebe3f28d41df964e69fb72ea8291185ac206b1ecdfbb1263efc4744c1dff"
Sep 11 14:53:24.973860 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:24.973842    2581 scope.go:117] "RemoveContainer" containerID="87c0ebe3f28d41df964e69fb72ea8291185ac206b1ecdfbb1263efc4744c1dff"
Sep 11 14:53:24.974168 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:53:24.974144    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87c0ebe3f28d41df964e69fb72ea8291185ac206b1ecdfbb1263efc4744c1dff\": container with ID starting with 87c0ebe3f28d41df964e69fb72ea8291185ac206b1ecdfbb1263efc4744c1dff not found: ID does not exist" containerID="87c0ebe3f28d41df964e69fb72ea8291185ac206b1ecdfbb1263efc4744c1dff"
Sep 11 14:53:24.974231 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:24.974181    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87c0ebe3f28d41df964e69fb72ea8291185ac206b1ecdfbb1263efc4744c1dff"} err="failed to get container status \"87c0ebe3f28d41df964e69fb72ea8291185ac206b1ecdfbb1263efc4744c1dff\": rpc error: code = NotFound desc = could not find container \"87c0ebe3f28d41df964e69fb72ea8291185ac206b1ecdfbb1263efc4744c1dff\": container with ID starting with 87c0ebe3f28d41df964e69fb72ea8291185ac206b1ecdfbb1263efc4744c1dff not found: ID does not exist"
Sep 11 14:53:25.000565 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:25.000534    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-bf44e879-kserve-67dc6b47c9-m4pmh"]
Sep 11 14:53:25.006091 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:25.006062    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-bf44e879-kserve-67dc6b47c9-m4pmh"]
Sep 11 14:53:25.499137 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:25.499103    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45c6d388-acf7-4db2-bac6-b03afc3a58b0" path="/var/lib/kubelet/pods/45c6d388-acf7-4db2-bac6-b03afc3a58b0/volumes"
Sep 11 14:53:26.536331 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:26.536264    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-bf44e879-kserve-586f486895-292qh" podUID="046c2c1b-7022-4e06-9730-cebaebc98014" containerName="main" containerID="cri-o://6f6700caa463007c7e70a4f236ffb3b3250747ad27d88a01107ebbd1dc6a975d" gracePeriod=45
Sep 11 14:53:26.785108 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:26.785086    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-bf44e879-kserve-586f486895-292qh"
Sep 11 14:53:26.891211 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:26.891113    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/046c2c1b-7022-4e06-9730-cebaebc98014-home\") pod \"046c2c1b-7022-4e06-9730-cebaebc98014\" (UID: \"046c2c1b-7022-4e06-9730-cebaebc98014\") "
Sep 11 14:53:26.891211 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:26.891164    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/046c2c1b-7022-4e06-9730-cebaebc98014-tmp-dir\") pod \"046c2c1b-7022-4e06-9730-cebaebc98014\" (UID: \"046c2c1b-7022-4e06-9730-cebaebc98014\") "
Sep 11 14:53:26.891454 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:26.891238    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-69kvk\" (UniqueName: \"kubernetes.io/projected/046c2c1b-7022-4e06-9730-cebaebc98014-kube-api-access-69kvk\") pod \"046c2c1b-7022-4e06-9730-cebaebc98014\" (UID: \"046c2c1b-7022-4e06-9730-cebaebc98014\") "
Sep 11 14:53:26.891454 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:26.891265    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/046c2c1b-7022-4e06-9730-cebaebc98014-tls-certs\") pod \"046c2c1b-7022-4e06-9730-cebaebc98014\" (UID: \"046c2c1b-7022-4e06-9730-cebaebc98014\") "
Sep 11 14:53:26.891454 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:26.891284    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/046c2c1b-7022-4e06-9730-cebaebc98014-model-cache\") pod \"046c2c1b-7022-4e06-9730-cebaebc98014\" (UID: \"046c2c1b-7022-4e06-9730-cebaebc98014\") "
Sep 11 14:53:26.891454 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:26.891404    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/046c2c1b-7022-4e06-9730-cebaebc98014-dshm\") pod \"046c2c1b-7022-4e06-9730-cebaebc98014\" (UID: \"046c2c1b-7022-4e06-9730-cebaebc98014\") "
Sep 11 14:53:26.891454 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:26.891410    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/046c2c1b-7022-4e06-9730-cebaebc98014-home" (OuterVolumeSpecName: "home") pod "046c2c1b-7022-4e06-9730-cebaebc98014" (UID: "046c2c1b-7022-4e06-9730-cebaebc98014"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:53:26.891694 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:26.891493    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/046c2c1b-7022-4e06-9730-cebaebc98014-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "046c2c1b-7022-4e06-9730-cebaebc98014" (UID: "046c2c1b-7022-4e06-9730-cebaebc98014"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:53:26.891694 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:26.891505    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/046c2c1b-7022-4e06-9730-cebaebc98014-model-cache" (OuterVolumeSpecName: "model-cache") pod "046c2c1b-7022-4e06-9730-cebaebc98014" (UID: "046c2c1b-7022-4e06-9730-cebaebc98014"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:53:26.891764 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:26.891708    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/046c2c1b-7022-4e06-9730-cebaebc98014-home\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:53:26.891764 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:26.891724    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/046c2c1b-7022-4e06-9730-cebaebc98014-tmp-dir\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:53:26.891764 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:26.891739    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/046c2c1b-7022-4e06-9730-cebaebc98014-model-cache\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:53:26.893393 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:26.893372    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/046c2c1b-7022-4e06-9730-cebaebc98014-dshm" (OuterVolumeSpecName: "dshm") pod "046c2c1b-7022-4e06-9730-cebaebc98014" (UID: "046c2c1b-7022-4e06-9730-cebaebc98014"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 14:53:26.893465 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:26.893420    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/046c2c1b-7022-4e06-9730-cebaebc98014-kube-api-access-69kvk" (OuterVolumeSpecName: "kube-api-access-69kvk") pod "046c2c1b-7022-4e06-9730-cebaebc98014" (UID: "046c2c1b-7022-4e06-9730-cebaebc98014"). InnerVolumeSpecName "kube-api-access-69kvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 14:53:26.893519 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:26.893478    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/046c2c1b-7022-4e06-9730-cebaebc98014-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "046c2c1b-7022-4e06-9730-cebaebc98014" (UID: "046c2c1b-7022-4e06-9730-cebaebc98014"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 14:53:26.977777 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:26.977740    2581 generic.go:356] "Generic (PLEG): container finished" podID="046c2c1b-7022-4e06-9730-cebaebc98014" containerID="6f6700caa463007c7e70a4f236ffb3b3250747ad27d88a01107ebbd1dc6a975d" exitCode=0
Sep 11 14:53:26.978005 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:26.977788    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-bf44e879-kserve-586f486895-292qh"
Sep 11 14:53:26.978005 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:26.977805    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-bf44e879-kserve-586f486895-292qh" event={"ID":"046c2c1b-7022-4e06-9730-cebaebc98014","Type":"ContainerDied","Data":"6f6700caa463007c7e70a4f236ffb3b3250747ad27d88a01107ebbd1dc6a975d"}
Sep 11 14:53:26.978005 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:26.977836    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-bf44e879-kserve-586f486895-292qh" event={"ID":"046c2c1b-7022-4e06-9730-cebaebc98014","Type":"ContainerDied","Data":"8d9c5ecdbecbaec1391b130853ae0cd37ab87924946788110fa321d261540d7a"}
Sep 11 14:53:26.978005 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:26.977852    2581 scope.go:117] "RemoveContainer" containerID="6f6700caa463007c7e70a4f236ffb3b3250747ad27d88a01107ebbd1dc6a975d"
Sep 11 14:53:26.986334 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:26.986314    2581 scope.go:117] "RemoveContainer" containerID="6f6700caa463007c7e70a4f236ffb3b3250747ad27d88a01107ebbd1dc6a975d"
Sep 11 14:53:26.986580 ip-10-0-135-150 kubenswrapper[2581]: E0911 14:53:26.986564    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f6700caa463007c7e70a4f236ffb3b3250747ad27d88a01107ebbd1dc6a975d\": container with ID starting with 6f6700caa463007c7e70a4f236ffb3b3250747ad27d88a01107ebbd1dc6a975d not found: ID does not exist" containerID="6f6700caa463007c7e70a4f236ffb3b3250747ad27d88a01107ebbd1dc6a975d"
Sep 11 14:53:26.986622 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:26.986588    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f6700caa463007c7e70a4f236ffb3b3250747ad27d88a01107ebbd1dc6a975d"} err="failed to get container status \"6f6700caa463007c7e70a4f236ffb3b3250747ad27d88a01107ebbd1dc6a975d\": rpc error: code = NotFound desc = could not find container \"6f6700caa463007c7e70a4f236ffb3b3250747ad27d88a01107ebbd1dc6a975d\": container with ID starting with 6f6700caa463007c7e70a4f236ffb3b3250747ad27d88a01107ebbd1dc6a975d not found: ID does not exist"
Sep 11 14:53:26.992557 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:26.992541    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69kvk\" (UniqueName: \"kubernetes.io/projected/046c2c1b-7022-4e06-9730-cebaebc98014-kube-api-access-69kvk\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:53:26.992627 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:26.992559    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/046c2c1b-7022-4e06-9730-cebaebc98014-tls-certs\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:53:26.992627 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:26.992569    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/046c2c1b-7022-4e06-9730-cebaebc98014-dshm\") on node \"ip-10-0-135-150.ec2.internal\" DevicePath \"\""
Sep 11 14:53:27.001662 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:27.001639    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-bf44e879-kserve-586f486895-292qh"]
Sep 11 14:53:27.009335 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:27.009310    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-bf44e879-kserve-586f486895-292qh"]
Sep 11 14:53:27.504449 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:27.504416    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="046c2c1b-7022-4e06-9730-cebaebc98014" path="/var/lib/kubelet/pods/046c2c1b-7022-4e06-9730-cebaebc98014/volumes"
Sep 11 14:53:57.892320 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:57.892230    2581 scope.go:117] "RemoveContainer" containerID="fd3c1a39752dca882af405f658acf36d09afce852b1080102f2e729567ce72c8"
Sep 11 14:53:57.900383 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:57.900366    2581 scope.go:117] "RemoveContainer" containerID="a6b550f26678a67b7d1395ecf07d554cf6d63e4e11c6be35a7c7d294c54ec3d6"
Sep 11 14:53:57.907729 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:53:57.907707    2581 scope.go:117] "RemoveContainer" containerID="ae5679e4c1c00dc5d2be51890d214a340ba9366e755f288f33a5dfa8c32575c7"
Sep 11 14:54:07.512456 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:07.512427    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-v2qsw_f6ded6ea-11f1-49f8-bd47-cc2fe4016904/manager/1.log"
Sep 11 14:54:07.636072 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:07.636044    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d8f96d84b-4lss9_7897ae2e-9365-4a43-95a5-6206a1f8d0e0/copy-manifests/0.log"
Sep 11 14:54:07.644663 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:07.644635    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d8f96d84b-4lss9_7897ae2e-9365-4a43-95a5-6206a1f8d0e0/manager/0.log"
Sep 11 14:54:08.652237 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:08.652206    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29818965-vtp8m_36ffa317-c41c-4b70-9bab-39865d4fd0e8/cleanup/0.log"
Sep 11 14:54:10.810150 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:10.810108    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct_1fa7ea94-b71e-4e6f-98c8-68bd8eb103e7/istio-proxy/0.log"
Sep 11 14:54:11.058109 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:11.058080    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-57f955fb8f-j97rf_a9d0d8dc-ecfc-4fc7-89a0-7ae86ca0ff57/kube-auth-proxy/0.log"
Sep 11 14:54:11.331651 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:11.331621    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-hxbs8_2cc78453-02ad-45d4-8efe-b247d33d55d4/istio-proxy/1.log"
Sep 11 14:54:12.227728 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:12.227700    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-pjmfg_f769f831-73e9-4992-a406-b5aebd8a37e1/main/0.log"
Sep 11 14:54:12.353530 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:12.353493    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-9ffx4_c677d32f-fae0-46b8-a9b0-de4f57a388ec/main/0.log"
Sep 11 14:54:12.851054 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:12.851024    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-fz7tk_c1d188ff-0917-4693-b832-cd5b85c76c72/main/0.log"
Sep 11 14:54:18.763565 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:18.763530    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jc5ln/must-gather-qdjqq"]
Sep 11 14:54:18.763963 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:18.763852    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45c6d388-acf7-4db2-bac6-b03afc3a58b0" containerName="main"
Sep 11 14:54:18.763963 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:18.763863    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="45c6d388-acf7-4db2-bac6-b03afc3a58b0" containerName="main"
Sep 11 14:54:18.763963 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:18.763879    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="046c2c1b-7022-4e06-9730-cebaebc98014" containerName="main"
Sep 11 14:54:18.763963 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:18.763885    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="046c2c1b-7022-4e06-9730-cebaebc98014" containerName="main"
Sep 11 14:54:18.763963 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:18.763940    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="45c6d388-acf7-4db2-bac6-b03afc3a58b0" containerName="main"
Sep 11 14:54:18.763963 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:18.763948    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="046c2c1b-7022-4e06-9730-cebaebc98014" containerName="main"
Sep 11 14:54:18.767112 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:18.767094    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jc5ln/must-gather-qdjqq"
Sep 11 14:54:18.771038 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:18.771016    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jc5ln\"/\"kube-root-ca.crt\""
Sep 11 14:54:18.771799 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:18.771782    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jc5ln\"/\"default-dockercfg-pzsf7\""
Sep 11 14:54:18.771901 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:18.771816    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jc5ln\"/\"openshift-service-ca.crt\""
Sep 11 14:54:18.782143 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:18.782076    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jc5ln/must-gather-qdjqq"]
Sep 11 14:54:18.860521 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:18.860478    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62w5d\" (UniqueName: \"kubernetes.io/projected/169255fd-caea-46bd-9210-541b33ff2ec8-kube-api-access-62w5d\") pod \"must-gather-qdjqq\" (UID: \"169255fd-caea-46bd-9210-541b33ff2ec8\") " pod="openshift-must-gather-jc5ln/must-gather-qdjqq"
Sep 11 14:54:18.860715 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:18.860601    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/169255fd-caea-46bd-9210-541b33ff2ec8-must-gather-output\") pod \"must-gather-qdjqq\" (UID: \"169255fd-caea-46bd-9210-541b33ff2ec8\") " pod="openshift-must-gather-jc5ln/must-gather-qdjqq"
Sep 11 14:54:18.961834 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:18.961795    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62w5d\" (UniqueName: \"kubernetes.io/projected/169255fd-caea-46bd-9210-541b33ff2ec8-kube-api-access-62w5d\") pod \"must-gather-qdjqq\" (UID: \"169255fd-caea-46bd-9210-541b33ff2ec8\") " pod="openshift-must-gather-jc5ln/must-gather-qdjqq"
Sep 11 14:54:18.962100 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:18.961889    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/169255fd-caea-46bd-9210-541b33ff2ec8-must-gather-output\") pod \"must-gather-qdjqq\" (UID: \"169255fd-caea-46bd-9210-541b33ff2ec8\") " pod="openshift-must-gather-jc5ln/must-gather-qdjqq"
Sep 11 14:54:18.962243 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:18.962225    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/169255fd-caea-46bd-9210-541b33ff2ec8-must-gather-output\") pod \"must-gather-qdjqq\" (UID: \"169255fd-caea-46bd-9210-541b33ff2ec8\") " pod="openshift-must-gather-jc5ln/must-gather-qdjqq"
Sep 11 14:54:18.977284 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:18.977257    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62w5d\" (UniqueName: \"kubernetes.io/projected/169255fd-caea-46bd-9210-541b33ff2ec8-kube-api-access-62w5d\") pod \"must-gather-qdjqq\" (UID: \"169255fd-caea-46bd-9210-541b33ff2ec8\") " pod="openshift-must-gather-jc5ln/must-gather-qdjqq"
Sep 11 14:54:19.075708 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:19.075625    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jc5ln/must-gather-qdjqq"
Sep 11 14:54:19.352176 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:19.352119    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jc5ln/must-gather-qdjqq"]
Sep 11 14:54:19.355290 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:54:19.355261    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod169255fd_caea_46bd_9210_541b33ff2ec8.slice/crio-2eda3d3393adc2549ee6dda29a1eff5f61a9b3ff986f00ba150d63379bd4eaad WatchSource:0}: Error finding container 2eda3d3393adc2549ee6dda29a1eff5f61a9b3ff986f00ba150d63379bd4eaad: Status 404 returned error can't find the container with id 2eda3d3393adc2549ee6dda29a1eff5f61a9b3ff986f00ba150d63379bd4eaad
Sep 11 14:54:19.357902 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:19.357882    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 14:54:20.231757 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:20.231548    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jc5ln/must-gather-qdjqq" event={"ID":"169255fd-caea-46bd-9210-541b33ff2ec8","Type":"ContainerStarted","Data":"ea7c4cb7ffd55ffa899dc1206be8ccd928292adbce7bfd06a1ead9283c40071b"}
Sep 11 14:54:20.231757 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:20.231608    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jc5ln/must-gather-qdjqq" event={"ID":"169255fd-caea-46bd-9210-541b33ff2ec8","Type":"ContainerStarted","Data":"a4228e620a19cd255ebf763a6f3fbfd3736e7565b1d3a7bff1e77104da59ddb8"}
Sep 11 14:54:20.231757 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:20.231627    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jc5ln/must-gather-qdjqq" event={"ID":"169255fd-caea-46bd-9210-541b33ff2ec8","Type":"ContainerStarted","Data":"2eda3d3393adc2549ee6dda29a1eff5f61a9b3ff986f00ba150d63379bd4eaad"}
Sep 11 14:54:20.256624 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:20.256517    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jc5ln/must-gather-qdjqq" podStartSLOduration=1.615974952 podStartE2EDuration="2.256502069s" totalImagesPullingTime="640.527117ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:54:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:54:20.254203945 +0000 UTC m=+2363.404754021" watchObservedRunningTime="2026-09-11 14:54:20.256502069 +0000 UTC m=+2363.407052133"
Sep 11 14:54:21.836222 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:21.836172    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-dqmtt_78d2df0e-635a-4123-9a80-f9898562e613/global-pull-secret-syncer/0.log"
Sep 11 14:54:21.987459 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:21.987407    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-pnxhk_c81645fc-5b10-41bb-9908-7512097108e8/konnectivity-agent/0.log"
Sep 11 14:54:22.075621 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:22.075580    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-150.ec2.internal_9bd85cdb5fbcf445a1156f9ccc86edcd/haproxy/0.log"
Sep 11 14:54:28.748029 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:28.747992    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7b8fc55444-8rwqg_ceed54bd-5296-4185-9bc9-53a732bbd63f/metrics-server/0.log"
Sep 11 14:54:29.045997 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:29.045933    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zwkxm_e8ad0979-ef70-493c-8e4f-8a96a151dd79/node-exporter/0.log"
Sep 11 14:54:29.072812 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:29.072783    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zwkxm_e8ad0979-ef70-493c-8e4f-8a96a151dd79/kube-rbac-proxy/0.log"
Sep 11 14:54:29.102550 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:29.102511    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zwkxm_e8ad0979-ef70-493c-8e4f-8a96a151dd79/init-textfile/0.log"
Sep 11 14:54:29.761650 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:29.761617    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-549f9d7d48-t7dbk_03849958-e526-46c3-a90c-fd764025d2f2/thanos-query/0.log"
Sep 11 14:54:29.788643 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:29.788608    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-549f9d7d48-t7dbk_03849958-e526-46c3-a90c-fd764025d2f2/kube-rbac-proxy-web/0.log"
Sep 11 14:54:29.817781 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:29.817750    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-549f9d7d48-t7dbk_03849958-e526-46c3-a90c-fd764025d2f2/kube-rbac-proxy/0.log"
Sep 11 14:54:29.845078 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:29.845047    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-549f9d7d48-t7dbk_03849958-e526-46c3-a90c-fd764025d2f2/prom-label-proxy/0.log"
Sep 11 14:54:29.875133 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:29.875100    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-549f9d7d48-t7dbk_03849958-e526-46c3-a90c-fd764025d2f2/kube-rbac-proxy-rules/0.log"
Sep 11 14:54:29.903139 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:29.903014    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-549f9d7d48-t7dbk_03849958-e526-46c3-a90c-fd764025d2f2/kube-rbac-proxy-metrics/0.log"
Sep 11 14:54:31.684822 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:31.684776    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jc5ln/perf-node-gather-daemonset-4fgvx"]
Sep 11 14:54:31.693041 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:31.693007    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-4fgvx"
Sep 11 14:54:31.700615 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:31.700575    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jc5ln/perf-node-gather-daemonset-4fgvx"]
Sep 11 14:54:31.801524 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:31.801481    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d1e5d136-d43d-4ac1-90ef-25a781a89992-podres\") pod \"perf-node-gather-daemonset-4fgvx\" (UID: \"d1e5d136-d43d-4ac1-90ef-25a781a89992\") " pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-4fgvx"
Sep 11 14:54:31.801787 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:31.801766    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d1e5d136-d43d-4ac1-90ef-25a781a89992-proc\") pod \"perf-node-gather-daemonset-4fgvx\" (UID: \"d1e5d136-d43d-4ac1-90ef-25a781a89992\") " pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-4fgvx"
Sep 11 14:54:31.802089 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:31.802065    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pxtt\" (UniqueName: \"kubernetes.io/projected/d1e5d136-d43d-4ac1-90ef-25a781a89992-kube-api-access-8pxtt\") pod \"perf-node-gather-daemonset-4fgvx\" (UID: \"d1e5d136-d43d-4ac1-90ef-25a781a89992\") " pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-4fgvx"
Sep 11 14:54:31.802242 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:31.802225    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d1e5d136-d43d-4ac1-90ef-25a781a89992-lib-modules\") pod \"perf-node-gather-daemonset-4fgvx\" (UID: \"d1e5d136-d43d-4ac1-90ef-25a781a89992\") " pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-4fgvx"
Sep 11 14:54:31.802358 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:31.802345    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d1e5d136-d43d-4ac1-90ef-25a781a89992-sys\") pod \"perf-node-gather-daemonset-4fgvx\" (UID: \"d1e5d136-d43d-4ac1-90ef-25a781a89992\") " pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-4fgvx"
Sep 11 14:54:31.903730 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:31.903693    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d1e5d136-d43d-4ac1-90ef-25a781a89992-lib-modules\") pod \"perf-node-gather-daemonset-4fgvx\" (UID: \"d1e5d136-d43d-4ac1-90ef-25a781a89992\") " pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-4fgvx"
Sep 11 14:54:31.903730 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:31.903736    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d1e5d136-d43d-4ac1-90ef-25a781a89992-sys\") pod \"perf-node-gather-daemonset-4fgvx\" (UID: \"d1e5d136-d43d-4ac1-90ef-25a781a89992\") " pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-4fgvx"
Sep 11 14:54:31.904029 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:31.903791    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d1e5d136-d43d-4ac1-90ef-25a781a89992-podres\") pod \"perf-node-gather-daemonset-4fgvx\" (UID: \"d1e5d136-d43d-4ac1-90ef-25a781a89992\") " pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-4fgvx"
Sep 11 14:54:31.904029 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:31.903809    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d1e5d136-d43d-4ac1-90ef-25a781a89992-proc\") pod \"perf-node-gather-daemonset-4fgvx\" (UID: \"d1e5d136-d43d-4ac1-90ef-25a781a89992\") " pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-4fgvx"
Sep 11 14:54:31.904029 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:31.903840    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pxtt\" (UniqueName: \"kubernetes.io/projected/d1e5d136-d43d-4ac1-90ef-25a781a89992-kube-api-access-8pxtt\") pod \"perf-node-gather-daemonset-4fgvx\" (UID: \"d1e5d136-d43d-4ac1-90ef-25a781a89992\") " pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-4fgvx"
Sep 11 14:54:31.904029 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:31.903896    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d1e5d136-d43d-4ac1-90ef-25a781a89992-lib-modules\") pod \"perf-node-gather-daemonset-4fgvx\" (UID: \"d1e5d136-d43d-4ac1-90ef-25a781a89992\") " pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-4fgvx"
Sep 11 14:54:31.904029 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:31.903926    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d1e5d136-d43d-4ac1-90ef-25a781a89992-sys\") pod \"perf-node-gather-daemonset-4fgvx\" (UID: \"d1e5d136-d43d-4ac1-90ef-25a781a89992\") " pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-4fgvx"
Sep 11 14:54:31.904029 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:31.903942    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d1e5d136-d43d-4ac1-90ef-25a781a89992-proc\") pod \"perf-node-gather-daemonset-4fgvx\" (UID: \"d1e5d136-d43d-4ac1-90ef-25a781a89992\") " pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-4fgvx"
Sep 11 14:54:31.904029 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:31.903957    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d1e5d136-d43d-4ac1-90ef-25a781a89992-podres\") pod \"perf-node-gather-daemonset-4fgvx\" (UID: \"d1e5d136-d43d-4ac1-90ef-25a781a89992\") " pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-4fgvx"
Sep 11 14:54:31.917821 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:31.917784    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pxtt\" (UniqueName: \"kubernetes.io/projected/d1e5d136-d43d-4ac1-90ef-25a781a89992-kube-api-access-8pxtt\") pod \"perf-node-gather-daemonset-4fgvx\" (UID: \"d1e5d136-d43d-4ac1-90ef-25a781a89992\") " pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-4fgvx"
Sep 11 14:54:32.009490 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:32.009449    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-4fgvx"
Sep 11 14:54:32.319795 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:32.319754    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jc5ln/perf-node-gather-daemonset-4fgvx"]
Sep 11 14:54:32.323322 ip-10-0-135-150 kubenswrapper[2581]: W0911 14:54:32.323281    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd1e5d136_d43d_4ac1_90ef_25a781a89992.slice/crio-81ee671a5fefe6eb37190ebc004a76506739cf7556b716ad2f5e1e68df26e2cc WatchSource:0}: Error finding container 81ee671a5fefe6eb37190ebc004a76506739cf7556b716ad2f5e1e68df26e2cc: Status 404 returned error can't find the container with id 81ee671a5fefe6eb37190ebc004a76506739cf7556b716ad2f5e1e68df26e2cc
Sep 11 14:54:32.334551 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:32.334514    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-4fgvx" event={"ID":"d1e5d136-d43d-4ac1-90ef-25a781a89992","Type":"ContainerStarted","Data":"81ee671a5fefe6eb37190ebc004a76506739cf7556b716ad2f5e1e68df26e2cc"}
Sep 11 14:54:32.547571 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:32.547507    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-76d48cb76f-rckgx_91e6b88f-64e0-4141-bd4d-eac6ad6e13cb/console/0.log"
Sep 11 14:54:33.343793 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:33.343747    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-4fgvx" event={"ID":"d1e5d136-d43d-4ac1-90ef-25a781a89992","Type":"ContainerStarted","Data":"338f45d8cc22a85aebe709681a2deaa6c1de21444c9fb557634682919b91c683"}
Sep 11 14:54:33.344493 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:33.344464    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-4fgvx"
Sep 11 14:54:33.368261 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:33.368210    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-4fgvx" podStartSLOduration=2.36819206 podStartE2EDuration="2.36819206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:54:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:54:33.367275548 +0000 UTC m=+2376.517825606" watchObservedRunningTime="2026-09-11 14:54:33.36819206 +0000 UTC m=+2376.518742123"
Sep 11 14:54:34.686573 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:34.686542    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gdl2n_98689afa-2179-4240-9ad9-cbe7744a84c1/dns/0.log"
Sep 11 14:54:34.729700 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:34.729672    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gdl2n_98689afa-2179-4240-9ad9-cbe7744a84c1/kube-rbac-proxy/0.log"
Sep 11 14:54:35.032703 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:35.032661    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wk6kn_757db884-d01a-4ab4-8b46-f9a41a3d7369/dns-node-resolver/0.log"
Sep 11 14:54:35.863559 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:35.863522    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qdthk_2b4e57a8-379d-4375-95b4-8176b1e4e410/node-ca/0.log"
Sep 11 14:54:36.987750 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:36.987715    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct_1fa7ea94-b71e-4e6f-98c8-68bd8eb103e7/istio-proxy/0.log"
Sep 11 14:54:37.353914 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:37.353886    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-57f955fb8f-j97rf_a9d0d8dc-ecfc-4fc7-89a0-7ae86ca0ff57/kube-auth-proxy/0.log"
Sep 11 14:54:37.436612 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:37.436575    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-hxbs8_2cc78453-02ad-45d4-8efe-b247d33d55d4/istio-proxy/0.log"
Sep 11 14:54:37.447610 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:37.447579    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-hxbs8_2cc78453-02ad-45d4-8efe-b247d33d55d4/istio-proxy/1.log"
Sep 11 14:54:38.497093 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:38.497046    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5s8kg_7aefb1ab-dcdc-451e-b710-81a121895035/serve-healthcheck-canary/0.log"
Sep 11 14:54:39.306355 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:39.306322    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-psv5r_0e83e20c-2474-4cb0-9a1c-f7025b7383c5/kube-rbac-proxy/0.log"
Sep 11 14:54:39.333090 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:39.333062    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-psv5r_0e83e20c-2474-4cb0-9a1c-f7025b7383c5/exporter/0.log"
Sep 11 14:54:39.362412 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:39.362377    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-psv5r_0e83e20c-2474-4cb0-9a1c-f7025b7383c5/extractor/0.log"
Sep 11 14:54:40.372505 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:40.372473    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-jc5ln/perf-node-gather-daemonset-4fgvx"
Sep 11 14:54:41.965228 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:41.965197    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct_1fa7ea94-b71e-4e6f-98c8-68bd8eb103e7/istio-proxy/0.log"
Sep 11 14:54:42.124584 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:42.124547    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-v2qsw_f6ded6ea-11f1-49f8-bd47-cc2fe4016904/manager/0.log"
Sep 11 14:54:42.154813 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:42.154777    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-v2qsw_f6ded6ea-11f1-49f8-bd47-cc2fe4016904/manager/1.log"
Sep 11 14:54:42.192007 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:42.191954    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d8f96d84b-4lss9_7897ae2e-9365-4a43-95a5-6206a1f8d0e0/manager/0.log"
Sep 11 14:54:42.220344 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:42.220320    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d8f96d84b-4lss9_7897ae2e-9365-4a43-95a5-6206a1f8d0e0/copy-manifests/0.log"
Sep 11 14:54:42.333632 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:42.333597    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-57f955fb8f-j97rf_a9d0d8dc-ecfc-4fc7-89a0-7ae86ca0ff57/kube-auth-proxy/0.log"
Sep 11 14:54:42.393884 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:42.393843    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-hxbs8_2cc78453-02ad-45d4-8efe-b247d33d55d4/istio-proxy/0.log"
Sep 11 14:54:42.406401 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:42.406364    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-hxbs8_2cc78453-02ad-45d4-8efe-b247d33d55d4/istio-proxy/1.log"
Sep 11 14:54:43.732348 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:43.732314    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-744bf4ff48-8rn2q_f0948f34-1747-491e-b499-fe02725bc43c/manager/1.log"
Sep 11 14:54:43.732812 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:43.732620    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-744bf4ff48-8rn2q_f0948f34-1747-491e-b499-fe02725bc43c/manager/0.log"
Sep 11 14:54:43.789578 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:43.789529    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-585fl_2332a63b-b2db-48ec-9185-d8bcd25cf754/openshift-lws-operator/0.log"
Sep 11 14:54:44.116619 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:44.116589    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-4l4ct_1fa7ea94-b71e-4e6f-98c8-68bd8eb103e7/istio-proxy/0.log"
Sep 11 14:54:44.456963 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:44.456888    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-57f955fb8f-j97rf_a9d0d8dc-ecfc-4fc7-89a0-7ae86ca0ff57/kube-auth-proxy/0.log"
Sep 11 14:54:44.516625 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:44.516586    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-hxbs8_2cc78453-02ad-45d4-8efe-b247d33d55d4/istio-proxy/0.log"
Sep 11 14:54:44.527341 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:44.527305    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-hxbs8_2cc78453-02ad-45d4-8efe-b247d33d55d4/istio-proxy/1.log"
Sep 11 14:54:49.584395 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:49.584357    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-z7cxh_7dd4a673-2bd6-4dd4-a23f-f510ecb8793f/kube-storage-version-migrator-operator/1.log"
Sep 11 14:54:49.585370 ip-10-0-135-150 kubenswrapper[2581]: I0911 14:54:49.585347    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-z7cxh_7dd4a673-2bd6-4dd4-a23f-f510ecb8793f/kube-storage-version-migrator-operator/0.log"