May 25 12:13:06.920890 ip-10-0-134-255 systemd[1]: Starting Kubernetes Kubelet...
May 25 12:13:07.355826 ip-10-0-134-255 kubenswrapper[2555]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 25 12:13:07.355826 ip-10-0-134-255 kubenswrapper[2555]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
May 25 12:13:07.355826 ip-10-0-134-255 kubenswrapper[2555]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 25 12:13:07.355826 ip-10-0-134-255 kubenswrapper[2555]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
May 25 12:13:07.355826 ip-10-0-134-255 kubenswrapper[2555]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 25 12:13:07.358305 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.358221    2555 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
May 25 12:13:07.366120 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366097    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 12:13:07.366120 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366115    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 12:13:07.366120 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366119    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 12:13:07.366120 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366122    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 12:13:07.366120 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366125    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 12:13:07.366120 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366127    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 12:13:07.366327 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366131    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 12:13:07.366327 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366135    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 12:13:07.366327 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366138    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 12:13:07.366327 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366141    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 12:13:07.366327 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366143    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 12:13:07.366327 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366146    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 12:13:07.366327 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366149    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 12:13:07.366327 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366151    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 12:13:07.366327 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366154    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 12:13:07.366327 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366156    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 12:13:07.366327 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366159    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 12:13:07.366327 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366161    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 12:13:07.366327 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366164    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 12:13:07.366327 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366167    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 12:13:07.366327 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366169    2555 feature_gate.go:328] unrecognized feature gate: Example
May 25 12:13:07.366327 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366172    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 12:13:07.366327 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366174    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 12:13:07.366327 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366177    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 12:13:07.366327 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366180    2555 feature_gate.go:328] unrecognized feature gate: Example2
May 25 12:13:07.366327 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366182    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 12:13:07.366805 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366188    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 12:13:07.366805 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366190    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 12:13:07.366805 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366193    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 12:13:07.366805 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366196    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 12:13:07.366805 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366198    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 12:13:07.366805 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366201    2555 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 12:13:07.366805 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366203    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 12:13:07.366805 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366206    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 12:13:07.366805 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366209    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 12:13:07.366805 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366211    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 12:13:07.366805 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366214    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 12:13:07.366805 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366216    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 12:13:07.366805 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366219    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 12:13:07.366805 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366222    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 12:13:07.366805 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366225    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 12:13:07.366805 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366233    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 12:13:07.366805 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366236    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 12:13:07.366805 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366239    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 12:13:07.366805 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366241    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 12:13:07.366805 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366245    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 12:13:07.367283 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366247    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 12:13:07.367283 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366250    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 12:13:07.367283 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366253    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 12:13:07.367283 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366255    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 12:13:07.367283 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366257    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 12:13:07.367283 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366260    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 12:13:07.367283 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366262    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 12:13:07.367283 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366264    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 12:13:07.367283 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366268    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 12:13:07.367283 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366270    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 12:13:07.367283 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366272    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 12:13:07.367283 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366275    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 12:13:07.367283 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366277    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 12:13:07.367283 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366280    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 12:13:07.367283 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366282    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 12:13:07.367283 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366285    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 12:13:07.367283 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366287    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 12:13:07.367283 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366290    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 12:13:07.367283 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366292    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 12:13:07.367283 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366294    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 12:13:07.367759 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366298    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 12:13:07.367759 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366302    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 12:13:07.367759 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366304    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 12:13:07.367759 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366307    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 12:13:07.367759 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366309    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 12:13:07.367759 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366313    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 12:13:07.367759 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366316    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 12:13:07.367759 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366319    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 12:13:07.367759 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366321    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 12:13:07.367759 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366324    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 12:13:07.367759 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366326    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 12:13:07.367759 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366329    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 12:13:07.367759 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366332    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 12:13:07.367759 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366335    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 12:13:07.367759 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366337    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 12:13:07.367759 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366340    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 12:13:07.367759 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366343    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 12:13:07.367759 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366348    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 12:13:07.367759 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366352    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 12:13:07.368202 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.366355    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 12:13:07.368202 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.367861    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 12:13:07.368202 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.367869    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 12:13:07.368202 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.367873    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 12:13:07.368202 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.367876    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 12:13:07.368202 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.367879    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 12:13:07.368202 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.367882    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 12:13:07.368202 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.367885    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 12:13:07.368202 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.367887    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 12:13:07.368202 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.367890    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 12:13:07.368202 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.367892    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 12:13:07.368202 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.367895    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 12:13:07.368202 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.367897    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 12:13:07.368202 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.367900    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 12:13:07.368202 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.367903    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 12:13:07.368202 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.367905    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 12:13:07.368202 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.367907    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 12:13:07.368202 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.367910    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 12:13:07.368202 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.367912    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 12:13:07.368202 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.367915    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 12:13:07.368672 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.367917    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 12:13:07.368672 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.367920    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 12:13:07.368672 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.367922    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 12:13:07.368672 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.367925    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 12:13:07.368672 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.367927    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 12:13:07.368672 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.367930    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 12:13:07.368672 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.367932    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 12:13:07.368672 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.367935    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 12:13:07.368672 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.367938    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 12:13:07.368672 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.367940    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 12:13:07.368672 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.367942    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 12:13:07.368672 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.367945    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 12:13:07.368672 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.367947    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 12:13:07.368672 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.367951    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 12:13:07.368672 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.367953    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 12:13:07.368672 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.367956    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 12:13:07.368672 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.367958    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 12:13:07.368672 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.367961    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 12:13:07.368672 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.367963    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 12:13:07.369214 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.367966    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 12:13:07.369214 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.367968    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 12:13:07.369214 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.367971    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 12:13:07.369214 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.367973    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 12:13:07.369214 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.367976    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 12:13:07.369214 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.367978    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 12:13:07.369214 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.367982    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 12:13:07.369214 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.367984    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 12:13:07.369214 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.367986    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 12:13:07.369214 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.367989    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 12:13:07.369214 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.367991    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 12:13:07.369214 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.367994    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 12:13:07.369214 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.367996    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 12:13:07.369214 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.367998    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 12:13:07.369214 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368001    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 12:13:07.369214 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368003    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 12:13:07.369214 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368007    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 12:13:07.369214 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368010    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 12:13:07.369214 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368014    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 12:13:07.369214 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368017    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 12:13:07.369696 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368019    2555 feature_gate.go:328] unrecognized feature gate: Example
May 25 12:13:07.369696 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368022    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 12:13:07.369696 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368024    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 12:13:07.369696 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368026    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 12:13:07.369696 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368030    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 12:13:07.369696 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368032    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 12:13:07.369696 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368035    2555 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 12:13:07.369696 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368037    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 12:13:07.369696 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368040    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 12:13:07.369696 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368042    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 12:13:07.369696 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368045    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 12:13:07.369696 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368047    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 12:13:07.369696 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368050    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 12:13:07.369696 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368052    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 12:13:07.369696 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368054    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 12:13:07.369696 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368057    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 12:13:07.369696 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368059    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 12:13:07.369696 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368062    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 12:13:07.369696 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368064    2555 feature_gate.go:328] unrecognized feature gate: Example2
May 25 12:13:07.369696 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368067    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 12:13:07.370167 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368071    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 12:13:07.370167 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368074    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 12:13:07.370167 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368077    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 12:13:07.370167 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368080    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 12:13:07.370167 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368083    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 12:13:07.370167 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368086    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 12:13:07.370167 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368089    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 12:13:07.370167 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368092    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 12:13:07.370167 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368159    2555 flags.go:64] FLAG: --address="0.0.0.0"
May 25 12:13:07.370167 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368165    2555 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
May 25 12:13:07.370167 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368172    2555 flags.go:64] FLAG: --anonymous-auth="true"
May 25 12:13:07.370167 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368176    2555 flags.go:64] FLAG: --application-metrics-count-limit="100"
May 25 12:13:07.370167 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368181    2555 flags.go:64] FLAG: --authentication-token-webhook="false"
May 25 12:13:07.370167 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368184    2555 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
May 25 12:13:07.370167 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368188    2555 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
May 25 12:13:07.370167 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368192    2555 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
May 25 12:13:07.370167 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368196    2555 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
May 25 12:13:07.370167 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368199    2555 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 25 12:13:07.370167 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368202    2555 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
May 25 12:13:07.370167 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368206    2555 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
May 25 12:13:07.370167 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368209    2555 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
May 25 12:13:07.370668 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368212    2555 flags.go:64] FLAG: --cgroup-root=""
May 25 12:13:07.370668 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368215    2555 flags.go:64] FLAG: --cgroups-per-qos="true"
May 25 12:13:07.370668 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368218    2555 flags.go:64] FLAG: --client-ca-file=""
May 25 12:13:07.370668 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368221    2555 flags.go:64] FLAG: --cloud-config=""
May 25 12:13:07.370668 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368224    2555 flags.go:64] FLAG: --cloud-provider="external"
May 25 12:13:07.370668 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368227    2555 flags.go:64] FLAG: --cluster-dns="[]"
May 25 12:13:07.370668 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368231    2555 flags.go:64] FLAG: --cluster-domain=""
May 25 12:13:07.370668 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368234    2555 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
May 25 12:13:07.370668 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368237    2555 flags.go:64] FLAG: --config-dir=""
May 25 12:13:07.370668 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368240    2555 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 25 12:13:07.370668 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368243    2555 flags.go:64] FLAG: --container-log-max-files="5"
May 25 12:13:07.370668 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368247    2555 flags.go:64] FLAG: --container-log-max-size="10Mi"
May 25 12:13:07.370668 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368250    2555 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
May 25 12:13:07.370668 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368256    2555 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
May 25 12:13:07.370668 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368259    2555 flags.go:64] FLAG: --containerd-namespace="k8s.io"
May 25 12:13:07.370668 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368262    2555 flags.go:64] FLAG: --contention-profiling="false"
May 25 12:13:07.370668 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368265    2555 flags.go:64] FLAG: --cpu-cfs-quota="true"
May 25 12:13:07.370668 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368268    2555 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
May 25 12:13:07.370668 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368271    2555 flags.go:64] FLAG: --cpu-manager-policy="none"
May 25 12:13:07.370668 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368274    2555 flags.go:64] FLAG: --cpu-manager-policy-options=""
May 25 12:13:07.370668 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368278    2555 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
May 25 12:13:07.370668 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368281    2555 flags.go:64] FLAG: --enable-controller-attach-detach="true"
May 25 12:13:07.370668 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368284    2555 flags.go:64] FLAG: --enable-debugging-handlers="true"
May 25 12:13:07.370668 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368287    2555 flags.go:64] FLAG: --enable-load-reader="false"
May 25 12:13:07.370668 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368291    2555 flags.go:64] FLAG: --enable-server="true"
May 25 12:13:07.371277 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368294    2555 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
May 25 12:13:07.371277 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368298    2555 flags.go:64] FLAG: --event-burst="100"
May 25 12:13:07.371277 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368301    2555 flags.go:64] FLAG: --event-qps="50"
May 25 12:13:07.371277 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368304    2555 flags.go:64] FLAG: --event-storage-age-limit="default=0"
May 25 12:13:07.371277 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368307    2555 flags.go:64] FLAG: --event-storage-event-limit="default=0"
May 25 12:13:07.371277 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368310    2555 flags.go:64] FLAG: --eviction-hard=""
May 25 12:13:07.371277 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368313    2555 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
May 25 12:13:07.371277 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368316    2555 flags.go:64] FLAG: --eviction-minimum-reclaim=""
May 25 12:13:07.371277 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368319    2555 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
May 25 12:13:07.371277 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368322    2555 flags.go:64] FLAG: --eviction-soft=""
May 25 12:13:07.371277 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368325    2555 flags.go:64] FLAG: --eviction-soft-grace-period=""
May 25 12:13:07.371277 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368328    2555 flags.go:64] FLAG: --exit-on-lock-contention="false"
May 25 12:13:07.371277 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368331    2555 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
May 25 12:13:07.371277 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368334    2555 flags.go:64] FLAG: --experimental-mounter-path=""
May 25 12:13:07.371277 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368337    2555 flags.go:64] FLAG: --fail-cgroupv1="false"
May 25 12:13:07.371277 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368340    2555 flags.go:64] FLAG: --fail-swap-on="true"
May 25 12:13:07.371277 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368343    2555 flags.go:64] FLAG: --feature-gates=""
May 25 12:13:07.371277 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368347    2555 flags.go:64] FLAG: --file-check-frequency="20s"
May 25 12:13:07.371277 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368350    2555 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
May 25 12:13:07.371277 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368352    2555 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
May 25 12:13:07.371277 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368357    2555 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
May 25 12:13:07.371277 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368360    2555 flags.go:64] FLAG: --healthz-port="10248"
May 25 12:13:07.371277 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368363    2555 flags.go:64] FLAG: --help="false"
May 25 12:13:07.371277 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368366    2555 flags.go:64] FLAG: --hostname-override="ip-10-0-134-255.ec2.internal"
May 25 12:13:07.371277 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368369    2555 flags.go:64] FLAG: --housekeeping-interval="10s"
May 25 12:13:07.371874 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368372    2555 flags.go:64] FLAG: --http-check-frequency="20s"
May 25 12:13:07.371874 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368375    2555 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
May 25 12:13:07.371874 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368378    2555 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
May 25 12:13:07.371874 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368382    2555 flags.go:64] FLAG: --image-gc-high-threshold="85"
May 25 12:13:07.371874 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368384    2555 flags.go:64] FLAG: --image-gc-low-threshold="80"
May 25 12:13:07.371874 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368387    2555 flags.go:64] FLAG: --image-service-endpoint=""
May 25 12:13:07.371874 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368390    2555 flags.go:64] FLAG: --kernel-memcg-notification="false"
May 25 12:13:07.371874 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368395    2555 flags.go:64] FLAG: --kube-api-burst="100"
May 25 12:13:07.371874 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368398    2555 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
May 25 12:13:07.371874 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368402    2555 flags.go:64] FLAG: --kube-api-qps="50"
May 25 12:13:07.371874 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368405    2555 flags.go:64] FLAG: --kube-reserved=""
May 25 12:13:07.371874 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368407    2555 flags.go:64] FLAG: --kube-reserved-cgroup=""
May 25 12:13:07.371874 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368410    2555 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
May 25 12:13:07.371874 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368413    2555 flags.go:64] FLAG: --kubelet-cgroups=""
May 25 12:13:07.371874 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368416    2555 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
May 25 12:13:07.371874 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368419    2555 flags.go:64] FLAG: --lock-file=""
May 25 12:13:07.371874 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368421    2555 flags.go:64] FLAG: --log-cadvisor-usage="false"
May 25 12:13:07.371874 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368424    2555 flags.go:64] FLAG: --log-flush-frequency="5s"
May 25 12:13:07.371874 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368427    2555 flags.go:64] FLAG: --log-json-info-buffer-size="0"
May 25 12:13:07.371874 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368436    2555 flags.go:64] FLAG: --log-json-split-stream="false"
May 25 12:13:07.371874 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368439    2555 flags.go:64] FLAG: --log-text-info-buffer-size="0"
May 25 12:13:07.371874 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368442    2555 flags.go:64] FLAG: --log-text-split-stream="false"
May 25 12:13:07.371874 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368444    2555 flags.go:64] FLAG: --logging-format="text"
May 25 12:13:07.372430 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368447    2555 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 25 12:13:07.372430 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368450    2555 flags.go:64] FLAG: --make-iptables-util-chains="true"
May 25 12:13:07.372430 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368453    2555 flags.go:64] FLAG: --manifest-url=""
May 25 12:13:07.372430 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368456    2555 flags.go:64] FLAG: --manifest-url-header=""
May 25 12:13:07.372430 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368461    2555 flags.go:64] FLAG: --max-housekeeping-interval="15s"
May 25 12:13:07.372430 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368464    2555 flags.go:64] FLAG: --max-open-files="1000000"
May 25 12:13:07.372430 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368468    2555 flags.go:64] FLAG: --max-pods="110"
May 25 12:13:07.372430 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368471    2555 flags.go:64] FLAG: --maximum-dead-containers="-1"
May 25 12:13:07.372430 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368474    2555 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
May 25 12:13:07.372430 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368477    2555 flags.go:64] FLAG: --memory-manager-policy="None"
May 25 12:13:07.372430 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368480    2555 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
May 25 12:13:07.372430 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368483    2555 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
May 25 12:13:07.372430 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368485    2555 flags.go:64] FLAG: --node-ip="0.0.0.0"
May 25 12:13:07.372430 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368488    2555 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
May 25 12:13:07.372430 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368495    2555 flags.go:64] FLAG: --node-status-max-images="50"
May 25 12:13:07.372430 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368498    2555 flags.go:64] FLAG: --node-status-update-frequency="10s"
May 25 12:13:07.372430 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368501    2555 flags.go:64] FLAG: --oom-score-adj="-999"
May 25 12:13:07.372430 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368504    2555 flags.go:64] FLAG: --pod-cidr=""
May 25 12:13:07.372430 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368508    2555 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc"
May 25 12:13:07.372430 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368513    2555 flags.go:64] FLAG: --pod-manifest-path=""
May 25 12:13:07.372430 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368516    2555 flags.go:64] FLAG: --pod-max-pids="-1"
May 25 12:13:07.372430 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368519    2555 flags.go:64] FLAG: --pods-per-core="0"
May 25 12:13:07.372430 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368522    2555 flags.go:64] FLAG: --port="10250"
May 25 12:13:07.372430 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368525    2555 flags.go:64] FLAG: --protect-kernel-defaults="false"
May 25 12:13:07.373018 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368528    2555 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0827f4c241cd2b8aa"
May 25 12:13:07.373018 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368531    2555 flags.go:64] FLAG: --qos-reserved=""
May 25 12:13:07.373018 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368534    2555 flags.go:64] FLAG: --read-only-port="10255"
May 25 12:13:07.373018 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368537    2555 flags.go:64] FLAG: --register-node="true"
May 25 12:13:07.373018 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368539    2555 flags.go:64] FLAG: --register-schedulable="true"
May 25 12:13:07.373018 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368542    2555 flags.go:64] FLAG: --register-with-taints=""
May 25 12:13:07.373018 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368546    2555 flags.go:64] FLAG: --registry-burst="10"
May 25 12:13:07.373018 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368548    2555 flags.go:64] FLAG: --registry-qps="5"
May 25 12:13:07.373018 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368551    2555 flags.go:64] FLAG: --reserved-cpus=""
May 25 12:13:07.373018 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368554    2555 flags.go:64] FLAG: --reserved-memory=""
May 25 12:13:07.373018 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368557    2555 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
May 25 12:13:07.373018 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368560    2555 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
May 25 12:13:07.373018 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368565    2555 flags.go:64] FLAG: --rotate-certificates="false"
May 25 12:13:07.373018 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368567    2555 flags.go:64] FLAG: --rotate-server-certificates="false"
May 25 12:13:07.373018 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368570    2555 flags.go:64] FLAG: --runonce="false"
May 25 12:13:07.373018 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368573    2555 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
May 25 12:13:07.373018 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368576    2555 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
May 25 12:13:07.373018 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368579    2555 flags.go:64] FLAG: --seccomp-default="false"
May 25 12:13:07.373018 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368581    2555 flags.go:64] FLAG: --serialize-image-pulls="true"
May 25 12:13:07.373018 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368584    2555 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
May 25 12:13:07.373018 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368587    2555 flags.go:64] FLAG: --storage-driver-db="cadvisor"
May 25 12:13:07.373018 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368590    2555 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
May 25 12:13:07.373018 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368593    2555 flags.go:64] FLAG: --storage-driver-password="root"
May 25 12:13:07.373018 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368596    2555 flags.go:64] FLAG: --storage-driver-secure="false"
May 25 12:13:07.373018 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368598    2555 flags.go:64] FLAG: --storage-driver-table="stats"
May 25 12:13:07.373018 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368601    2555 flags.go:64] FLAG: --storage-driver-user="root"
May 25 12:13:07.373662 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368604    2555 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
May 25 12:13:07.373662 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368607    2555 flags.go:64] FLAG: --sync-frequency="1m0s"
May 25 12:13:07.373662 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368610    2555 flags.go:64] FLAG: --system-cgroups=""
May 25 12:13:07.373662 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368613    2555 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
May 25 12:13:07.373662 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368618    2555 flags.go:64] FLAG: --system-reserved-cgroup=""
May 25 12:13:07.373662 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368621    2555 flags.go:64] FLAG: --tls-cert-file=""
May 25 12:13:07.373662 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368624    2555 flags.go:64] FLAG: --tls-cipher-suites="[]"
May 25 12:13:07.373662 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368628    2555 flags.go:64] FLAG: --tls-min-version=""
May 25 12:13:07.373662 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368631    2555 flags.go:64] FLAG: --tls-private-key-file=""
May 25 12:13:07.373662 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368634    2555 flags.go:64] FLAG: --topology-manager-policy="none"
May 25 12:13:07.373662 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368637    2555 flags.go:64] FLAG: --topology-manager-policy-options=""
May 25 12:13:07.373662 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368652    2555 flags.go:64] FLAG: --topology-manager-scope="container"
May 25 12:13:07.373662 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368656    2555 flags.go:64] FLAG: --v="2"
May 25 12:13:07.373662 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368660    2555 flags.go:64] FLAG: --version="false"
May 25 12:13:07.373662 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368663    2555 flags.go:64] FLAG: --vmodule=""
May 25 12:13:07.373662 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368667    2555 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
May 25 12:13:07.373662 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.368671    2555 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
May 25 12:13:07.373662 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368756    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 12:13:07.373662 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368761    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 12:13:07.373662 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368764    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 12:13:07.373662 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368766    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 12:13:07.373662 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368769    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 12:13:07.373662 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368772    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 12:13:07.373662 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368774    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 12:13:07.374210 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368777    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 12:13:07.374210 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368780    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 12:13:07.374210 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368782    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 12:13:07.374210 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368785    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 12:13:07.374210 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368787    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 12:13:07.374210 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368790    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 12:13:07.374210 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368792    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 12:13:07.374210 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368795    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 12:13:07.374210 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368799    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 12:13:07.374210 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368803    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 12:13:07.374210 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368806    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 12:13:07.374210 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368809    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 12:13:07.374210 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368812    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 12:13:07.374210 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368815    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 12:13:07.374210 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368818    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 12:13:07.374210 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368820    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 12:13:07.374210 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368823    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 12:13:07.374210 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368826    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 12:13:07.374210 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368828    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 12:13:07.374705 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368831    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 12:13:07.374705 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368833    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 12:13:07.374705 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368845    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 12:13:07.374705 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368849    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 12:13:07.374705 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368852    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 12:13:07.374705 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368854    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 12:13:07.374705 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368857    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 12:13:07.374705 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368859    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 12:13:07.374705 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368862    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 12:13:07.374705 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368865    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 12:13:07.374705 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368867    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 12:13:07.374705 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368870    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 12:13:07.374705 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368872    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 12:13:07.374705 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368875    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 12:13:07.374705 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368877    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 12:13:07.374705 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368880    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 12:13:07.374705 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368882    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 12:13:07.374705 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368885    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 12:13:07.374705 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368887    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 12:13:07.375158 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368890    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 12:13:07.375158 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368892    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 12:13:07.375158 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368894    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 12:13:07.375158 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368900    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 12:13:07.375158 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368904    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 12:13:07.375158 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368906    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 12:13:07.375158 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368909    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 12:13:07.375158 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368911    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 12:13:07.375158 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368914    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 12:13:07.375158 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368916    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 12:13:07.375158 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368919    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 12:13:07.375158 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368921    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 12:13:07.375158 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368924    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 12:13:07.375158 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368927    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 12:13:07.375158 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368929    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 12:13:07.375158 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368931    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 12:13:07.375158 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368934    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 12:13:07.375158 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368937    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 12:13:07.375158 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368939    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 12:13:07.375608 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368942    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 12:13:07.375608 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368944    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 12:13:07.375608 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368947    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 12:13:07.375608 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368949    2555 feature_gate.go:328] unrecognized feature gate: Example
May 25 12:13:07.375608 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368952    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 12:13:07.375608 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368954    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 12:13:07.375608 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368957    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 12:13:07.375608 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368959    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 12:13:07.375608 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368962    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 12:13:07.375608 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368964    2555 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 12:13:07.375608 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368967    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 12:13:07.375608 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368969    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 12:13:07.375608 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368972    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 12:13:07.375608 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368974    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 12:13:07.375608 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368977    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 12:13:07.375608 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368979    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 12:13:07.375608 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368983    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 12:13:07.375608 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368985    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 12:13:07.375608 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368988    2555 feature_gate.go:328] unrecognized feature gate: Example2
May 25 12:13:07.375608 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368991    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 12:13:07.376094 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368994    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 12:13:07.376094 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.368996    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 12:13:07.376094 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.369774    2555 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 25 12:13:07.376665 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.376636    2555 server.go:530] "Kubelet version" kubeletVersion="v1.33.11"
May 25 12:13:07.376700 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.376666    2555 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
May 25 12:13:07.376728 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376710    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 12:13:07.376728 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376716    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 12:13:07.376728 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376721    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 12:13:07.376728 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376726    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 12:13:07.376728 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376729    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 12:13:07.376864 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376733    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 12:13:07.376864 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376736    2555 feature_gate.go:328] unrecognized feature gate: Example2
May 25 12:13:07.376864 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376739    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 12:13:07.376864 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376742    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 12:13:07.376864 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376744    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 12:13:07.376864 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376747    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 12:13:07.376864 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376750    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 12:13:07.376864 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376753    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 12:13:07.376864 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376755    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 12:13:07.376864 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376758    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 12:13:07.376864 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376761    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 12:13:07.376864 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376763    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 12:13:07.376864 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376766    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 12:13:07.376864 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376769    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 12:13:07.376864 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376771    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 12:13:07.376864 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376774    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 12:13:07.376864 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376777    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 12:13:07.376864 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376779    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 12:13:07.376864 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376782    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 12:13:07.376864 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376785    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 12:13:07.377369 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376788    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 12:13:07.377369 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376790    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 12:13:07.377369 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376793    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 12:13:07.377369 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376796    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 12:13:07.377369 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376799    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 12:13:07.377369 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376801    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 12:13:07.377369 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376804    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 12:13:07.377369 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376807    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 12:13:07.377369 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376810    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 12:13:07.377369 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376812    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 12:13:07.377369 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376815    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 12:13:07.377369 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376818    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 12:13:07.377369 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376820    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 12:13:07.377369 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376823    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 12:13:07.377369 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376826    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 12:13:07.377369 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376828    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 12:13:07.377369 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376831    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 12:13:07.377369 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376833    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 12:13:07.377369 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376843    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 12:13:07.377369 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376846    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 12:13:07.377850 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376849    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 12:13:07.377850 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376851    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 12:13:07.377850 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376854    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 12:13:07.377850 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376857    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 12:13:07.377850 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376859    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 12:13:07.377850 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376862    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 12:13:07.377850 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376865    2555 feature_gate.go:328] unrecognized feature gate: Example
May 25 12:13:07.377850 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376869    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 12:13:07.377850 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376872    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 12:13:07.377850 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376875    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 12:13:07.377850 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376878    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 12:13:07.377850 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376881    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 12:13:07.377850 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376884    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 12:13:07.377850 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376887    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 12:13:07.377850 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376890    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 12:13:07.377850 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376893    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 12:13:07.377850 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376896    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 12:13:07.377850 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376899    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 12:13:07.377850 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376901    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 12:13:07.378302 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376905    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 12:13:07.378302 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376907    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 12:13:07.378302 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376910    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 12:13:07.378302 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376912    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 12:13:07.378302 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376915    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 12:13:07.378302 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376917    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 12:13:07.378302 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376920    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 12:13:07.378302 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376923    2555 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 12:13:07.378302 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376925    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 12:13:07.378302 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376928    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 12:13:07.378302 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376931    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 12:13:07.378302 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376933    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 12:13:07.378302 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376936    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 12:13:07.378302 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376938    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 12:13:07.378302 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376941    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 12:13:07.378302 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376943    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 12:13:07.378302 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376946    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 12:13:07.378302 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376948    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 12:13:07.378302 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376951    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 12:13:07.378302 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376953    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 12:13:07.378863 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376956    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 12:13:07.378863 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.376959    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 12:13:07.378863 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.376964    2555 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 25 12:13:07.378863 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377055    2555 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 12:13:07.378863 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377059    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 12:13:07.378863 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377062    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 12:13:07.378863 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377065    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 12:13:07.378863 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377068    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 12:13:07.378863 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377071    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 12:13:07.378863 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377073    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 12:13:07.378863 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377077    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 12:13:07.378863 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377079    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 12:13:07.378863 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377082    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 12:13:07.378863 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377087    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 12:13:07.378863 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377090    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 12:13:07.379223 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377093    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 12:13:07.379223 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377096    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 12:13:07.379223 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377099    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 12:13:07.379223 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377101    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 12:13:07.379223 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377104    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 12:13:07.379223 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377106    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 12:13:07.379223 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377109    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 12:13:07.379223 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377111    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 12:13:07.379223 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377114    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 12:13:07.379223 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377117    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 12:13:07.379223 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377120    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 12:13:07.379223 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377122    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 12:13:07.379223 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377125    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 12:13:07.379223 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377127    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 12:13:07.379223 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377130    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 12:13:07.379223 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377132    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 12:13:07.379223 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377135    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 12:13:07.379223 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377137    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 12:13:07.379223 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377140    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 12:13:07.379680 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377142    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 12:13:07.379680 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377145    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 12:13:07.379680 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377147    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 12:13:07.379680 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377150    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 12:13:07.379680 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377153    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 12:13:07.379680 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377155    2555 feature_gate.go:328] unrecognized feature gate: Example
May 25 12:13:07.379680 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377158    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 12:13:07.379680 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377160    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 12:13:07.379680 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377163    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 12:13:07.379680 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377166    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 12:13:07.379680 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377168    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 12:13:07.379680 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377170    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 12:13:07.379680 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377173    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 12:13:07.379680 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377176    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 12:13:07.379680 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377178    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 12:13:07.379680 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377181    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 12:13:07.379680 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377183    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 12:13:07.379680 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377185    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 12:13:07.379680 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377188    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 12:13:07.379680 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377190    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 12:13:07.380162 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377192    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 12:13:07.380162 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377195    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 12:13:07.380162 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377197    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 12:13:07.380162 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377200    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 12:13:07.380162 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377202    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 12:13:07.380162 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377204    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 12:13:07.380162 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377207    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 12:13:07.380162 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377209    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 12:13:07.380162 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377211    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 12:13:07.380162 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377214    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 12:13:07.380162 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377216    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 12:13:07.380162 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377219    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 12:13:07.380162 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377221    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 12:13:07.380162 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377224    2555 feature_gate.go:328] unrecognized feature gate: Example2
May 25 12:13:07.380162 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377226    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 12:13:07.380162 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377229    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 12:13:07.380162 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377232    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 12:13:07.380162 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377234    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 12:13:07.380162 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377236    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 12:13:07.380162 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377239    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 12:13:07.380636 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377241    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 12:13:07.380636 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377244    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 12:13:07.380636 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377246    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 12:13:07.380636 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377249    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 12:13:07.380636 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377251    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 12:13:07.380636 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377254    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 12:13:07.380636 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377256    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 12:13:07.380636 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377259    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 12:13:07.380636 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377261    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 12:13:07.380636 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377264    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 12:13:07.380636 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377266    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 12:13:07.380636 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377268    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 12:13:07.380636 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377271    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 12:13:07.380636 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377273    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 12:13:07.380636 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:07.377276    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 12:13:07.381044 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.377280    2555 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 25 12:13:07.381044 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.378163    2555 server.go:962] "Client rotation is on, will bootstrap in background"
May 25 12:13:07.381417 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.381404    2555 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
May 25 12:13:07.382458 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.382447    2555 server.go:1019] "Starting client certificate rotation"
May 25 12:13:07.382929 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.382915    2555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 25 12:13:07.382966 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.382951    2555 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 25 12:13:07.407215 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.407198    2555 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 25 12:13:07.410908 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.410891    2555 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 25 12:13:07.425497 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.425478    2555 log.go:25] "Validated CRI v1 runtime API"
May 25 12:13:07.430580 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.430565    2555 log.go:25] "Validated CRI v1 image API"
May 25 12:13:07.431796 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.431782    2555 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
May 25 12:13:07.434107 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.434089    2555 fs.go:135] Filesystem UUIDs: map[0513c0c7-47df-46b8-881e-6b2656a63d1c:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 a9640797-e200-448d-8488-950160ed5e0d:/dev/nvme0n1p4]
May 25 12:13:07.434167 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.434107    2555 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
May 25 12:13:07.436754 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.436736    2555 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 25 12:13:07.438524 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.438417    2555 manager.go:217] Machine: {Timestamp:2026-05-25 12:13:07.437938552 +0000 UTC m=+0.397988948 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3220383 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec282436deff5baae63074a75f3f8b3c SystemUUID:ec282436-deff-5baa-e630-74a75f3f8b3c BootID:8e03c3d8-2bb1-4eec-b562-a8b17b39ada9 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:24:90:0d:4c:4b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:24:90:0d:4c:4b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6a:93:ac:b2:81:9e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
May 25 12:13:07.438524 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.438517    2555 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
May 25 12:13:07.438632 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.438588    2555 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
May 25 12:13:07.441031 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.441006    2555 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
May 25 12:13:07.441164 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.441033    2555 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-255.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
May 25 12:13:07.441739 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.441729    2555 topology_manager.go:138] "Creating topology manager with none policy"
May 25 12:13:07.441773 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.441742    2555 container_manager_linux.go:306] "Creating device plugin manager"
May 25 12:13:07.441773 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.441754    2555 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
May 25 12:13:07.442590 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.442579    2555 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
May 25 12:13:07.444465 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.444455    2555 state_mem.go:36] "Initialized new in-memory state store"
May 25 12:13:07.444562 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.444553    2555 server.go:1267] "Using root directory" path="/var/lib/kubelet"
May 25 12:13:07.446867 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.446858    2555 kubelet.go:491] "Attempting to sync node with API server"
May 25 12:13:07.446898 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.446875    2555 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
May 25 12:13:07.446898 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.446887    2555 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
May 25 12:13:07.446898 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.446895    2555 kubelet.go:397] "Adding apiserver pod source"
May 25 12:13:07.446985 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.446903    2555 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
May 25 12:13:07.448026 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.448015    2555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 25 12:13:07.448073 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.448032    2555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 25 12:13:07.451088 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.451072    2555 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1"
May 25 12:13:07.452898 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.452885    2555 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
May 25 12:13:07.453832 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.453821    2555 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
May 25 12:13:07.453883 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.453837    2555 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
May 25 12:13:07.453883 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.453843    2555 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
May 25 12:13:07.453883 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.453848    2555 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
May 25 12:13:07.453883 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.453857    2555 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
May 25 12:13:07.453883 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.453862    2555 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
May 25 12:13:07.453883 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.453868    2555 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
May 25 12:13:07.453883 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.453873    2555 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
May 25 12:13:07.453883 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.453879    2555 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
May 25 12:13:07.453883 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.453885    2555 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
May 25 12:13:07.454189 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.453893    2555 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
May 25 12:13:07.454189 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.454178    2555 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
May 25 12:13:07.455068 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.455059    2555 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
May 25 12:13:07.455107 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.455069    2555 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
May 25 12:13:07.458197 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.458183    2555 watchdog_linux.go:99] "Systemd watchdog is not enabled"
May 25 12:13:07.458270 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.458223    2555 server.go:1295] "Started kubelet"
May 25 12:13:07.461240 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.461153    2555 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
May 25 12:13:07.461240 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.461235    2555 server_v1.go:47] "podresources" method="list" useActivePods=true
May 25 12:13:07.461436 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.461419    2555 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-255.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
May 25 12:13:07.461487 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.461414    2555 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
May 25 12:13:07.461634 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:07.461612    2555 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-255.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
May 25 12:13:07.461742 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:07.461724    2555 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
May 25 12:13:07.462172 ip-10-0-134-255 systemd[1]: Started Kubernetes Kubelet.
May 25 12:13:07.462789 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.462584    2555 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wcfmf"
May 25 12:13:07.463652 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.463162    2555 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
May 25 12:13:07.465437 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.465410    2555 server.go:317] "Adding debug handlers to kubelet server"
May 25 12:13:07.467395 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.467379    2555 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wcfmf"
May 25 12:13:07.467544 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:07.466297    2555 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-255.ec2.internal.18b2cde54b3c0495  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-255.ec2.internal,UID:ip-10-0-134-255.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-255.ec2.internal,},FirstTimestamp:2026-05-25 12:13:07.458196629 +0000 UTC m=+0.418247028,LastTimestamp:2026-05-25 12:13:07.458196629 +0000 UTC m=+0.418247028,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-255.ec2.internal,}"
May 25 12:13:07.470195 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.470177    2555 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
May 25 12:13:07.470195 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.470187    2555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
May 25 12:13:07.470337 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.470299    2555 volume_manager.go:295] "The desired_state_of_world populator starts"
May 25 12:13:07.470337 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.470311    2555 volume_manager.go:297] "Starting Kubelet Volume Manager"
May 25 12:13:07.470429 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.470357    2555 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
May 25 12:13:07.470484 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.470438    2555 reconstruct.go:97] "Volume reconstruction finished"
May 25 12:13:07.470484 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.470448    2555 reconciler.go:26] "Reconciler: start to sync state"
May 25 12:13:07.470863 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.470847    2555 factory.go:55] Registering systemd factory
May 25 12:13:07.470924 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.470866    2555 factory.go:223] Registration of the systemd container factory successfully
May 25 12:13:07.471043 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:07.471030    2555 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-255.ec2.internal\" not found"
May 25 12:13:07.471105 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.471096    2555 factory.go:153] Registering CRI-O factory
May 25 12:13:07.471137 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.471109    2555 factory.go:223] Registration of the crio container factory successfully
May 25 12:13:07.471176 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.471158    2555 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
May 25 12:13:07.471176 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.471170    2555 factory.go:103] Registering Raw factory
May 25 12:13:07.471232 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.471180    2555 manager.go:1196] Started watching for new ooms in manager
May 25 12:13:07.471757 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.471739    2555 manager.go:319] Starting recovery of all containers
May 25 12:13:07.472053 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:07.472031    2555 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
May 25 12:13:07.476684 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.476663    2555 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 12:13:07.480887 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:07.480859    2555 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-134-255.ec2.internal\" not found" node="ip-10-0-134-255.ec2.internal"
May 25 12:13:07.482296 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.482270    2555 manager.go:324] Recovery completed
May 25 12:13:07.487524 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.487427    2555 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 12:13:07.489277 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.489262    2555 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-255.ec2.internal" event="NodeHasSufficientMemory"
May 25 12:13:07.489358 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.489294    2555 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-255.ec2.internal" event="NodeHasNoDiskPressure"
May 25 12:13:07.489358 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.489308    2555 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-255.ec2.internal" event="NodeHasSufficientPID"
May 25 12:13:07.489766 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.489754    2555 cpu_manager.go:222] "Starting CPU manager" policy="none"
May 25 12:13:07.489808 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.489767    2555 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
May 25 12:13:07.489808 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.489786    2555 state_mem.go:36] "Initialized new in-memory state store"
May 25 12:13:07.491962 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.491951    2555 policy_none.go:49] "None policy: Start"
May 25 12:13:07.491995 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.491966    2555 memory_manager.go:186] "Starting memorymanager" policy="None"
May 25 12:13:07.491995 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.491976    2555 state_mem.go:35] "Initializing new in-memory state store"
May 25 12:13:07.534046 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.531749    2555 manager.go:341] "Starting Device Plugin manager"
May 25 12:13:07.534046 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:07.531788    2555 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
May 25 12:13:07.534046 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.531798    2555 server.go:85] "Starting device plugin registration server"
May 25 12:13:07.534046 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.532151    2555 eviction_manager.go:189] "Eviction manager: starting control loop"
May 25 12:13:07.534046 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.532161    2555 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
May 25 12:13:07.534046 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.532261    2555 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
May 25 12:13:07.534046 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.532391    2555 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
May 25 12:13:07.534046 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.532406    2555 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
May 25 12:13:07.534046 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:07.532746    2555 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
May 25 12:13:07.534046 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:07.532782    2555 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-255.ec2.internal\" not found"
May 25 12:13:07.592247 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.592217    2555 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
May 25 12:13:07.593340 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.593329    2555 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
May 25 12:13:07.593413 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.593346    2555 status_manager.go:230] "Starting to sync pod status with apiserver"
May 25 12:13:07.593413 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.593361    2555 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
May 25 12:13:07.593413 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.593367    2555 kubelet.go:2451] "Starting kubelet main sync loop"
May 25 12:13:07.593413 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:07.593399    2555 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
May 25 12:13:07.595829 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.595815    2555 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 12:13:07.632934 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.632892    2555 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 12:13:07.633669 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.633655    2555 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-255.ec2.internal" event="NodeHasSufficientMemory"
May 25 12:13:07.633759 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.633680    2555 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-255.ec2.internal" event="NodeHasNoDiskPressure"
May 25 12:13:07.633759 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.633690    2555 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-255.ec2.internal" event="NodeHasSufficientPID"
May 25 12:13:07.633759 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.633711    2555 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-255.ec2.internal"
May 25 12:13:07.641599 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.641586    2555 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-255.ec2.internal"
May 25 12:13:07.641654 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:07.641605    2555 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-255.ec2.internal\": node \"ip-10-0-134-255.ec2.internal\" not found"
May 25 12:13:07.671798 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:07.671778    2555 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-255.ec2.internal\" not found"
May 25 12:13:07.693953 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.693927    2555 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-255.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-255.ec2.internal"]
May 25 12:13:07.694021 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.694001    2555 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 12:13:07.695078 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.695064    2555 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-255.ec2.internal" event="NodeHasSufficientMemory"
May 25 12:13:07.695125 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.695091    2555 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-255.ec2.internal" event="NodeHasNoDiskPressure"
May 25 12:13:07.695125 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.695103    2555 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-255.ec2.internal" event="NodeHasSufficientPID"
May 25 12:13:07.696281 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.696269    2555 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 12:13:07.696432 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.696419    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-255.ec2.internal"
May 25 12:13:07.696477 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.696446    2555 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 12:13:07.696905 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.696890    2555 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-255.ec2.internal" event="NodeHasSufficientMemory"
May 25 12:13:07.696905 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.696896    2555 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-255.ec2.internal" event="NodeHasSufficientMemory"
May 25 12:13:07.697011 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.696911    2555 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-255.ec2.internal" event="NodeHasNoDiskPressure"
May 25 12:13:07.697011 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.696917    2555 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-255.ec2.internal" event="NodeHasNoDiskPressure"
May 25 12:13:07.697011 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.696921    2555 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-255.ec2.internal" event="NodeHasSufficientPID"
May 25 12:13:07.697011 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.696926    2555 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-255.ec2.internal" event="NodeHasSufficientPID"
May 25 12:13:07.698040 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.698015    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-255.ec2.internal"
May 25 12:13:07.698040 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.698038    2555 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 12:13:07.698593 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.698580    2555 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-255.ec2.internal" event="NodeHasSufficientMemory"
May 25 12:13:07.698664 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.698603    2555 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-255.ec2.internal" event="NodeHasNoDiskPressure"
May 25 12:13:07.698664 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.698616    2555 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-255.ec2.internal" event="NodeHasSufficientPID"
May 25 12:13:07.714139 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:07.714121    2555 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-255.ec2.internal\" not found" node="ip-10-0-134-255.ec2.internal"
May 25 12:13:07.718328 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:07.718313    2555 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-255.ec2.internal\" not found" node="ip-10-0-134-255.ec2.internal"
May 25 12:13:07.771845 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:07.771828    2555 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-255.ec2.internal\" not found"
May 25 12:13:07.871339 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.871318    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e477f180b7e0e366bb0a0001192721d9-config\") pod \"kube-apiserver-proxy-ip-10-0-134-255.ec2.internal\" (UID: \"e477f180b7e0e366bb0a0001192721d9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-255.ec2.internal"
May 25 12:13:07.871410 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.871351    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fa0099618f996f4bb5ffb3952e434c51-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-255.ec2.internal\" (UID: \"fa0099618f996f4bb5ffb3952e434c51\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-255.ec2.internal"
May 25 12:13:07.871410 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.871371    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa0099618f996f4bb5ffb3952e434c51-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-255.ec2.internal\" (UID: \"fa0099618f996f4bb5ffb3952e434c51\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-255.ec2.internal"
May 25 12:13:07.872364 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:07.872350    2555 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-255.ec2.internal\" not found"
May 25 12:13:07.971959 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.971894    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e477f180b7e0e366bb0a0001192721d9-config\") pod \"kube-apiserver-proxy-ip-10-0-134-255.ec2.internal\" (UID: \"e477f180b7e0e366bb0a0001192721d9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-255.ec2.internal"
May 25 12:13:07.971959 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.971932    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fa0099618f996f4bb5ffb3952e434c51-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-255.ec2.internal\" (UID: \"fa0099618f996f4bb5ffb3952e434c51\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-255.ec2.internal"
May 25 12:13:07.971959 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.971953    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e477f180b7e0e366bb0a0001192721d9-config\") pod \"kube-apiserver-proxy-ip-10-0-134-255.ec2.internal\" (UID: \"e477f180b7e0e366bb0a0001192721d9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-255.ec2.internal"
May 25 12:13:07.971959 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.971963    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa0099618f996f4bb5ffb3952e434c51-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-255.ec2.internal\" (UID: \"fa0099618f996f4bb5ffb3952e434c51\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-255.ec2.internal"
May 25 12:13:07.972124 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.971991    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa0099618f996f4bb5ffb3952e434c51-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-255.ec2.internal\" (UID: \"fa0099618f996f4bb5ffb3952e434c51\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-255.ec2.internal"
May 25 12:13:07.972124 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:07.972012    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fa0099618f996f4bb5ffb3952e434c51-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-255.ec2.internal\" (UID: \"fa0099618f996f4bb5ffb3952e434c51\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-255.ec2.internal"
May 25 12:13:07.972943 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:07.972927    2555 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-255.ec2.internal\" not found"
May 25 12:13:08.017096 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:08.017073    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-255.ec2.internal"
May 25 12:13:08.020799 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:08.020785    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-255.ec2.internal"
May 25 12:13:08.073564 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:08.073534    2555 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-255.ec2.internal\" not found"
May 25 12:13:08.173966 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:08.173941    2555 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-255.ec2.internal\" not found"
May 25 12:13:08.274400 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:08.274345    2555 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-255.ec2.internal\" not found"
May 25 12:13:08.374769 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:08.374745    2555 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-255.ec2.internal\" not found"
May 25 12:13:08.381908 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:08.381889    2555 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
May 25 12:13:08.382017 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:08.382003    2555 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 25 12:13:08.382087 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:08.382037    2555 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 25 12:13:08.469842 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:08.469811    2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-24 12:08:07 +0000 UTC" deadline="2027-12-10 11:45:56.351852403 +0000 UTC"
May 25 12:13:08.469979 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:08.469839    2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13535h32m47.882016708s"
May 25 12:13:08.470892 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:08.470867    2555 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
May 25 12:13:08.475693 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:08.475673    2555 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-255.ec2.internal\" not found"
May 25 12:13:08.480433 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:08.480416    2555 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 25 12:13:08.484968 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:08.484951    2555 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 12:13:08.503856 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:08.503839    2555 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-sk4s7"
May 25 12:13:08.511980 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:08.511963    2555 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-sk4s7"
May 25 12:13:08.535273 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:08.535252    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa0099618f996f4bb5ffb3952e434c51.slice/crio-f2838a8df1bbe22446a483ed07493d9d8ab178908d340e1214c34323ce0e4836 WatchSource:0}: Error finding container f2838a8df1bbe22446a483ed07493d9d8ab178908d340e1214c34323ce0e4836: Status 404 returned error can't find the container with id f2838a8df1bbe22446a483ed07493d9d8ab178908d340e1214c34323ce0e4836
May 25 12:13:08.535752 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:08.535731    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode477f180b7e0e366bb0a0001192721d9.slice/crio-007d5b2bbb4300a098b1f8d1600822e410ad0ca1d9424b35ee2922ef1654d559 WatchSource:0}: Error finding container 007d5b2bbb4300a098b1f8d1600822e410ad0ca1d9424b35ee2922ef1654d559: Status 404 returned error can't find the container with id 007d5b2bbb4300a098b1f8d1600822e410ad0ca1d9424b35ee2922ef1654d559
May 25 12:13:08.540156 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:08.540142    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 12:13:08.570896 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:08.570877    2555 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-255.ec2.internal"
May 25 12:13:08.579480 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:08.579461    2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 25 12:13:08.581352 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:08.581340    2555 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-255.ec2.internal"
May 25 12:13:08.590462 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:08.590442    2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 25 12:13:08.597807 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:08.597767    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-255.ec2.internal" event={"ID":"e477f180b7e0e366bb0a0001192721d9","Type":"ContainerStarted","Data":"007d5b2bbb4300a098b1f8d1600822e410ad0ca1d9424b35ee2922ef1654d559"}
May 25 12:13:08.598594 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:08.598573    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-255.ec2.internal" event={"ID":"fa0099618f996f4bb5ffb3952e434c51","Type":"ContainerStarted","Data":"f2838a8df1bbe22446a483ed07493d9d8ab178908d340e1214c34323ce0e4836"}
May 25 12:13:08.813543 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:08.813489    2555 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 12:13:09.417993 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.417965    2555 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 12:13:09.447462 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.447437    2555 apiserver.go:52] "Watching apiserver"
May 25 12:13:09.454635 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.454609    2555 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
May 25 12:13:09.456852 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.456816    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-jpm95","kube-system/kube-apiserver-proxy-ip-10-0-134-255.ec2.internal","openshift-multus/multus-7hwgl","openshift-multus/multus-additional-cni-plugins-6tbkc","openshift-ovn-kubernetes/ovnkube-node-jxpcm","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nqql","openshift-cluster-node-tuning-operator/tuned-vzv5c","openshift-dns/node-resolver-dtbvj","openshift-image-registry/node-ca-brxmt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-255.ec2.internal","openshift-multus/network-metrics-daemon-nwlcx","openshift-network-diagnostics/network-check-target-4wlq9","openshift-network-operator/iptables-alerter-49hs2"]
May 25 12:13:09.459522 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.459492    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vzv5c"
May 25 12:13:09.459722 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.459700    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7hwgl"
May 25 12:13:09.460995 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.460964    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6tbkc"
May 25 12:13:09.461572 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.461435    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jjgjh\""
May 25 12:13:09.461688 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.461614    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
May 25 12:13:09.461688 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.461663    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-8bc9b\""
May 25 12:13:09.462033 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.462015    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
May 25 12:13:09.462111 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.462036    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
May 25 12:13:09.462190 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.462172    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
May 25 12:13:09.462246 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.462223    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
May 25 12:13:09.462246 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.462179    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
May 25 12:13:09.462845 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.462827    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
May 25 12:13:09.462980 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.462956    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
May 25 12:13:09.463036 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.463014    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-k9g6m\""
May 25 12:13:09.463289 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.463269    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.463611 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.463597    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nqql"
May 25 12:13:09.468004 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.465521    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jpm95"
May 25 12:13:09.468004 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.465690    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
May 25 12:13:09.468004 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.465921    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
May 25 12:13:09.468004 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.466536    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
May 25 12:13:09.468004 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.466583    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
May 25 12:13:09.468004 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.466538    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nkhm5\""
May 25 12:13:09.468004 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.466770    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4rw66\""
May 25 12:13:09.468004 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.466955    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
May 25 12:13:09.468004 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.466975    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
May 25 12:13:09.468004 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.467879    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
May 25 12:13:09.468004 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.467920    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
May 25 12:13:09.468526 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.468239    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dtbvj"
May 25 12:13:09.468526 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.468296    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
May 25 12:13:09.468526 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.468406    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
May 25 12:13:09.468683 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.468243    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
May 25 12:13:09.469221 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.469203    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5h6bn\""
May 25 12:13:09.470050 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.470029    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
May 25 12:13:09.470678 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.470413    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
May 25 12:13:09.470678 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.470590    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4n65r\""
May 25 12:13:09.471439 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.471403    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-brxmt"
May 25 12:13:09.472861 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.472843    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwlcx"
May 25 12:13:09.472957 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:09.472918    2555 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwlcx" podUID="30f9c156-1292-4927-9159-eba19cf76012"
May 25 12:13:09.473453 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.473263    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
May 25 12:13:09.473453 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.473320    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2wxcc\""
May 25 12:13:09.473595 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.473454    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
May 25 12:13:09.473595 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.473269    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
May 25 12:13:09.475820 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.475802    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4wlq9"
May 25 12:13:09.475913 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.475823    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-49hs2"
May 25 12:13:09.475913 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:09.475871    2555 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4wlq9" podUID="2d835613-0b9d-4b52-bc2f-a40577b71208"
May 25 12:13:09.477522 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.477502    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
May 25 12:13:09.477616 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.477535    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
May 25 12:13:09.477616 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.477573    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-f97m4\""
May 25 12:13:09.477927 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.477675    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
May 25 12:13:09.479235 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.479212    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/798578bf-c83f-402e-97ad-b3944200e4f6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6tbkc\" (UID: \"798578bf-c83f-402e-97ad-b3944200e4f6\") " pod="openshift-multus/multus-additional-cni-plugins-6tbkc"
May 25 12:13:09.479325 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.479251    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/80908050-ac10-44a0-9f07-4599e6fb0960-multus-conf-dir\") pod \"multus-7hwgl\" (UID: \"80908050-ac10-44a0-9f07-4599e6fb0960\") " pod="openshift-multus/multus-7hwgl"
May 25 12:13:09.479325 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.479277    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd562\" (UniqueName: \"kubernetes.io/projected/80908050-ac10-44a0-9f07-4599e6fb0960-kube-api-access-gd562\") pod \"multus-7hwgl\" (UID: \"80908050-ac10-44a0-9f07-4599e6fb0960\") " pod="openshift-multus/multus-7hwgl"
May 25 12:13:09.479325 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.479300    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/798578bf-c83f-402e-97ad-b3944200e4f6-os-release\") pod \"multus-additional-cni-plugins-6tbkc\" (UID: \"798578bf-c83f-402e-97ad-b3944200e4f6\") " pod="openshift-multus/multus-additional-cni-plugins-6tbkc"
May 25 12:13:09.479325 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.479322    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/80908050-ac10-44a0-9f07-4599e6fb0960-host-run-netns\") pod \"multus-7hwgl\" (UID: \"80908050-ac10-44a0-9f07-4599e6fb0960\") " pod="openshift-multus/multus-7hwgl"
May 25 12:13:09.479478 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.479344    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/80908050-ac10-44a0-9f07-4599e6fb0960-host-var-lib-cni-multus\") pod \"multus-7hwgl\" (UID: \"80908050-ac10-44a0-9f07-4599e6fb0960\") " pod="openshift-multus/multus-7hwgl"
May 25 12:13:09.479478 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.479368    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3ba2b664-042c-4db1-92e7-bf94eb2a7802-socket-dir\") pod \"aws-ebs-csi-driver-node-4nqql\" (UID: \"3ba2b664-042c-4db1-92e7-bf94eb2a7802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nqql"
May 25 12:13:09.479478 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.479389    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d776d0f6-f465-4279-81fc-33535ec8ea83-etc-modprobe-d\") pod \"tuned-vzv5c\" (UID: \"d776d0f6-f465-4279-81fc-33535ec8ea83\") " pod="openshift-cluster-node-tuning-operator/tuned-vzv5c"
May 25 12:13:09.479478 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.479419    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d776d0f6-f465-4279-81fc-33535ec8ea83-etc-sysconfig\") pod \"tuned-vzv5c\" (UID: \"d776d0f6-f465-4279-81fc-33535ec8ea83\") " pod="openshift-cluster-node-tuning-operator/tuned-vzv5c"
May 25 12:13:09.479478 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.479465    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3ba2b664-042c-4db1-92e7-bf94eb2a7802-sys-fs\") pod \"aws-ebs-csi-driver-node-4nqql\" (UID: \"3ba2b664-042c-4db1-92e7-bf94eb2a7802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nqql"
May 25 12:13:09.479683 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.479490    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcm7w\" (UniqueName: \"kubernetes.io/projected/3ba2b664-042c-4db1-92e7-bf94eb2a7802-kube-api-access-hcm7w\") pod \"aws-ebs-csi-driver-node-4nqql\" (UID: \"3ba2b664-042c-4db1-92e7-bf94eb2a7802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nqql"
May 25 12:13:09.479683 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.479514    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ab76ee4f-2142-41d5-99e2-0de355f6564d-hosts-file\") pod \"node-resolver-dtbvj\" (UID: \"ab76ee4f-2142-41d5-99e2-0de355f6564d\") " pod="openshift-dns/node-resolver-dtbvj"
May 25 12:13:09.479683 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.479537    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d776d0f6-f465-4279-81fc-33535ec8ea83-tmp\") pod \"tuned-vzv5c\" (UID: \"d776d0f6-f465-4279-81fc-33535ec8ea83\") " pod="openshift-cluster-node-tuning-operator/tuned-vzv5c"
May 25 12:13:09.479683 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.479559    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/94f59572-7f6a-47f6-8091-56651d472fcf-host-slash\") pod \"ovnkube-node-jxpcm\" (UID: \"94f59572-7f6a-47f6-8091-56651d472fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.479683 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.479596    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/94f59572-7f6a-47f6-8091-56651d472fcf-run-openvswitch\") pod \"ovnkube-node-jxpcm\" (UID: \"94f59572-7f6a-47f6-8091-56651d472fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.479683 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.479620    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3ba2b664-042c-4db1-92e7-bf94eb2a7802-device-dir\") pod \"aws-ebs-csi-driver-node-4nqql\" (UID: \"3ba2b664-042c-4db1-92e7-bf94eb2a7802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nqql"
May 25 12:13:09.479683 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.479655    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d776d0f6-f465-4279-81fc-33535ec8ea83-sys\") pod \"tuned-vzv5c\" (UID: \"d776d0f6-f465-4279-81fc-33535ec8ea83\") " pod="openshift-cluster-node-tuning-operator/tuned-vzv5c"
May 25 12:13:09.479960 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.479685    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clwtf\" (UniqueName: \"kubernetes.io/projected/94f59572-7f6a-47f6-8091-56651d472fcf-kube-api-access-clwtf\") pod \"ovnkube-node-jxpcm\" (UID: \"94f59572-7f6a-47f6-8091-56651d472fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.479960 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.479712    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm8gt\" (UniqueName: \"kubernetes.io/projected/798578bf-c83f-402e-97ad-b3944200e4f6-kube-api-access-mm8gt\") pod \"multus-additional-cni-plugins-6tbkc\" (UID: \"798578bf-c83f-402e-97ad-b3944200e4f6\") " pod="openshift-multus/multus-additional-cni-plugins-6tbkc"
May 25 12:13:09.479960 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.479734    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/80908050-ac10-44a0-9f07-4599e6fb0960-system-cni-dir\") pod \"multus-7hwgl\" (UID: \"80908050-ac10-44a0-9f07-4599e6fb0960\") " pod="openshift-multus/multus-7hwgl"
May 25 12:13:09.479960 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.479757    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/80908050-ac10-44a0-9f07-4599e6fb0960-host-var-lib-kubelet\") pod \"multus-7hwgl\" (UID: \"80908050-ac10-44a0-9f07-4599e6fb0960\") " pod="openshift-multus/multus-7hwgl"
May 25 12:13:09.479960 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.479779    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/798578bf-c83f-402e-97ad-b3944200e4f6-cni-binary-copy\") pod \"multus-additional-cni-plugins-6tbkc\" (UID: \"798578bf-c83f-402e-97ad-b3944200e4f6\") " pod="openshift-multus/multus-additional-cni-plugins-6tbkc"
May 25 12:13:09.479960 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.479801    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3ba2b664-042c-4db1-92e7-bf94eb2a7802-etc-selinux\") pod \"aws-ebs-csi-driver-node-4nqql\" (UID: \"3ba2b664-042c-4db1-92e7-bf94eb2a7802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nqql"
May 25 12:13:09.479960 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.479841    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b77364d1-80eb-4af1-9764-ca0f0858270a-host\") pod \"node-ca-brxmt\" (UID: \"b77364d1-80eb-4af1-9764-ca0f0858270a\") " pod="openshift-image-registry/node-ca-brxmt"
May 25 12:13:09.479960 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.479871    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d776d0f6-f465-4279-81fc-33535ec8ea83-etc-tuned\") pod \"tuned-vzv5c\" (UID: \"d776d0f6-f465-4279-81fc-33535ec8ea83\") " pod="openshift-cluster-node-tuning-operator/tuned-vzv5c"
May 25 12:13:09.479960 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.479924    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/94f59572-7f6a-47f6-8091-56651d472fcf-host-cni-netd\") pod \"ovnkube-node-jxpcm\" (UID: \"94f59572-7f6a-47f6-8091-56651d472fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.480272 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.479980    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/80908050-ac10-44a0-9f07-4599e6fb0960-cni-binary-copy\") pod \"multus-7hwgl\" (UID: \"80908050-ac10-44a0-9f07-4599e6fb0960\") " pod="openshift-multus/multus-7hwgl"
May 25 12:13:09.480272 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.480008    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/80908050-ac10-44a0-9f07-4599e6fb0960-multus-socket-dir-parent\") pod \"multus-7hwgl\" (UID: \"80908050-ac10-44a0-9f07-4599e6fb0960\") " pod="openshift-multus/multus-7hwgl"
May 25 12:13:09.480272 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.480034    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/72a67a33-cbd6-49ec-a72d-65cdf815803a-konnectivity-ca\") pod \"konnectivity-agent-jpm95\" (UID: \"72a67a33-cbd6-49ec-a72d-65cdf815803a\") " pod="kube-system/konnectivity-agent-jpm95"
May 25 12:13:09.480272 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.480058    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94f59572-7f6a-47f6-8091-56651d472fcf-host-run-ovn-kubernetes\") pod \"ovnkube-node-jxpcm\" (UID: \"94f59572-7f6a-47f6-8091-56651d472fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.480272 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.480081    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/80908050-ac10-44a0-9f07-4599e6fb0960-host-var-lib-cni-bin\") pod \"multus-7hwgl\" (UID: \"80908050-ac10-44a0-9f07-4599e6fb0960\") " pod="openshift-multus/multus-7hwgl"
May 25 12:13:09.480272 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.480103    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/80908050-ac10-44a0-9f07-4599e6fb0960-host-run-multus-certs\") pod \"multus-7hwgl\" (UID: \"80908050-ac10-44a0-9f07-4599e6fb0960\") " pod="openshift-multus/multus-7hwgl"
May 25 12:13:09.480272 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.480125    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d776d0f6-f465-4279-81fc-33535ec8ea83-etc-sysctl-conf\") pod \"tuned-vzv5c\" (UID: \"d776d0f6-f465-4279-81fc-33535ec8ea83\") " pod="openshift-cluster-node-tuning-operator/tuned-vzv5c"
May 25 12:13:09.480272 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.480147    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/94f59572-7f6a-47f6-8091-56651d472fcf-etc-openvswitch\") pod \"ovnkube-node-jxpcm\" (UID: \"94f59572-7f6a-47f6-8091-56651d472fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.480272 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.480170    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/80908050-ac10-44a0-9f07-4599e6fb0960-multus-cni-dir\") pod \"multus-7hwgl\" (UID: \"80908050-ac10-44a0-9f07-4599e6fb0960\") " pod="openshift-multus/multus-7hwgl"
May 25 12:13:09.480272 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.480192    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/798578bf-c83f-402e-97ad-b3944200e4f6-cnibin\") pod \"multus-additional-cni-plugins-6tbkc\" (UID: \"798578bf-c83f-402e-97ad-b3944200e4f6\") " pod="openshift-multus/multus-additional-cni-plugins-6tbkc"
May 25 12:13:09.480272 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.480215    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/798578bf-c83f-402e-97ad-b3944200e4f6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6tbkc\" (UID: \"798578bf-c83f-402e-97ad-b3944200e4f6\") " pod="openshift-multus/multus-additional-cni-plugins-6tbkc"
May 25 12:13:09.480272 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.480236    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4bgf\" (UniqueName: \"kubernetes.io/projected/b77364d1-80eb-4af1-9764-ca0f0858270a-kube-api-access-p4bgf\") pod \"node-ca-brxmt\" (UID: \"b77364d1-80eb-4af1-9764-ca0f0858270a\") " pod="openshift-image-registry/node-ca-brxmt"
May 25 12:13:09.480800 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.480282    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/80908050-ac10-44a0-9f07-4599e6fb0960-host-run-k8s-cni-cncf-io\") pod \"multus-7hwgl\" (UID: \"80908050-ac10-44a0-9f07-4599e6fb0960\") " pod="openshift-multus/multus-7hwgl"
May 25 12:13:09.480800 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.480327    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/80908050-ac10-44a0-9f07-4599e6fb0960-hostroot\") pod \"multus-7hwgl\" (UID: \"80908050-ac10-44a0-9f07-4599e6fb0960\") " pod="openshift-multus/multus-7hwgl"
May 25 12:13:09.480800 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.480365    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab76ee4f-2142-41d5-99e2-0de355f6564d-tmp-dir\") pod \"node-resolver-dtbvj\" (UID: \"ab76ee4f-2142-41d5-99e2-0de355f6564d\") " pod="openshift-dns/node-resolver-dtbvj"
May 25 12:13:09.480800 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.480389    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/94f59572-7f6a-47f6-8091-56651d472fcf-log-socket\") pod \"ovnkube-node-jxpcm\" (UID: \"94f59572-7f6a-47f6-8091-56651d472fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.480800 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.480441    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/94f59572-7f6a-47f6-8091-56651d472fcf-ovnkube-script-lib\") pod \"ovnkube-node-jxpcm\" (UID: \"94f59572-7f6a-47f6-8091-56651d472fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.480800 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.480467    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/798578bf-c83f-402e-97ad-b3944200e4f6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6tbkc\" (UID: \"798578bf-c83f-402e-97ad-b3944200e4f6\") " pod="openshift-multus/multus-additional-cni-plugins-6tbkc"
May 25 12:13:09.480800 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.480488    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/94f59572-7f6a-47f6-8091-56651d472fcf-host-run-netns\") pod \"ovnkube-node-jxpcm\" (UID: \"94f59572-7f6a-47f6-8091-56651d472fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.480800 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.480502    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94f59572-7f6a-47f6-8091-56651d472fcf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jxpcm\" (UID: \"94f59572-7f6a-47f6-8091-56651d472fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.480800 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.480542    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/80908050-ac10-44a0-9f07-4599e6fb0960-etc-kubernetes\") pod \"multus-7hwgl\" (UID: \"80908050-ac10-44a0-9f07-4599e6fb0960\") " pod="openshift-multus/multus-7hwgl"
May 25 12:13:09.480800 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.480620    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d776d0f6-f465-4279-81fc-33535ec8ea83-run\") pod \"tuned-vzv5c\" (UID: \"d776d0f6-f465-4279-81fc-33535ec8ea83\") " pod="openshift-cluster-node-tuning-operator/tuned-vzv5c"
May 25 12:13:09.480800 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.480652    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d776d0f6-f465-4279-81fc-33535ec8ea83-lib-modules\") pod \"tuned-vzv5c\" (UID: \"d776d0f6-f465-4279-81fc-33535ec8ea83\") " pod="openshift-cluster-node-tuning-operator/tuned-vzv5c"
May 25 12:13:09.480800 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.480677    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d776d0f6-f465-4279-81fc-33535ec8ea83-var-lib-kubelet\") pod \"tuned-vzv5c\" (UID: \"d776d0f6-f465-4279-81fc-33535ec8ea83\") " pod="openshift-cluster-node-tuning-operator/tuned-vzv5c"
May 25 12:13:09.480800 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.480699    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/94f59572-7f6a-47f6-8091-56651d472fcf-host-kubelet\") pod \"ovnkube-node-jxpcm\" (UID: \"94f59572-7f6a-47f6-8091-56651d472fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.480800 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.480722    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/94f59572-7f6a-47f6-8091-56651d472fcf-var-lib-openvswitch\") pod \"ovnkube-node-jxpcm\" (UID: \"94f59572-7f6a-47f6-8091-56651d472fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.480800 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.480744    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/94f59572-7f6a-47f6-8091-56651d472fcf-env-overrides\") pod \"ovnkube-node-jxpcm\" (UID: \"94f59572-7f6a-47f6-8091-56651d472fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.480800 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.480766    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/80908050-ac10-44a0-9f07-4599e6fb0960-cnibin\") pod \"multus-7hwgl\" (UID: \"80908050-ac10-44a0-9f07-4599e6fb0960\") " pod="openshift-multus/multus-7hwgl"
May 25 12:13:09.480800 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.480787    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/80908050-ac10-44a0-9f07-4599e6fb0960-os-release\") pod \"multus-7hwgl\" (UID: \"80908050-ac10-44a0-9f07-4599e6fb0960\") " pod="openshift-multus/multus-7hwgl"
May 25 12:13:09.481450 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.480830    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3ba2b664-042c-4db1-92e7-bf94eb2a7802-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4nqql\" (UID: \"3ba2b664-042c-4db1-92e7-bf94eb2a7802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nqql"
May 25 12:13:09.481450 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.480881    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3ba2b664-042c-4db1-92e7-bf94eb2a7802-registration-dir\") pod \"aws-ebs-csi-driver-node-4nqql\" (UID: \"3ba2b664-042c-4db1-92e7-bf94eb2a7802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nqql"
May 25 12:13:09.481450 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.480909    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/72a67a33-cbd6-49ec-a72d-65cdf815803a-agent-certs\") pod \"konnectivity-agent-jpm95\" (UID: \"72a67a33-cbd6-49ec-a72d-65cdf815803a\") " pod="kube-system/konnectivity-agent-jpm95"
May 25 12:13:09.481450 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.480931    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/94f59572-7f6a-47f6-8091-56651d472fcf-run-systemd\") pod \"ovnkube-node-jxpcm\" (UID: \"94f59572-7f6a-47f6-8091-56651d472fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.481450 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.480953    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/94f59572-7f6a-47f6-8091-56651d472fcf-host-cni-bin\") pod \"ovnkube-node-jxpcm\" (UID: \"94f59572-7f6a-47f6-8091-56651d472fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.481450 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.480992    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/94f59572-7f6a-47f6-8091-56651d472fcf-ovnkube-config\") pod \"ovnkube-node-jxpcm\" (UID: \"94f59572-7f6a-47f6-8091-56651d472fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.481450 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.481023    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggqlh\" (UniqueName: \"kubernetes.io/projected/ab76ee4f-2142-41d5-99e2-0de355f6564d-kube-api-access-ggqlh\") pod \"node-resolver-dtbvj\" (UID: \"ab76ee4f-2142-41d5-99e2-0de355f6564d\") " pod="openshift-dns/node-resolver-dtbvj"
May 25 12:13:09.481450 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.481047    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/94f59572-7f6a-47f6-8091-56651d472fcf-systemd-units\") pod \"ovnkube-node-jxpcm\" (UID: \"94f59572-7f6a-47f6-8091-56651d472fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.481450 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.481069    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/94f59572-7f6a-47f6-8091-56651d472fcf-run-ovn\") pod \"ovnkube-node-jxpcm\" (UID: \"94f59572-7f6a-47f6-8091-56651d472fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.481450 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.481093    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/80908050-ac10-44a0-9f07-4599e6fb0960-multus-daemon-config\") pod \"multus-7hwgl\" (UID: \"80908050-ac10-44a0-9f07-4599e6fb0960\") " pod="openshift-multus/multus-7hwgl"
May 25 12:13:09.481450 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.481117    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b77364d1-80eb-4af1-9764-ca0f0858270a-serviceca\") pod \"node-ca-brxmt\" (UID: \"b77364d1-80eb-4af1-9764-ca0f0858270a\") " pod="openshift-image-registry/node-ca-brxmt"
May 25 12:13:09.481450 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.481166    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d776d0f6-f465-4279-81fc-33535ec8ea83-host\") pod \"tuned-vzv5c\" (UID: \"d776d0f6-f465-4279-81fc-33535ec8ea83\") " pod="openshift-cluster-node-tuning-operator/tuned-vzv5c"
May 25 12:13:09.481450 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.481193    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mg5l\" (UniqueName: \"kubernetes.io/projected/d776d0f6-f465-4279-81fc-33535ec8ea83-kube-api-access-5mg5l\") pod \"tuned-vzv5c\" (UID: \"d776d0f6-f465-4279-81fc-33535ec8ea83\") " pod="openshift-cluster-node-tuning-operator/tuned-vzv5c"
May 25 12:13:09.481450 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.481214    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/94f59572-7f6a-47f6-8091-56651d472fcf-ovn-node-metrics-cert\") pod \"ovnkube-node-jxpcm\" (UID: \"94f59572-7f6a-47f6-8091-56651d472fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.481450 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.481238    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/798578bf-c83f-402e-97ad-b3944200e4f6-system-cni-dir\") pod \"multus-additional-cni-plugins-6tbkc\" (UID: \"798578bf-c83f-402e-97ad-b3944200e4f6\") " pod="openshift-multus/multus-additional-cni-plugins-6tbkc"
May 25 12:13:09.481450 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.481259    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d776d0f6-f465-4279-81fc-33535ec8ea83-etc-kubernetes\") pod \"tuned-vzv5c\" (UID: \"d776d0f6-f465-4279-81fc-33535ec8ea83\") " pod="openshift-cluster-node-tuning-operator/tuned-vzv5c"
May 25 12:13:09.482055 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.481294    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d776d0f6-f465-4279-81fc-33535ec8ea83-etc-sysctl-d\") pod \"tuned-vzv5c\" (UID: \"d776d0f6-f465-4279-81fc-33535ec8ea83\") " pod="openshift-cluster-node-tuning-operator/tuned-vzv5c"
May 25 12:13:09.482055 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.481330    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/94f59572-7f6a-47f6-8091-56651d472fcf-node-log\") pod \"ovnkube-node-jxpcm\" (UID: \"94f59572-7f6a-47f6-8091-56651d472fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.482055 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.481356    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d776d0f6-f465-4279-81fc-33535ec8ea83-etc-systemd\") pod \"tuned-vzv5c\" (UID: \"d776d0f6-f465-4279-81fc-33535ec8ea83\") " pod="openshift-cluster-node-tuning-operator/tuned-vzv5c"
May 25 12:13:09.512860 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.512820    2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-24 12:08:08 +0000 UTC" deadline="2027-12-17 03:32:35.332082315 +0000 UTC"
May 25 12:13:09.512860 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.512859    2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13695h19m25.81922636s"
May 25 12:13:09.572147 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.572124    2555 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
May 25 12:13:09.582274 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.582250    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/80908050-ac10-44a0-9f07-4599e6fb0960-etc-kubernetes\") pod \"multus-7hwgl\" (UID: \"80908050-ac10-44a0-9f07-4599e6fb0960\") " pod="openshift-multus/multus-7hwgl"
May 25 12:13:09.582382 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.582289    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d776d0f6-f465-4279-81fc-33535ec8ea83-run\") pod \"tuned-vzv5c\" (UID: \"d776d0f6-f465-4279-81fc-33535ec8ea83\") " pod="openshift-cluster-node-tuning-operator/tuned-vzv5c"
May 25 12:13:09.582382 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.582316    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d776d0f6-f465-4279-81fc-33535ec8ea83-lib-modules\") pod \"tuned-vzv5c\" (UID: \"d776d0f6-f465-4279-81fc-33535ec8ea83\") " pod="openshift-cluster-node-tuning-operator/tuned-vzv5c"
May 25 12:13:09.582382 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.582339    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d776d0f6-f465-4279-81fc-33535ec8ea83-var-lib-kubelet\") pod \"tuned-vzv5c\" (UID: \"d776d0f6-f465-4279-81fc-33535ec8ea83\") " pod="openshift-cluster-node-tuning-operator/tuned-vzv5c"
May 25 12:13:09.582382 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.582363    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/94f59572-7f6a-47f6-8091-56651d472fcf-host-kubelet\") pod \"ovnkube-node-jxpcm\" (UID: \"94f59572-7f6a-47f6-8091-56651d472fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.582382 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.582362    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/80908050-ac10-44a0-9f07-4599e6fb0960-etc-kubernetes\") pod \"multus-7hwgl\" (UID: \"80908050-ac10-44a0-9f07-4599e6fb0960\") " pod="openshift-multus/multus-7hwgl"
May 25 12:13:09.582563 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.582386    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d776d0f6-f465-4279-81fc-33535ec8ea83-run\") pod \"tuned-vzv5c\" (UID: \"d776d0f6-f465-4279-81fc-33535ec8ea83\") " pod="openshift-cluster-node-tuning-operator/tuned-vzv5c"
May 25 12:13:09.582563 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.582386    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/94f59572-7f6a-47f6-8091-56651d472fcf-var-lib-openvswitch\") pod \"ovnkube-node-jxpcm\" (UID: \"94f59572-7f6a-47f6-8091-56651d472fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.582563 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.582417    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/94f59572-7f6a-47f6-8091-56651d472fcf-host-kubelet\") pod \"ovnkube-node-jxpcm\" (UID: \"94f59572-7f6a-47f6-8091-56651d472fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.582563 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.582415    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d776d0f6-f465-4279-81fc-33535ec8ea83-var-lib-kubelet\") pod \"tuned-vzv5c\" (UID: \"d776d0f6-f465-4279-81fc-33535ec8ea83\") " pod="openshift-cluster-node-tuning-operator/tuned-vzv5c"
May 25 12:13:09.582563 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.582430    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/94f59572-7f6a-47f6-8091-56651d472fcf-env-overrides\") pod \"ovnkube-node-jxpcm\" (UID: \"94f59572-7f6a-47f6-8091-56651d472fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.582563 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.582454    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/80908050-ac10-44a0-9f07-4599e6fb0960-cnibin\") pod \"multus-7hwgl\" (UID: \"80908050-ac10-44a0-9f07-4599e6fb0960\") " pod="openshift-multus/multus-7hwgl"
May 25 12:13:09.582563 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.582465    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d776d0f6-f465-4279-81fc-33535ec8ea83-lib-modules\") pod \"tuned-vzv5c\" (UID: \"d776d0f6-f465-4279-81fc-33535ec8ea83\") " pod="openshift-cluster-node-tuning-operator/tuned-vzv5c"
May 25 12:13:09.582563 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.582455    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/94f59572-7f6a-47f6-8091-56651d472fcf-var-lib-openvswitch\") pod \"ovnkube-node-jxpcm\" (UID: \"94f59572-7f6a-47f6-8091-56651d472fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.582563 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.582480    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/80908050-ac10-44a0-9f07-4599e6fb0960-os-release\") pod \"multus-7hwgl\" (UID: \"80908050-ac10-44a0-9f07-4599e6fb0960\") " pod="openshift-multus/multus-7hwgl"
May 25 12:13:09.582563 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.582533    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/80908050-ac10-44a0-9f07-4599e6fb0960-cnibin\") pod \"multus-7hwgl\" (UID: \"80908050-ac10-44a0-9f07-4599e6fb0960\") " pod="openshift-multus/multus-7hwgl"
May 25 12:13:09.582563 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.582544    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/80908050-ac10-44a0-9f07-4599e6fb0960-os-release\") pod \"multus-7hwgl\" (UID: \"80908050-ac10-44a0-9f07-4599e6fb0960\") " pod="openshift-multus/multus-7hwgl"
May 25 12:13:09.582563 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.582540    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3ba2b664-042c-4db1-92e7-bf94eb2a7802-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4nqql\" (UID: \"3ba2b664-042c-4db1-92e7-bf94eb2a7802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nqql"
May 25 12:13:09.583052 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.582575    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3ba2b664-042c-4db1-92e7-bf94eb2a7802-registration-dir\") pod \"aws-ebs-csi-driver-node-4nqql\" (UID: \"3ba2b664-042c-4db1-92e7-bf94eb2a7802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nqql"
May 25 12:13:09.583052 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.582602    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/72a67a33-cbd6-49ec-a72d-65cdf815803a-agent-certs\") pod \"konnectivity-agent-jpm95\" (UID: \"72a67a33-cbd6-49ec-a72d-65cdf815803a\") " pod="kube-system/konnectivity-agent-jpm95"
May 25 12:13:09.583052 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.582611    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3ba2b664-042c-4db1-92e7-bf94eb2a7802-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4nqql\" (UID: \"3ba2b664-042c-4db1-92e7-bf94eb2a7802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nqql"
May 25 12:13:09.583052 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.582625    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/94f59572-7f6a-47f6-8091-56651d472fcf-run-systemd\") pod \"ovnkube-node-jxpcm\" (UID: \"94f59572-7f6a-47f6-8091-56651d472fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.583052 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.582662    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/94f59572-7f6a-47f6-8091-56651d472fcf-host-cni-bin\") pod \"ovnkube-node-jxpcm\" (UID: \"94f59572-7f6a-47f6-8091-56651d472fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.583052 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.582668    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3ba2b664-042c-4db1-92e7-bf94eb2a7802-registration-dir\") pod \"aws-ebs-csi-driver-node-4nqql\" (UID: \"3ba2b664-042c-4db1-92e7-bf94eb2a7802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nqql"
May 25 12:13:09.583052 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.582685    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/94f59572-7f6a-47f6-8091-56651d472fcf-ovnkube-config\") pod \"ovnkube-node-jxpcm\" (UID: \"94f59572-7f6a-47f6-8091-56651d472fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.583052 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.582691    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/94f59572-7f6a-47f6-8091-56651d472fcf-run-systemd\") pod \"ovnkube-node-jxpcm\" (UID: \"94f59572-7f6a-47f6-8091-56651d472fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.583052 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.582715    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/94f59572-7f6a-47f6-8091-56651d472fcf-host-cni-bin\") pod \"ovnkube-node-jxpcm\" (UID: \"94f59572-7f6a-47f6-8091-56651d472fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.583052 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.582709    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ggqlh\" (UniqueName: \"kubernetes.io/projected/ab76ee4f-2142-41d5-99e2-0de355f6564d-kube-api-access-ggqlh\") pod \"node-resolver-dtbvj\" (UID: \"ab76ee4f-2142-41d5-99e2-0de355f6564d\") " pod="openshift-dns/node-resolver-dtbvj"
May 25 12:13:09.583052 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.582744    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/94f59572-7f6a-47f6-8091-56651d472fcf-systemd-units\") pod \"ovnkube-node-jxpcm\" (UID: \"94f59572-7f6a-47f6-8091-56651d472fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.583052 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.582759    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/94f59572-7f6a-47f6-8091-56651d472fcf-run-ovn\") pod \"ovnkube-node-jxpcm\" (UID: \"94f59572-7f6a-47f6-8091-56651d472fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.583052 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.582786    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30f9c156-1292-4927-9159-eba19cf76012-metrics-certs\") pod \"network-metrics-daemon-nwlcx\" (UID: \"30f9c156-1292-4927-9159-eba19cf76012\") " pod="openshift-multus/network-metrics-daemon-nwlcx"
May 25 12:13:09.583052 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.582809    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/80908050-ac10-44a0-9f07-4599e6fb0960-multus-daemon-config\") pod \"multus-7hwgl\" (UID: \"80908050-ac10-44a0-9f07-4599e6fb0960\") " pod="openshift-multus/multus-7hwgl"
May 25 12:13:09.583052 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.582826    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/94f59572-7f6a-47f6-8091-56651d472fcf-systemd-units\") pod \"ovnkube-node-jxpcm\" (UID: \"94f59572-7f6a-47f6-8091-56651d472fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.583052 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.582836    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b77364d1-80eb-4af1-9764-ca0f0858270a-serviceca\") pod \"node-ca-brxmt\" (UID: \"b77364d1-80eb-4af1-9764-ca0f0858270a\") " pod="openshift-image-registry/node-ca-brxmt"
May 25 12:13:09.583052 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.582859    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d776d0f6-f465-4279-81fc-33535ec8ea83-host\") pod \"tuned-vzv5c\" (UID: \"d776d0f6-f465-4279-81fc-33535ec8ea83\") " pod="openshift-cluster-node-tuning-operator/tuned-vzv5c"
May 25 12:13:09.583765 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.582881    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5mg5l\" (UniqueName: \"kubernetes.io/projected/d776d0f6-f465-4279-81fc-33535ec8ea83-kube-api-access-5mg5l\") pod \"tuned-vzv5c\" (UID: \"d776d0f6-f465-4279-81fc-33535ec8ea83\") " pod="openshift-cluster-node-tuning-operator/tuned-vzv5c"
May 25 12:13:09.583765 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.582903    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/94f59572-7f6a-47f6-8091-56651d472fcf-ovn-node-metrics-cert\") pod \"ovnkube-node-jxpcm\" (UID: \"94f59572-7f6a-47f6-8091-56651d472fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.583765 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.582927    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxnd9\" (UniqueName: \"kubernetes.io/projected/30f9c156-1292-4927-9159-eba19cf76012-kube-api-access-fxnd9\") pod \"network-metrics-daemon-nwlcx\" (UID: \"30f9c156-1292-4927-9159-eba19cf76012\") " pod="openshift-multus/network-metrics-daemon-nwlcx"
May 25 12:13:09.583765 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.582952    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/798578bf-c83f-402e-97ad-b3944200e4f6-system-cni-dir\") pod \"multus-additional-cni-plugins-6tbkc\" (UID: \"798578bf-c83f-402e-97ad-b3944200e4f6\") " pod="openshift-multus/multus-additional-cni-plugins-6tbkc"
May 25 12:13:09.583765 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.582943    2555 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
May 25 12:13:09.583765 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.582993    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d776d0f6-f465-4279-81fc-33535ec8ea83-etc-kubernetes\") pod \"tuned-vzv5c\" (UID: \"d776d0f6-f465-4279-81fc-33535ec8ea83\") " pod="openshift-cluster-node-tuning-operator/tuned-vzv5c"
May 25 12:13:09.583765 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.583003    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/94f59572-7f6a-47f6-8091-56651d472fcf-env-overrides\") pod \"ovnkube-node-jxpcm\" (UID: \"94f59572-7f6a-47f6-8091-56651d472fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.583765 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.583017    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d776d0f6-f465-4279-81fc-33535ec8ea83-etc-sysctl-d\") pod \"tuned-vzv5c\" (UID: \"d776d0f6-f465-4279-81fc-33535ec8ea83\") " pod="openshift-cluster-node-tuning-operator/tuned-vzv5c"
May 25 12:13:09.583765 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.583043    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/94f59572-7f6a-47f6-8091-56651d472fcf-node-log\") pod \"ovnkube-node-jxpcm\" (UID: \"94f59572-7f6a-47f6-8091-56651d472fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.583765 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.583063    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d776d0f6-f465-4279-81fc-33535ec8ea83-host\") pod \"tuned-vzv5c\" (UID: \"d776d0f6-f465-4279-81fc-33535ec8ea83\") " pod="openshift-cluster-node-tuning-operator/tuned-vzv5c"
May 25 12:13:09.583765 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.583077    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d776d0f6-f465-4279-81fc-33535ec8ea83-etc-systemd\") pod \"tuned-vzv5c\" (UID: \"d776d0f6-f465-4279-81fc-33535ec8ea83\") " pod="openshift-cluster-node-tuning-operator/tuned-vzv5c"
May 25 12:13:09.583765 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.583188    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/94f59572-7f6a-47f6-8091-56651d472fcf-ovnkube-config\") pod \"ovnkube-node-jxpcm\" (UID: \"94f59572-7f6a-47f6-8091-56651d472fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.583765 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.583195    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/798578bf-c83f-402e-97ad-b3944200e4f6-system-cni-dir\") pod \"multus-additional-cni-plugins-6tbkc\" (UID: \"798578bf-c83f-402e-97ad-b3944200e4f6\") " pod="openshift-multus/multus-additional-cni-plugins-6tbkc"
May 25 12:13:09.583765 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.583222    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d776d0f6-f465-4279-81fc-33535ec8ea83-etc-kubernetes\") pod \"tuned-vzv5c\" (UID: \"d776d0f6-f465-4279-81fc-33535ec8ea83\") " pod="openshift-cluster-node-tuning-operator/tuned-vzv5c"
May 25 12:13:09.583765 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.583223    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/94f59572-7f6a-47f6-8091-56651d472fcf-run-ovn\") pod \"ovnkube-node-jxpcm\" (UID: \"94f59572-7f6a-47f6-8091-56651d472fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.583765 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.583234    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/798578bf-c83f-402e-97ad-b3944200e4f6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6tbkc\" (UID: \"798578bf-c83f-402e-97ad-b3944200e4f6\") " pod="openshift-multus/multus-additional-cni-plugins-6tbkc"
May 25 12:13:09.583765 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.583255    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/94f59572-7f6a-47f6-8091-56651d472fcf-node-log\") pod \"ovnkube-node-jxpcm\" (UID: \"94f59572-7f6a-47f6-8091-56651d472fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.583765 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.583274    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/80908050-ac10-44a0-9f07-4599e6fb0960-multus-conf-dir\") pod \"multus-7hwgl\" (UID: \"80908050-ac10-44a0-9f07-4599e6fb0960\") " pod="openshift-multus/multus-7hwgl"
May 25 12:13:09.584491 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.583284    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d776d0f6-f465-4279-81fc-33535ec8ea83-etc-systemd\") pod \"tuned-vzv5c\" (UID: \"d776d0f6-f465-4279-81fc-33535ec8ea83\") " pod="openshift-cluster-node-tuning-operator/tuned-vzv5c"
May 25 12:13:09.584491 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.583305    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d776d0f6-f465-4279-81fc-33535ec8ea83-etc-sysctl-d\") pod \"tuned-vzv5c\" (UID: \"d776d0f6-f465-4279-81fc-33535ec8ea83\") " pod="openshift-cluster-node-tuning-operator/tuned-vzv5c"
May 25 12:13:09.584491 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.583344    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/80908050-ac10-44a0-9f07-4599e6fb0960-multus-conf-dir\") pod \"multus-7hwgl\" (UID: \"80908050-ac10-44a0-9f07-4599e6fb0960\") " pod="openshift-multus/multus-7hwgl"
May 25 12:13:09.584491 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.583403    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gd562\" (UniqueName: \"kubernetes.io/projected/80908050-ac10-44a0-9f07-4599e6fb0960-kube-api-access-gd562\") pod \"multus-7hwgl\" (UID: \"80908050-ac10-44a0-9f07-4599e6fb0960\") " pod="openshift-multus/multus-7hwgl"
May 25 12:13:09.584491 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.583443    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/798578bf-c83f-402e-97ad-b3944200e4f6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6tbkc\" (UID: \"798578bf-c83f-402e-97ad-b3944200e4f6\") " pod="openshift-multus/multus-additional-cni-plugins-6tbkc"
May 25 12:13:09.584491 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.583460    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/798578bf-c83f-402e-97ad-b3944200e4f6-os-release\") pod \"multus-additional-cni-plugins-6tbkc\" (UID: \"798578bf-c83f-402e-97ad-b3944200e4f6\") " pod="openshift-multus/multus-additional-cni-plugins-6tbkc"
May 25 12:13:09.584491 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.583568    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/80908050-ac10-44a0-9f07-4599e6fb0960-host-run-netns\") pod \"multus-7hwgl\" (UID: \"80908050-ac10-44a0-9f07-4599e6fb0960\") " pod="openshift-multus/multus-7hwgl"
May 25 12:13:09.584491 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.583618    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/798578bf-c83f-402e-97ad-b3944200e4f6-os-release\") pod \"multus-additional-cni-plugins-6tbkc\" (UID: \"798578bf-c83f-402e-97ad-b3944200e4f6\") " pod="openshift-multus/multus-additional-cni-plugins-6tbkc"
May 25 12:13:09.584491 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.583630    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b77364d1-80eb-4af1-9764-ca0f0858270a-serviceca\") pod \"node-ca-brxmt\" (UID: \"b77364d1-80eb-4af1-9764-ca0f0858270a\") " pod="openshift-image-registry/node-ca-brxmt"
May 25 12:13:09.584491 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.583685    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/80908050-ac10-44a0-9f07-4599e6fb0960-host-run-netns\") pod \"multus-7hwgl\" (UID: \"80908050-ac10-44a0-9f07-4599e6fb0960\") " pod="openshift-multus/multus-7hwgl"
May 25 12:13:09.584491 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.583692    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/80908050-ac10-44a0-9f07-4599e6fb0960-host-var-lib-cni-multus\") pod \"multus-7hwgl\" (UID: \"80908050-ac10-44a0-9f07-4599e6fb0960\") " pod="openshift-multus/multus-7hwgl"
May 25 12:13:09.584491 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.583720    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3ba2b664-042c-4db1-92e7-bf94eb2a7802-socket-dir\") pod \"aws-ebs-csi-driver-node-4nqql\" (UID: \"3ba2b664-042c-4db1-92e7-bf94eb2a7802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nqql"
May 25 12:13:09.584491 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.583723    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/80908050-ac10-44a0-9f07-4599e6fb0960-host-var-lib-cni-multus\") pod \"multus-7hwgl\" (UID: \"80908050-ac10-44a0-9f07-4599e6fb0960\") " pod="openshift-multus/multus-7hwgl"
May 25 12:13:09.584491 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.583759    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d776d0f6-f465-4279-81fc-33535ec8ea83-etc-modprobe-d\") pod \"tuned-vzv5c\" (UID: \"d776d0f6-f465-4279-81fc-33535ec8ea83\") " pod="openshift-cluster-node-tuning-operator/tuned-vzv5c"
May 25 12:13:09.584491 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.583816    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d776d0f6-f465-4279-81fc-33535ec8ea83-etc-sysconfig\") pod \"tuned-vzv5c\" (UID: \"d776d0f6-f465-4279-81fc-33535ec8ea83\") " pod="openshift-cluster-node-tuning-operator/tuned-vzv5c"
May 25 12:13:09.584491 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.583817    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3ba2b664-042c-4db1-92e7-bf94eb2a7802-socket-dir\") pod \"aws-ebs-csi-driver-node-4nqql\" (UID: \"3ba2b664-042c-4db1-92e7-bf94eb2a7802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nqql"
May 25 12:13:09.584491 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.583824    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/80908050-ac10-44a0-9f07-4599e6fb0960-multus-daemon-config\") pod \"multus-7hwgl\" (UID: \"80908050-ac10-44a0-9f07-4599e6fb0960\") " pod="openshift-multus/multus-7hwgl"
May 25 12:13:09.584491 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.583868    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d776d0f6-f465-4279-81fc-33535ec8ea83-etc-sysconfig\") pod \"tuned-vzv5c\" (UID: \"d776d0f6-f465-4279-81fc-33535ec8ea83\") " pod="openshift-cluster-node-tuning-operator/tuned-vzv5c"
May 25 12:13:09.585272 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.583873    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d776d0f6-f465-4279-81fc-33535ec8ea83-etc-modprobe-d\") pod \"tuned-vzv5c\" (UID: \"d776d0f6-f465-4279-81fc-33535ec8ea83\") " pod="openshift-cluster-node-tuning-operator/tuned-vzv5c"
May 25 12:13:09.585272 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.583870    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3ba2b664-042c-4db1-92e7-bf94eb2a7802-sys-fs\") pod \"aws-ebs-csi-driver-node-4nqql\" (UID: \"3ba2b664-042c-4db1-92e7-bf94eb2a7802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nqql"
May 25 12:13:09.585272 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.583906    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hcm7w\" (UniqueName: \"kubernetes.io/projected/3ba2b664-042c-4db1-92e7-bf94eb2a7802-kube-api-access-hcm7w\") pod \"aws-ebs-csi-driver-node-4nqql\" (UID: \"3ba2b664-042c-4db1-92e7-bf94eb2a7802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nqql"
May 25 12:13:09.585272 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.583915    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3ba2b664-042c-4db1-92e7-bf94eb2a7802-sys-fs\") pod \"aws-ebs-csi-driver-node-4nqql\" (UID: \"3ba2b664-042c-4db1-92e7-bf94eb2a7802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nqql"
May 25 12:13:09.585272 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.583931    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ab76ee4f-2142-41d5-99e2-0de355f6564d-hosts-file\") pod \"node-resolver-dtbvj\" (UID: \"ab76ee4f-2142-41d5-99e2-0de355f6564d\") " pod="openshift-dns/node-resolver-dtbvj"
May 25 12:13:09.585272 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.583956    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d776d0f6-f465-4279-81fc-33535ec8ea83-tmp\") pod \"tuned-vzv5c\" (UID: \"d776d0f6-f465-4279-81fc-33535ec8ea83\") " pod="openshift-cluster-node-tuning-operator/tuned-vzv5c"
May 25 12:13:09.585272 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.584004    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ab76ee4f-2142-41d5-99e2-0de355f6564d-hosts-file\") pod \"node-resolver-dtbvj\" (UID: \"ab76ee4f-2142-41d5-99e2-0de355f6564d\") " pod="openshift-dns/node-resolver-dtbvj"
May 25 12:13:09.585272 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.584036    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/94f59572-7f6a-47f6-8091-56651d472fcf-host-slash\") pod \"ovnkube-node-jxpcm\" (UID: \"94f59572-7f6a-47f6-8091-56651d472fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.585272 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.584062    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/94f59572-7f6a-47f6-8091-56651d472fcf-run-openvswitch\") pod \"ovnkube-node-jxpcm\" (UID: \"94f59572-7f6a-47f6-8091-56651d472fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.585272 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.584089    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/53719c4f-a55b-4140-9d2b-1d690a6bc5bb-host-slash\") pod \"iptables-alerter-49hs2\" (UID: \"53719c4f-a55b-4140-9d2b-1d690a6bc5bb\") " pod="openshift-network-operator/iptables-alerter-49hs2"
May 25 12:13:09.585272 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.584117    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3ba2b664-042c-4db1-92e7-bf94eb2a7802-device-dir\") pod \"aws-ebs-csi-driver-node-4nqql\" (UID: \"3ba2b664-042c-4db1-92e7-bf94eb2a7802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nqql"
May 25 12:13:09.585272 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.584123    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/94f59572-7f6a-47f6-8091-56651d472fcf-host-slash\") pod \"ovnkube-node-jxpcm\" (UID: \"94f59572-7f6a-47f6-8091-56651d472fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.585272 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.584140    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/94f59572-7f6a-47f6-8091-56651d472fcf-run-openvswitch\") pod \"ovnkube-node-jxpcm\" (UID: \"94f59572-7f6a-47f6-8091-56651d472fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.585272 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.584170    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d776d0f6-f465-4279-81fc-33535ec8ea83-sys\") pod \"tuned-vzv5c\" (UID: \"d776d0f6-f465-4279-81fc-33535ec8ea83\") " pod="openshift-cluster-node-tuning-operator/tuned-vzv5c"
May 25 12:13:09.585272 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.584192    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3ba2b664-042c-4db1-92e7-bf94eb2a7802-device-dir\") pod \"aws-ebs-csi-driver-node-4nqql\" (UID: \"3ba2b664-042c-4db1-92e7-bf94eb2a7802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nqql"
May 25 12:13:09.585272 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.584200    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-clwtf\" (UniqueName: \"kubernetes.io/projected/94f59572-7f6a-47f6-8091-56651d472fcf-kube-api-access-clwtf\") pod \"ovnkube-node-jxpcm\" (UID: \"94f59572-7f6a-47f6-8091-56651d472fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.585272 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.584207    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d776d0f6-f465-4279-81fc-33535ec8ea83-sys\") pod \"tuned-vzv5c\" (UID: \"d776d0f6-f465-4279-81fc-33535ec8ea83\") " pod="openshift-cluster-node-tuning-operator/tuned-vzv5c"
May 25 12:13:09.586010 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.584234    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk77x\" (UniqueName: \"kubernetes.io/projected/53719c4f-a55b-4140-9d2b-1d690a6bc5bb-kube-api-access-fk77x\") pod \"iptables-alerter-49hs2\" (UID: \"53719c4f-a55b-4140-9d2b-1d690a6bc5bb\") " pod="openshift-network-operator/iptables-alerter-49hs2"
May 25 12:13:09.586010 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.584266    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mm8gt\" (UniqueName: \"kubernetes.io/projected/798578bf-c83f-402e-97ad-b3944200e4f6-kube-api-access-mm8gt\") pod \"multus-additional-cni-plugins-6tbkc\" (UID: \"798578bf-c83f-402e-97ad-b3944200e4f6\") " pod="openshift-multus/multus-additional-cni-plugins-6tbkc"
May 25 12:13:09.586010 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.584289    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/80908050-ac10-44a0-9f07-4599e6fb0960-system-cni-dir\") pod \"multus-7hwgl\" (UID: \"80908050-ac10-44a0-9f07-4599e6fb0960\") " pod="openshift-multus/multus-7hwgl"
May 25 12:13:09.586010 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.584331    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/80908050-ac10-44a0-9f07-4599e6fb0960-host-var-lib-kubelet\") pod \"multus-7hwgl\" (UID: \"80908050-ac10-44a0-9f07-4599e6fb0960\") " pod="openshift-multus/multus-7hwgl"
May 25 12:13:09.586010 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.584364    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/798578bf-c83f-402e-97ad-b3944200e4f6-cni-binary-copy\") pod \"multus-additional-cni-plugins-6tbkc\" (UID: \"798578bf-c83f-402e-97ad-b3944200e4f6\") " pod="openshift-multus/multus-additional-cni-plugins-6tbkc"
May 25 12:13:09.586010 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.584387    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/80908050-ac10-44a0-9f07-4599e6fb0960-system-cni-dir\") pod \"multus-7hwgl\" (UID: \"80908050-ac10-44a0-9f07-4599e6fb0960\") " pod="openshift-multus/multus-7hwgl"
May 25 12:13:09.586010 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.584391    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3ba2b664-042c-4db1-92e7-bf94eb2a7802-etc-selinux\") pod \"aws-ebs-csi-driver-node-4nqql\" (UID: \"3ba2b664-042c-4db1-92e7-bf94eb2a7802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nqql"
May 25 12:13:09.586010 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.584423    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b77364d1-80eb-4af1-9764-ca0f0858270a-host\") pod \"node-ca-brxmt\" (UID: \"b77364d1-80eb-4af1-9764-ca0f0858270a\") " pod="openshift-image-registry/node-ca-brxmt"
May 25 12:13:09.586010 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.584449    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d776d0f6-f465-4279-81fc-33535ec8ea83-etc-tuned\") pod \"tuned-vzv5c\" (UID: \"d776d0f6-f465-4279-81fc-33535ec8ea83\") " pod="openshift-cluster-node-tuning-operator/tuned-vzv5c"
May 25 12:13:09.586010 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.584474    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/94f59572-7f6a-47f6-8091-56651d472fcf-host-cni-netd\") pod \"ovnkube-node-jxpcm\" (UID: \"94f59572-7f6a-47f6-8091-56651d472fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.586010 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.584479    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3ba2b664-042c-4db1-92e7-bf94eb2a7802-etc-selinux\") pod \"aws-ebs-csi-driver-node-4nqql\" (UID: \"3ba2b664-042c-4db1-92e7-bf94eb2a7802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nqql"
May 25 12:13:09.586010 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.584500    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/80908050-ac10-44a0-9f07-4599e6fb0960-cni-binary-copy\") pod \"multus-7hwgl\" (UID: \"80908050-ac10-44a0-9f07-4599e6fb0960\") " pod="openshift-multus/multus-7hwgl"
May 25 12:13:09.586010 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.584513    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b77364d1-80eb-4af1-9764-ca0f0858270a-host\") pod \"node-ca-brxmt\" (UID: \"b77364d1-80eb-4af1-9764-ca0f0858270a\") " pod="openshift-image-registry/node-ca-brxmt"
May 25 12:13:09.586010 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.584524    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/80908050-ac10-44a0-9f07-4599e6fb0960-multus-socket-dir-parent\") pod \"multus-7hwgl\" (UID: \"80908050-ac10-44a0-9f07-4599e6fb0960\") " pod="openshift-multus/multus-7hwgl"
May 25 12:13:09.586010 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.584527    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/94f59572-7f6a-47f6-8091-56651d472fcf-host-cni-netd\") pod \"ovnkube-node-jxpcm\" (UID: \"94f59572-7f6a-47f6-8091-56651d472fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.586010 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.584424    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/80908050-ac10-44a0-9f07-4599e6fb0960-host-var-lib-kubelet\") pod \"multus-7hwgl\" (UID: \"80908050-ac10-44a0-9f07-4599e6fb0960\") " pod="openshift-multus/multus-7hwgl"
May 25 12:13:09.586010 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.584549    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/72a67a33-cbd6-49ec-a72d-65cdf815803a-konnectivity-ca\") pod \"konnectivity-agent-jpm95\" (UID: \"72a67a33-cbd6-49ec-a72d-65cdf815803a\") " pod="kube-system/konnectivity-agent-jpm95"
May 25 12:13:09.586728 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.584606    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/80908050-ac10-44a0-9f07-4599e6fb0960-multus-socket-dir-parent\") pod \"multus-7hwgl\" (UID: \"80908050-ac10-44a0-9f07-4599e6fb0960\") " pod="openshift-multus/multus-7hwgl"
May 25 12:13:09.586728 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.584637    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94f59572-7f6a-47f6-8091-56651d472fcf-host-run-ovn-kubernetes\") pod \"ovnkube-node-jxpcm\" (UID: \"94f59572-7f6a-47f6-8091-56651d472fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.586728 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.584681    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45vpt\" (UniqueName: \"kubernetes.io/projected/2d835613-0b9d-4b52-bc2f-a40577b71208-kube-api-access-45vpt\") pod \"network-check-target-4wlq9\" (UID: \"2d835613-0b9d-4b52-bc2f-a40577b71208\") " pod="openshift-network-diagnostics/network-check-target-4wlq9"
May 25 12:13:09.586728 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.584708    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/80908050-ac10-44a0-9f07-4599e6fb0960-host-var-lib-cni-bin\") pod \"multus-7hwgl\" (UID: \"80908050-ac10-44a0-9f07-4599e6fb0960\") " pod="openshift-multus/multus-7hwgl"
May 25 12:13:09.586728 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.584733    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/80908050-ac10-44a0-9f07-4599e6fb0960-host-run-multus-certs\") pod \"multus-7hwgl\" (UID: \"80908050-ac10-44a0-9f07-4599e6fb0960\") " pod="openshift-multus/multus-7hwgl"
May 25 12:13:09.586728 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.584764    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d776d0f6-f465-4279-81fc-33535ec8ea83-etc-sysctl-conf\") pod \"tuned-vzv5c\" (UID: \"d776d0f6-f465-4279-81fc-33535ec8ea83\") " pod="openshift-cluster-node-tuning-operator/tuned-vzv5c"
May 25 12:13:09.586728 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.584788    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/94f59572-7f6a-47f6-8091-56651d472fcf-etc-openvswitch\") pod \"ovnkube-node-jxpcm\" (UID: \"94f59572-7f6a-47f6-8091-56651d472fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.586728 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.584813    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/80908050-ac10-44a0-9f07-4599e6fb0960-multus-cni-dir\") pod \"multus-7hwgl\" (UID: \"80908050-ac10-44a0-9f07-4599e6fb0960\") " pod="openshift-multus/multus-7hwgl"
May 25 12:13:09.586728 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.584848    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/798578bf-c83f-402e-97ad-b3944200e4f6-cnibin\") pod \"multus-additional-cni-plugins-6tbkc\" (UID: \"798578bf-c83f-402e-97ad-b3944200e4f6\") " pod="openshift-multus/multus-additional-cni-plugins-6tbkc"
May 25 12:13:09.586728 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.584874    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/798578bf-c83f-402e-97ad-b3944200e4f6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6tbkc\" (UID: \"798578bf-c83f-402e-97ad-b3944200e4f6\") " pod="openshift-multus/multus-additional-cni-plugins-6tbkc"
May 25 12:13:09.586728 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.584897    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-p4bgf\" (UniqueName: \"kubernetes.io/projected/b77364d1-80eb-4af1-9764-ca0f0858270a-kube-api-access-p4bgf\") pod \"node-ca-brxmt\" (UID: \"b77364d1-80eb-4af1-9764-ca0f0858270a\") " pod="openshift-image-registry/node-ca-brxmt"
May 25 12:13:09.586728 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.584941    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/80908050-ac10-44a0-9f07-4599e6fb0960-host-run-k8s-cni-cncf-io\") pod \"multus-7hwgl\" (UID: \"80908050-ac10-44a0-9f07-4599e6fb0960\") " pod="openshift-multus/multus-7hwgl"
May 25 12:13:09.586728 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.584964    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/80908050-ac10-44a0-9f07-4599e6fb0960-hostroot\") pod \"multus-7hwgl\" (UID: \"80908050-ac10-44a0-9f07-4599e6fb0960\") " pod="openshift-multus/multus-7hwgl"
May 25 12:13:09.586728 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.584987    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab76ee4f-2142-41d5-99e2-0de355f6564d-tmp-dir\") pod \"node-resolver-dtbvj\" (UID: \"ab76ee4f-2142-41d5-99e2-0de355f6564d\") " pod="openshift-dns/node-resolver-dtbvj"
May 25 12:13:09.586728 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.585010    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/94f59572-7f6a-47f6-8091-56651d472fcf-log-socket\") pod \"ovnkube-node-jxpcm\" (UID: \"94f59572-7f6a-47f6-8091-56651d472fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.586728 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.585023    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/80908050-ac10-44a0-9f07-4599e6fb0960-cni-binary-copy\") pod \"multus-7hwgl\" (UID: \"80908050-ac10-44a0-9f07-4599e6fb0960\") " pod="openshift-multus/multus-7hwgl"
May 25 12:13:09.586728 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.585021    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/798578bf-c83f-402e-97ad-b3944200e4f6-cni-binary-copy\") pod \"multus-additional-cni-plugins-6tbkc\" (UID: \"798578bf-c83f-402e-97ad-b3944200e4f6\") " pod="openshift-multus/multus-additional-cni-plugins-6tbkc"
May 25 12:13:09.587252 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.585041    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/94f59572-7f6a-47f6-8091-56651d472fcf-ovnkube-script-lib\") pod \"ovnkube-node-jxpcm\" (UID: \"94f59572-7f6a-47f6-8091-56651d472fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.587252 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.585084    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/53719c4f-a55b-4140-9d2b-1d690a6bc5bb-iptables-alerter-script\") pod \"iptables-alerter-49hs2\" (UID: \"53719c4f-a55b-4140-9d2b-1d690a6bc5bb\") " pod="openshift-network-operator/iptables-alerter-49hs2"
May 25 12:13:09.587252 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.585122    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/798578bf-c83f-402e-97ad-b3944200e4f6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6tbkc\" (UID: \"798578bf-c83f-402e-97ad-b3944200e4f6\") " pod="openshift-multus/multus-additional-cni-plugins-6tbkc"
May 25 12:13:09.587252 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.585149    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/94f59572-7f6a-47f6-8091-56651d472fcf-host-run-netns\") pod \"ovnkube-node-jxpcm\" (UID: \"94f59572-7f6a-47f6-8091-56651d472fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.587252 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.585176    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94f59572-7f6a-47f6-8091-56651d472fcf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jxpcm\" (UID: \"94f59572-7f6a-47f6-8091-56651d472fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.587252 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.585240    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94f59572-7f6a-47f6-8091-56651d472fcf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jxpcm\" (UID: \"94f59572-7f6a-47f6-8091-56651d472fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.587252 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.585491    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/72a67a33-cbd6-49ec-a72d-65cdf815803a-konnectivity-ca\") pod \"konnectivity-agent-jpm95\" (UID: \"72a67a33-cbd6-49ec-a72d-65cdf815803a\") " pod="kube-system/konnectivity-agent-jpm95"
May 25 12:13:09.587252 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.585500    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/94f59572-7f6a-47f6-8091-56651d472fcf-ovnkube-script-lib\") pod \"ovnkube-node-jxpcm\" (UID: \"94f59572-7f6a-47f6-8091-56651d472fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.587252 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.585551    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/94f59572-7f6a-47f6-8091-56651d472fcf-host-run-netns\") pod \"ovnkube-node-jxpcm\" (UID: \"94f59572-7f6a-47f6-8091-56651d472fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.587252 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.585595    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/798578bf-c83f-402e-97ad-b3944200e4f6-cnibin\") pod \"multus-additional-cni-plugins-6tbkc\" (UID: \"798578bf-c83f-402e-97ad-b3944200e4f6\") " pod="openshift-multus/multus-additional-cni-plugins-6tbkc"
May 25 12:13:09.587252 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.585623    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/80908050-ac10-44a0-9f07-4599e6fb0960-multus-cni-dir\") pod \"multus-7hwgl\" (UID: \"80908050-ac10-44a0-9f07-4599e6fb0960\") " pod="openshift-multus/multus-7hwgl"
May 25 12:13:09.587252 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.585637    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/798578bf-c83f-402e-97ad-b3944200e4f6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6tbkc\" (UID: \"798578bf-c83f-402e-97ad-b3944200e4f6\") " pod="openshift-multus/multus-additional-cni-plugins-6tbkc"
May 25 12:13:09.587252 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.585686    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94f59572-7f6a-47f6-8091-56651d472fcf-host-run-ovn-kubernetes\") pod \"ovnkube-node-jxpcm\" (UID: \"94f59572-7f6a-47f6-8091-56651d472fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.587252 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.585709    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/80908050-ac10-44a0-9f07-4599e6fb0960-host-run-multus-certs\") pod \"multus-7hwgl\" (UID: \"80908050-ac10-44a0-9f07-4599e6fb0960\") " pod="openshift-multus/multus-7hwgl"
May 25 12:13:09.587252 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.585761    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/80908050-ac10-44a0-9f07-4599e6fb0960-host-var-lib-cni-bin\") pod \"multus-7hwgl\" (UID: \"80908050-ac10-44a0-9f07-4599e6fb0960\") " pod="openshift-multus/multus-7hwgl"
May 25 12:13:09.587252 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.585788    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d776d0f6-f465-4279-81fc-33535ec8ea83-etc-sysctl-conf\") pod \"tuned-vzv5c\" (UID: \"d776d0f6-f465-4279-81fc-33535ec8ea83\") " pod="openshift-cluster-node-tuning-operator/tuned-vzv5c"
May 25 12:13:09.587252 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.585802    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/80908050-ac10-44a0-9f07-4599e6fb0960-hostroot\") pod \"multus-7hwgl\" (UID: \"80908050-ac10-44a0-9f07-4599e6fb0960\") " pod="openshift-multus/multus-7hwgl"
May 25 12:13:09.587864 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.585834    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/94f59572-7f6a-47f6-8091-56651d472fcf-etc-openvswitch\") pod \"ovnkube-node-jxpcm\" (UID: \"94f59572-7f6a-47f6-8091-56651d472fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.587864 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.586009    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/798578bf-c83f-402e-97ad-b3944200e4f6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6tbkc\" (UID: \"798578bf-c83f-402e-97ad-b3944200e4f6\") " pod="openshift-multus/multus-additional-cni-plugins-6tbkc"
May 25 12:13:09.587864 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.586011    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/80908050-ac10-44a0-9f07-4599e6fb0960-host-run-k8s-cni-cncf-io\") pod \"multus-7hwgl\" (UID: \"80908050-ac10-44a0-9f07-4599e6fb0960\") " pod="openshift-multus/multus-7hwgl"
May 25 12:13:09.587864 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.586046    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/94f59572-7f6a-47f6-8091-56651d472fcf-log-socket\") pod \"ovnkube-node-jxpcm\" (UID: \"94f59572-7f6a-47f6-8091-56651d472fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.587864 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.586103    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab76ee4f-2142-41d5-99e2-0de355f6564d-tmp-dir\") pod \"node-resolver-dtbvj\" (UID: \"ab76ee4f-2142-41d5-99e2-0de355f6564d\") " pod="openshift-dns/node-resolver-dtbvj"
May 25 12:13:09.587864 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.586664    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/94f59572-7f6a-47f6-8091-56651d472fcf-ovn-node-metrics-cert\") pod \"ovnkube-node-jxpcm\" (UID: \"94f59572-7f6a-47f6-8091-56651d472fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.587864 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.586775    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/72a67a33-cbd6-49ec-a72d-65cdf815803a-agent-certs\") pod \"konnectivity-agent-jpm95\" (UID: \"72a67a33-cbd6-49ec-a72d-65cdf815803a\") " pod="kube-system/konnectivity-agent-jpm95"
May 25 12:13:09.587864 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.586933    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d776d0f6-f465-4279-81fc-33535ec8ea83-tmp\") pod \"tuned-vzv5c\" (UID: \"d776d0f6-f465-4279-81fc-33535ec8ea83\") " pod="openshift-cluster-node-tuning-operator/tuned-vzv5c"
May 25 12:13:09.588198 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.588006    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d776d0f6-f465-4279-81fc-33535ec8ea83-etc-tuned\") pod \"tuned-vzv5c\" (UID: \"d776d0f6-f465-4279-81fc-33535ec8ea83\") " pod="openshift-cluster-node-tuning-operator/tuned-vzv5c"
May 25 12:13:09.589982 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.589933    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggqlh\" (UniqueName: \"kubernetes.io/projected/ab76ee4f-2142-41d5-99e2-0de355f6564d-kube-api-access-ggqlh\") pod \"node-resolver-dtbvj\" (UID: \"ab76ee4f-2142-41d5-99e2-0de355f6564d\") " pod="openshift-dns/node-resolver-dtbvj"
May 25 12:13:09.590066 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.590036    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mg5l\" (UniqueName: \"kubernetes.io/projected/d776d0f6-f465-4279-81fc-33535ec8ea83-kube-api-access-5mg5l\") pod \"tuned-vzv5c\" (UID: \"d776d0f6-f465-4279-81fc-33535ec8ea83\") " pod="openshift-cluster-node-tuning-operator/tuned-vzv5c"
May 25 12:13:09.590483 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.590464    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd562\" (UniqueName: \"kubernetes.io/projected/80908050-ac10-44a0-9f07-4599e6fb0960-kube-api-access-gd562\") pod \"multus-7hwgl\" (UID: \"80908050-ac10-44a0-9f07-4599e6fb0960\") " pod="openshift-multus/multus-7hwgl"
May 25 12:13:09.590786 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.590757    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcm7w\" (UniqueName: \"kubernetes.io/projected/3ba2b664-042c-4db1-92e7-bf94eb2a7802-kube-api-access-hcm7w\") pod \"aws-ebs-csi-driver-node-4nqql\" (UID: \"3ba2b664-042c-4db1-92e7-bf94eb2a7802\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nqql"
May 25 12:13:09.591550 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.591528    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clwtf\" (UniqueName: \"kubernetes.io/projected/94f59572-7f6a-47f6-8091-56651d472fcf-kube-api-access-clwtf\") pod \"ovnkube-node-jxpcm\" (UID: \"94f59572-7f6a-47f6-8091-56651d472fcf\") " pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.593757 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.593735    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm8gt\" (UniqueName: \"kubernetes.io/projected/798578bf-c83f-402e-97ad-b3944200e4f6-kube-api-access-mm8gt\") pod \"multus-additional-cni-plugins-6tbkc\" (UID: \"798578bf-c83f-402e-97ad-b3944200e4f6\") " pod="openshift-multus/multus-additional-cni-plugins-6tbkc"
May 25 12:13:09.594866 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.594847    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4bgf\" (UniqueName: \"kubernetes.io/projected/b77364d1-80eb-4af1-9764-ca0f0858270a-kube-api-access-p4bgf\") pod \"node-ca-brxmt\" (UID: \"b77364d1-80eb-4af1-9764-ca0f0858270a\") " pod="openshift-image-registry/node-ca-brxmt"
May 25 12:13:09.660437 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.660412    2555 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 12:13:09.686311 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.686241    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30f9c156-1292-4927-9159-eba19cf76012-metrics-certs\") pod \"network-metrics-daemon-nwlcx\" (UID: \"30f9c156-1292-4927-9159-eba19cf76012\") " pod="openshift-multus/network-metrics-daemon-nwlcx"
May 25 12:13:09.686311 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.686277    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fxnd9\" (UniqueName: \"kubernetes.io/projected/30f9c156-1292-4927-9159-eba19cf76012-kube-api-access-fxnd9\") pod \"network-metrics-daemon-nwlcx\" (UID: \"30f9c156-1292-4927-9159-eba19cf76012\") " pod="openshift-multus/network-metrics-daemon-nwlcx"
May 25 12:13:09.686489 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.686318    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/53719c4f-a55b-4140-9d2b-1d690a6bc5bb-host-slash\") pod \"iptables-alerter-49hs2\" (UID: \"53719c4f-a55b-4140-9d2b-1d690a6bc5bb\") " pod="openshift-network-operator/iptables-alerter-49hs2"
May 25 12:13:09.686489 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.686343    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fk77x\" (UniqueName: \"kubernetes.io/projected/53719c4f-a55b-4140-9d2b-1d690a6bc5bb-kube-api-access-fk77x\") pod \"iptables-alerter-49hs2\" (UID: \"53719c4f-a55b-4140-9d2b-1d690a6bc5bb\") " pod="openshift-network-operator/iptables-alerter-49hs2"
May 25 12:13:09.686489 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.686375    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-45vpt\" (UniqueName: \"kubernetes.io/projected/2d835613-0b9d-4b52-bc2f-a40577b71208-kube-api-access-45vpt\") pod \"network-check-target-4wlq9\" (UID: \"2d835613-0b9d-4b52-bc2f-a40577b71208\") " pod="openshift-network-diagnostics/network-check-target-4wlq9"
May 25 12:13:09.686489 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.686404    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/53719c4f-a55b-4140-9d2b-1d690a6bc5bb-iptables-alerter-script\") pod \"iptables-alerter-49hs2\" (UID: \"53719c4f-a55b-4140-9d2b-1d690a6bc5bb\") " pod="openshift-network-operator/iptables-alerter-49hs2"
May 25 12:13:09.686929 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.686757    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/53719c4f-a55b-4140-9d2b-1d690a6bc5bb-host-slash\") pod \"iptables-alerter-49hs2\" (UID: \"53719c4f-a55b-4140-9d2b-1d690a6bc5bb\") " pod="openshift-network-operator/iptables-alerter-49hs2"
May 25 12:13:09.686929 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:09.686827    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:13:09.687097 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:09.686934    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30f9c156-1292-4927-9159-eba19cf76012-metrics-certs podName:30f9c156-1292-4927-9159-eba19cf76012 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:10.186885162 +0000 UTC m=+3.146935549 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30f9c156-1292-4927-9159-eba19cf76012-metrics-certs") pod "network-metrics-daemon-nwlcx" (UID: "30f9c156-1292-4927-9159-eba19cf76012") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:13:09.687097 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.686978    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/53719c4f-a55b-4140-9d2b-1d690a6bc5bb-iptables-alerter-script\") pod \"iptables-alerter-49hs2\" (UID: \"53719c4f-a55b-4140-9d2b-1d690a6bc5bb\") " pod="openshift-network-operator/iptables-alerter-49hs2"
May 25 12:13:09.692769 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:09.692751    2555 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 12:13:09.692872 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:09.692776    2555 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 12:13:09.692872 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:09.692790    2555 projected.go:194] Error preparing data for projected volume kube-api-access-45vpt for pod openshift-network-diagnostics/network-check-target-4wlq9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:13:09.692872 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:09.692856    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d835613-0b9d-4b52-bc2f-a40577b71208-kube-api-access-45vpt podName:2d835613-0b9d-4b52-bc2f-a40577b71208 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:10.192831461 +0000 UTC m=+3.152881864 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-45vpt" (UniqueName: "kubernetes.io/projected/2d835613-0b9d-4b52-bc2f-a40577b71208-kube-api-access-45vpt") pod "network-check-target-4wlq9" (UID: "2d835613-0b9d-4b52-bc2f-a40577b71208") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:13:09.695240 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.695218    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk77x\" (UniqueName: \"kubernetes.io/projected/53719c4f-a55b-4140-9d2b-1d690a6bc5bb-kube-api-access-fk77x\") pod \"iptables-alerter-49hs2\" (UID: \"53719c4f-a55b-4140-9d2b-1d690a6bc5bb\") " pod="openshift-network-operator/iptables-alerter-49hs2"
May 25 12:13:09.695323 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.695240    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxnd9\" (UniqueName: \"kubernetes.io/projected/30f9c156-1292-4927-9159-eba19cf76012-kube-api-access-fxnd9\") pod \"network-metrics-daemon-nwlcx\" (UID: \"30f9c156-1292-4927-9159-eba19cf76012\") " pod="openshift-multus/network-metrics-daemon-nwlcx"
May 25 12:13:09.774710 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.774678    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vzv5c"
May 25 12:13:09.781936 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.781914    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7hwgl"
May 25 12:13:09.790704 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.790685    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6tbkc"
May 25 12:13:09.796490 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.796470    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:09.803052 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.803032    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nqql"
May 25 12:13:09.809577 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.809558    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jpm95"
May 25 12:13:09.816297 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.816280    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dtbvj"
May 25 12:13:09.821699 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.821679    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-brxmt"
May 25 12:13:09.830156 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:09.830141    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-49hs2"
May 25 12:13:10.191903 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:10.191883    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30f9c156-1292-4927-9159-eba19cf76012-metrics-certs\") pod \"network-metrics-daemon-nwlcx\" (UID: \"30f9c156-1292-4927-9159-eba19cf76012\") " pod="openshift-multus/network-metrics-daemon-nwlcx"
May 25 12:13:10.192039 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:10.191979    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:13:10.192039 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:10.192026    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30f9c156-1292-4927-9159-eba19cf76012-metrics-certs podName:30f9c156-1292-4927-9159-eba19cf76012 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:11.192013809 +0000 UTC m=+4.152064193 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30f9c156-1292-4927-9159-eba19cf76012-metrics-certs") pod "network-metrics-daemon-nwlcx" (UID: "30f9c156-1292-4927-9159-eba19cf76012") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:13:10.253267 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:10.253236    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd776d0f6_f465_4279_81fc_33535ec8ea83.slice/crio-f8cd9a97e2e81ff6d37a4b4e8edfd501854160f01a11969eb7c437a2323df2b1 WatchSource:0}: Error finding container f8cd9a97e2e81ff6d37a4b4e8edfd501854160f01a11969eb7c437a2323df2b1: Status 404 returned error can't find the container with id f8cd9a97e2e81ff6d37a4b4e8edfd501854160f01a11969eb7c437a2323df2b1
May 25 12:13:10.253858 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:10.253832    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80908050_ac10_44a0_9f07_4599e6fb0960.slice/crio-03450b22665b40dd5bc8c517981e4a54dac21cd761b5f4f9dde3fa73a774df0d WatchSource:0}: Error finding container 03450b22665b40dd5bc8c517981e4a54dac21cd761b5f4f9dde3fa73a774df0d: Status 404 returned error can't find the container with id 03450b22665b40dd5bc8c517981e4a54dac21cd761b5f4f9dde3fa73a774df0d
May 25 12:13:10.256063 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:10.255906    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94f59572_7f6a_47f6_8091_56651d472fcf.slice/crio-427c6dbc8bd772616160db930bc4dc456a34bb4ca99e6fc5c05ecae11e431fbe WatchSource:0}: Error finding container 427c6dbc8bd772616160db930bc4dc456a34bb4ca99e6fc5c05ecae11e431fbe: Status 404 returned error can't find the container with id 427c6dbc8bd772616160db930bc4dc456a34bb4ca99e6fc5c05ecae11e431fbe
May 25 12:13:10.258371 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:10.258352    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ba2b664_042c_4db1_92e7_bf94eb2a7802.slice/crio-d5df1596b7d5284857d507ce2c6840f863969385c04b2508531ffb576e5272e9 WatchSource:0}: Error finding container d5df1596b7d5284857d507ce2c6840f863969385c04b2508531ffb576e5272e9: Status 404 returned error can't find the container with id d5df1596b7d5284857d507ce2c6840f863969385c04b2508531ffb576e5272e9
May 25 12:13:10.259959 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:10.259865    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53719c4f_a55b_4140_9d2b_1d690a6bc5bb.slice/crio-e662b78454aeddefc19d555026f3d33c595655126c9e27a6efb91c4b2ae95951 WatchSource:0}: Error finding container e662b78454aeddefc19d555026f3d33c595655126c9e27a6efb91c4b2ae95951: Status 404 returned error can't find the container with id e662b78454aeddefc19d555026f3d33c595655126c9e27a6efb91c4b2ae95951
May 25 12:13:10.260230 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:10.260155    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod798578bf_c83f_402e_97ad_b3944200e4f6.slice/crio-7cf46eefaafcb5cf1e7bd7ea5595b7dbda229175b223835e64b2ea614bee2245 WatchSource:0}: Error finding container 7cf46eefaafcb5cf1e7bd7ea5595b7dbda229175b223835e64b2ea614bee2245: Status 404 returned error can't find the container with id 7cf46eefaafcb5cf1e7bd7ea5595b7dbda229175b223835e64b2ea614bee2245
May 25 12:13:10.261071 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:10.261033    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab76ee4f_2142_41d5_99e2_0de355f6564d.slice/crio-10249705a47fbf3779934dc3d8d126e23d2129db6a7036449e76698e7d93ec0b WatchSource:0}: Error finding container 10249705a47fbf3779934dc3d8d126e23d2129db6a7036449e76698e7d93ec0b: Status 404 returned error can't find the container with id 10249705a47fbf3779934dc3d8d126e23d2129db6a7036449e76698e7d93ec0b
May 25 12:13:10.261804 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:10.261786    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb77364d1_80eb_4af1_9764_ca0f0858270a.slice/crio-9f6c330cb04fc14349d10269f9d2dd76e73b3ef21a6439b88f2493b0a14dd4ec WatchSource:0}: Error finding container 9f6c330cb04fc14349d10269f9d2dd76e73b3ef21a6439b88f2493b0a14dd4ec: Status 404 returned error can't find the container with id 9f6c330cb04fc14349d10269f9d2dd76e73b3ef21a6439b88f2493b0a14dd4ec
May 25 12:13:10.263172 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:13:10.263154    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72a67a33_cbd6_49ec_a72d_65cdf815803a.slice/crio-8cb44fe03623ad39150554b7276fa22ec0a6faf173d2e6741e8321a64fc44fd9 WatchSource:0}: Error finding container 8cb44fe03623ad39150554b7276fa22ec0a6faf173d2e6741e8321a64fc44fd9: Status 404 returned error can't find the container with id 8cb44fe03623ad39150554b7276fa22ec0a6faf173d2e6741e8321a64fc44fd9
May 25 12:13:10.292233 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:10.292208    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-45vpt\" (UniqueName: \"kubernetes.io/projected/2d835613-0b9d-4b52-bc2f-a40577b71208-kube-api-access-45vpt\") pod \"network-check-target-4wlq9\" (UID: \"2d835613-0b9d-4b52-bc2f-a40577b71208\") " pod="openshift-network-diagnostics/network-check-target-4wlq9"
May 25 12:13:10.292350 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:10.292336    2555 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 12:13:10.292413 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:10.292358    2555 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 12:13:10.292413 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:10.292371    2555 projected.go:194] Error preparing data for projected volume kube-api-access-45vpt for pod openshift-network-diagnostics/network-check-target-4wlq9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:13:10.292507 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:10.292418    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d835613-0b9d-4b52-bc2f-a40577b71208-kube-api-access-45vpt podName:2d835613-0b9d-4b52-bc2f-a40577b71208 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:11.292405326 +0000 UTC m=+4.252455717 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-45vpt" (UniqueName: "kubernetes.io/projected/2d835613-0b9d-4b52-bc2f-a40577b71208-kube-api-access-45vpt") pod "network-check-target-4wlq9" (UID: "2d835613-0b9d-4b52-bc2f-a40577b71208") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:13:10.513269 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:10.513082    2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-24 12:08:08 +0000 UTC" deadline="2027-12-09 04:14:30.013832684 +0000 UTC"
May 25 12:13:10.513269 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:10.513256    2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13504h1m19.500581136s"
May 25 12:13:10.594705 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:10.594508    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4wlq9"
May 25 12:13:10.594820 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:10.594681    2555 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4wlq9" podUID="2d835613-0b9d-4b52-bc2f-a40577b71208"
May 25 12:13:10.605759 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:10.605414    2555 generic.go:358] "Generic (PLEG): container finished" podID="fa0099618f996f4bb5ffb3952e434c51" containerID="d733ce4f30734895d8032a1a92e34c58a291964f892a1b9cc41a3bfe7542119d" exitCode=0
May 25 12:13:10.605759 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:10.605483    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-255.ec2.internal" event={"ID":"fa0099618f996f4bb5ffb3952e434c51","Type":"ContainerDied","Data":"d733ce4f30734895d8032a1a92e34c58a291964f892a1b9cc41a3bfe7542119d"}
May 25 12:13:10.610840 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:10.610761    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-255.ec2.internal" event={"ID":"e477f180b7e0e366bb0a0001192721d9","Type":"ContainerStarted","Data":"a8d2f500a829d276d92f2ff0f110f4b71a7e9c1c35f4b16eef22965c865635fa"}
May 25 12:13:10.612969 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:10.612924    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-brxmt" event={"ID":"b77364d1-80eb-4af1-9764-ca0f0858270a","Type":"ContainerStarted","Data":"9f6c330cb04fc14349d10269f9d2dd76e73b3ef21a6439b88f2493b0a14dd4ec"}
May 25 12:13:10.615536 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:10.615504    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dtbvj" event={"ID":"ab76ee4f-2142-41d5-99e2-0de355f6564d","Type":"ContainerStarted","Data":"10249705a47fbf3779934dc3d8d126e23d2129db6a7036449e76698e7d93ec0b"}
May 25 12:13:10.617159 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:10.617137    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6tbkc" event={"ID":"798578bf-c83f-402e-97ad-b3944200e4f6","Type":"ContainerStarted","Data":"7cf46eefaafcb5cf1e7bd7ea5595b7dbda229175b223835e64b2ea614bee2245"}
May 25 12:13:10.620034 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:10.620011    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-49hs2" event={"ID":"53719c4f-a55b-4140-9d2b-1d690a6bc5bb","Type":"ContainerStarted","Data":"e662b78454aeddefc19d555026f3d33c595655126c9e27a6efb91c4b2ae95951"}
May 25 12:13:10.621837 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:10.621814    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm" event={"ID":"94f59572-7f6a-47f6-8091-56651d472fcf","Type":"ContainerStarted","Data":"427c6dbc8bd772616160db930bc4dc456a34bb4ca99e6fc5c05ecae11e431fbe"}
May 25 12:13:10.623951 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:10.623899    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7hwgl" event={"ID":"80908050-ac10-44a0-9f07-4599e6fb0960","Type":"ContainerStarted","Data":"03450b22665b40dd5bc8c517981e4a54dac21cd761b5f4f9dde3fa73a774df0d"}
May 25 12:13:10.627487 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:10.627461    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jpm95" event={"ID":"72a67a33-cbd6-49ec-a72d-65cdf815803a","Type":"ContainerStarted","Data":"8cb44fe03623ad39150554b7276fa22ec0a6faf173d2e6741e8321a64fc44fd9"}
May 25 12:13:10.630214 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:10.630194    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nqql" event={"ID":"3ba2b664-042c-4db1-92e7-bf94eb2a7802","Type":"ContainerStarted","Data":"d5df1596b7d5284857d507ce2c6840f863969385c04b2508531ffb576e5272e9"}
May 25 12:13:10.637127 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:10.637086    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vzv5c" event={"ID":"d776d0f6-f465-4279-81fc-33535ec8ea83","Type":"ContainerStarted","Data":"f8cd9a97e2e81ff6d37a4b4e8edfd501854160f01a11969eb7c437a2323df2b1"}
May 25 12:13:11.201273 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:11.200655    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30f9c156-1292-4927-9159-eba19cf76012-metrics-certs\") pod \"network-metrics-daemon-nwlcx\" (UID: \"30f9c156-1292-4927-9159-eba19cf76012\") " pod="openshift-multus/network-metrics-daemon-nwlcx"
May 25 12:13:11.201273 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:11.200847    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:13:11.201273 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:11.200908    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30f9c156-1292-4927-9159-eba19cf76012-metrics-certs podName:30f9c156-1292-4927-9159-eba19cf76012 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:13.200890956 +0000 UTC m=+6.160941346 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30f9c156-1292-4927-9159-eba19cf76012-metrics-certs") pod "network-metrics-daemon-nwlcx" (UID: "30f9c156-1292-4927-9159-eba19cf76012") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:13:11.302118 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:11.302080    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-45vpt\" (UniqueName: \"kubernetes.io/projected/2d835613-0b9d-4b52-bc2f-a40577b71208-kube-api-access-45vpt\") pod \"network-check-target-4wlq9\" (UID: \"2d835613-0b9d-4b52-bc2f-a40577b71208\") " pod="openshift-network-diagnostics/network-check-target-4wlq9"
May 25 12:13:11.309759 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:11.308893    2555 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 12:13:11.309759 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:11.308919    2555 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 12:13:11.309759 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:11.308933    2555 projected.go:194] Error preparing data for projected volume kube-api-access-45vpt for pod openshift-network-diagnostics/network-check-target-4wlq9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:13:11.309759 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:11.309016    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d835613-0b9d-4b52-bc2f-a40577b71208-kube-api-access-45vpt podName:2d835613-0b9d-4b52-bc2f-a40577b71208 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:13.30898087 +0000 UTC m=+6.269031272 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-45vpt" (UniqueName: "kubernetes.io/projected/2d835613-0b9d-4b52-bc2f-a40577b71208-kube-api-access-45vpt") pod "network-check-target-4wlq9" (UID: "2d835613-0b9d-4b52-bc2f-a40577b71208") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:13:11.594874 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:11.594444    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwlcx"
May 25 12:13:11.594874 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:11.594571    2555 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwlcx" podUID="30f9c156-1292-4927-9159-eba19cf76012"
May 25 12:13:11.666418 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:11.666006    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-255.ec2.internal" event={"ID":"fa0099618f996f4bb5ffb3952e434c51","Type":"ContainerStarted","Data":"a52882c0a8cea3e624917318c3984869534b9adb63f8f32c4c7fac25e4933f01"}
May 25 12:13:11.680073 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:11.680009    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-255.ec2.internal" podStartSLOduration=3.679992691 podStartE2EDuration="3.679992691s" podCreationTimestamp="2026-05-25 12:13:08 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:13:10.632676741 +0000 UTC m=+3.592727147" watchObservedRunningTime="2026-05-25 12:13:11.679992691 +0000 UTC m=+4.640043096"
May 25 12:13:11.680859 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:11.680811    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-255.ec2.internal" podStartSLOduration=3.680797323 podStartE2EDuration="3.680797323s" podCreationTimestamp="2026-05-25 12:13:08 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:13:11.679711927 +0000 UTC m=+4.639762337" watchObservedRunningTime="2026-05-25 12:13:11.680797323 +0000 UTC m=+4.640847729"
May 25 12:13:12.594195 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:12.593699    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4wlq9"
May 25 12:13:12.594195 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:12.593837    2555 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4wlq9" podUID="2d835613-0b9d-4b52-bc2f-a40577b71208"
May 25 12:13:13.218367 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:13.217762    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30f9c156-1292-4927-9159-eba19cf76012-metrics-certs\") pod \"network-metrics-daemon-nwlcx\" (UID: \"30f9c156-1292-4927-9159-eba19cf76012\") " pod="openshift-multus/network-metrics-daemon-nwlcx"
May 25 12:13:13.218367 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:13.217905    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:13:13.218367 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:13.217964    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30f9c156-1292-4927-9159-eba19cf76012-metrics-certs podName:30f9c156-1292-4927-9159-eba19cf76012 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:17.217947189 +0000 UTC m=+10.177997574 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30f9c156-1292-4927-9159-eba19cf76012-metrics-certs") pod "network-metrics-daemon-nwlcx" (UID: "30f9c156-1292-4927-9159-eba19cf76012") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:13:13.318894 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:13.318826    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-45vpt\" (UniqueName: \"kubernetes.io/projected/2d835613-0b9d-4b52-bc2f-a40577b71208-kube-api-access-45vpt\") pod \"network-check-target-4wlq9\" (UID: \"2d835613-0b9d-4b52-bc2f-a40577b71208\") " pod="openshift-network-diagnostics/network-check-target-4wlq9"
May 25 12:13:13.319096 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:13.319058    2555 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 12:13:13.319096 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:13.319078    2555 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 12:13:13.319096 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:13.319090    2555 projected.go:194] Error preparing data for projected volume kube-api-access-45vpt for pod openshift-network-diagnostics/network-check-target-4wlq9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:13:13.319322 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:13.319150    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d835613-0b9d-4b52-bc2f-a40577b71208-kube-api-access-45vpt podName:2d835613-0b9d-4b52-bc2f-a40577b71208 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:17.319129755 +0000 UTC m=+10.279180140 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-45vpt" (UniqueName: "kubernetes.io/projected/2d835613-0b9d-4b52-bc2f-a40577b71208-kube-api-access-45vpt") pod "network-check-target-4wlq9" (UID: "2d835613-0b9d-4b52-bc2f-a40577b71208") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:13:13.595309 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:13.594754    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwlcx"
May 25 12:13:13.595309 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:13.594883    2555 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwlcx" podUID="30f9c156-1292-4927-9159-eba19cf76012"
May 25 12:13:14.594561 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:14.594528    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4wlq9"
May 25 12:13:14.595080 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:14.594664    2555 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4wlq9" podUID="2d835613-0b9d-4b52-bc2f-a40577b71208"
May 25 12:13:15.594616 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:15.594580    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwlcx"
May 25 12:13:15.595057 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:15.594744    2555 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwlcx" podUID="30f9c156-1292-4927-9159-eba19cf76012"
May 25 12:13:16.594624 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:16.594589    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4wlq9"
May 25 12:13:16.595076 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:16.594733    2555 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4wlq9" podUID="2d835613-0b9d-4b52-bc2f-a40577b71208"
May 25 12:13:17.253206 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:17.253171    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30f9c156-1292-4927-9159-eba19cf76012-metrics-certs\") pod \"network-metrics-daemon-nwlcx\" (UID: \"30f9c156-1292-4927-9159-eba19cf76012\") " pod="openshift-multus/network-metrics-daemon-nwlcx"
May 25 12:13:17.253359 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:17.253326    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:13:17.253398 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:17.253387    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30f9c156-1292-4927-9159-eba19cf76012-metrics-certs podName:30f9c156-1292-4927-9159-eba19cf76012 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:25.253365428 +0000 UTC m=+18.213415856 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30f9c156-1292-4927-9159-eba19cf76012-metrics-certs") pod "network-metrics-daemon-nwlcx" (UID: "30f9c156-1292-4927-9159-eba19cf76012") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:13:17.353966 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:17.353869    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-45vpt\" (UniqueName: \"kubernetes.io/projected/2d835613-0b9d-4b52-bc2f-a40577b71208-kube-api-access-45vpt\") pod \"network-check-target-4wlq9\" (UID: \"2d835613-0b9d-4b52-bc2f-a40577b71208\") " pod="openshift-network-diagnostics/network-check-target-4wlq9"
May 25 12:13:17.354120 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:17.354020    2555 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 12:13:17.354120 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:17.354045    2555 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 12:13:17.354120 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:17.354057    2555 projected.go:194] Error preparing data for projected volume kube-api-access-45vpt for pod openshift-network-diagnostics/network-check-target-4wlq9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:13:17.354120 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:17.354114    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d835613-0b9d-4b52-bc2f-a40577b71208-kube-api-access-45vpt podName:2d835613-0b9d-4b52-bc2f-a40577b71208 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:25.3540967 +0000 UTC m=+18.314147088 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-45vpt" (UniqueName: "kubernetes.io/projected/2d835613-0b9d-4b52-bc2f-a40577b71208-kube-api-access-45vpt") pod "network-check-target-4wlq9" (UID: "2d835613-0b9d-4b52-bc2f-a40577b71208") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:13:17.595309 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:17.594902    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwlcx"
May 25 12:13:17.595309 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:17.595022    2555 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwlcx" podUID="30f9c156-1292-4927-9159-eba19cf76012"
May 25 12:13:18.594524 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:18.594494    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4wlq9"
May 25 12:13:18.594727 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:18.594603    2555 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4wlq9" podUID="2d835613-0b9d-4b52-bc2f-a40577b71208"
May 25 12:13:19.594059 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:19.594029    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwlcx"
May 25 12:13:19.594449 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:19.594142    2555 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwlcx" podUID="30f9c156-1292-4927-9159-eba19cf76012"
May 25 12:13:20.594006 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:20.593976    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4wlq9"
May 25 12:13:20.594174 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:20.594090    2555 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4wlq9" podUID="2d835613-0b9d-4b52-bc2f-a40577b71208"
May 25 12:13:21.594103 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:21.594064    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwlcx"
May 25 12:13:21.594256 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:21.594199    2555 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwlcx" podUID="30f9c156-1292-4927-9159-eba19cf76012"
May 25 12:13:22.594525 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:22.594485    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4wlq9"
May 25 12:13:22.594931 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:22.594596    2555 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4wlq9" podUID="2d835613-0b9d-4b52-bc2f-a40577b71208"
May 25 12:13:23.594578 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:23.594544    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwlcx"
May 25 12:13:23.594936 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:23.594657    2555 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwlcx" podUID="30f9c156-1292-4927-9159-eba19cf76012"
May 25 12:13:24.594306 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:24.594274    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4wlq9"
May 25 12:13:24.594521 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:24.594385    2555 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4wlq9" podUID="2d835613-0b9d-4b52-bc2f-a40577b71208"
May 25 12:13:25.317227 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:25.317201    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30f9c156-1292-4927-9159-eba19cf76012-metrics-certs\") pod \"network-metrics-daemon-nwlcx\" (UID: \"30f9c156-1292-4927-9159-eba19cf76012\") " pod="openshift-multus/network-metrics-daemon-nwlcx"
May 25 12:13:25.317616 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:25.317312    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:13:25.317616 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:25.317366    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30f9c156-1292-4927-9159-eba19cf76012-metrics-certs podName:30f9c156-1292-4927-9159-eba19cf76012 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:41.317350992 +0000 UTC m=+34.277401378 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30f9c156-1292-4927-9159-eba19cf76012-metrics-certs") pod "network-metrics-daemon-nwlcx" (UID: "30f9c156-1292-4927-9159-eba19cf76012") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:13:25.418458 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:25.418430    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-45vpt\" (UniqueName: \"kubernetes.io/projected/2d835613-0b9d-4b52-bc2f-a40577b71208-kube-api-access-45vpt\") pod \"network-check-target-4wlq9\" (UID: \"2d835613-0b9d-4b52-bc2f-a40577b71208\") " pod="openshift-network-diagnostics/network-check-target-4wlq9"
May 25 12:13:25.418634 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:25.418601    2555 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 12:13:25.418634 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:25.418622    2555 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 12:13:25.418634 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:25.418634    2555 projected.go:194] Error preparing data for projected volume kube-api-access-45vpt for pod openshift-network-diagnostics/network-check-target-4wlq9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:13:25.418805 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:25.418711    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d835613-0b9d-4b52-bc2f-a40577b71208-kube-api-access-45vpt podName:2d835613-0b9d-4b52-bc2f-a40577b71208 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:41.418692362 +0000 UTC m=+34.378742762 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-45vpt" (UniqueName: "kubernetes.io/projected/2d835613-0b9d-4b52-bc2f-a40577b71208-kube-api-access-45vpt") pod "network-check-target-4wlq9" (UID: "2d835613-0b9d-4b52-bc2f-a40577b71208") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:13:25.594669 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:25.594586    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwlcx"
May 25 12:13:25.594816 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:25.594707    2555 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwlcx" podUID="30f9c156-1292-4927-9159-eba19cf76012"
May 25 12:13:26.593606 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:26.593572    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4wlq9"
May 25 12:13:26.594041 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:26.593745    2555 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4wlq9" podUID="2d835613-0b9d-4b52-bc2f-a40577b71208"
May 25 12:13:27.594899 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:27.594867    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwlcx"
May 25 12:13:27.595308 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:27.594985    2555 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwlcx" podUID="30f9c156-1292-4927-9159-eba19cf76012"
May 25 12:13:28.594365 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:28.594027    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4wlq9"
May 25 12:13:28.594496 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:28.594476    2555 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4wlq9" podUID="2d835613-0b9d-4b52-bc2f-a40577b71208"
May 25 12:13:28.695490 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:28.695457    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dtbvj" event={"ID":"ab76ee4f-2142-41d5-99e2-0de355f6564d","Type":"ContainerStarted","Data":"f21e73386fd35a37458f0e0ab0c63eb03b87d1b48a4981d8d89f3f233c698e60"}
May 25 12:13:28.696828 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:28.696799    2555 generic.go:358] "Generic (PLEG): container finished" podID="798578bf-c83f-402e-97ad-b3944200e4f6" containerID="0b3a625f41050f23e1f27b84dee35e047efe9a2260e3f9084ea4184c044af368" exitCode=0
May 25 12:13:28.696963 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:28.696840    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6tbkc" event={"ID":"798578bf-c83f-402e-97ad-b3944200e4f6","Type":"ContainerDied","Data":"0b3a625f41050f23e1f27b84dee35e047efe9a2260e3f9084ea4184c044af368"}
May 25 12:13:28.699689 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:28.699665    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm" event={"ID":"94f59572-7f6a-47f6-8091-56651d472fcf","Type":"ContainerStarted","Data":"3d94b08bcb6f912ae4e50399d0f9d29891ab9a3cf8a1e564a26b854c8d84d023"}
May 25 12:13:28.699784 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:28.699698    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm" event={"ID":"94f59572-7f6a-47f6-8091-56651d472fcf","Type":"ContainerStarted","Data":"d0424464579d69e3b3c47f764c7145cf161bd560c39fe1576c499f6e6d9b0a30"}
May 25 12:13:28.699784 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:28.699714    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm" event={"ID":"94f59572-7f6a-47f6-8091-56651d472fcf","Type":"ContainerStarted","Data":"2e0cbb747286b08dcf18cb2e2439853a27ec1ce44ea57b5e2efcd46294f62294"}
May 25 12:13:28.699784 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:28.699729    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm" event={"ID":"94f59572-7f6a-47f6-8091-56651d472fcf","Type":"ContainerStarted","Data":"f12bb992d8608e09977ffe742827c41a4634182a294492a74db57458e9a14b27"}
May 25 12:13:28.699784 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:28.699741    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm" event={"ID":"94f59572-7f6a-47f6-8091-56651d472fcf","Type":"ContainerStarted","Data":"248083701ebf0645bb8b7c43ec2f1edd256bcef1b75d3b1deefa038a82d6bebd"}
May 25 12:13:28.699784 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:28.699752    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm" event={"ID":"94f59572-7f6a-47f6-8091-56651d472fcf","Type":"ContainerStarted","Data":"80e8f642357247a70b935dcc44677f00969f42ea28d14dde40e8dcb9f4d6723b"}
May 25 12:13:28.700801 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:28.700775    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7hwgl" event={"ID":"80908050-ac10-44a0-9f07-4599e6fb0960","Type":"ContainerStarted","Data":"6e617254687106531ba52014c0a162a82c01671a56d0de27f77377270e4a2138"}
May 25 12:13:28.702412 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:28.702109    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jpm95" event={"ID":"72a67a33-cbd6-49ec-a72d-65cdf815803a","Type":"ContainerStarted","Data":"e3b7641a60131dd2a5d0264a893f9212bdc0bb436762fb7cd50c85879259d440"}
May 25 12:13:28.703603 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:28.703581    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nqql" event={"ID":"3ba2b664-042c-4db1-92e7-bf94eb2a7802","Type":"ContainerStarted","Data":"9fd288270c6ccb86f201ab4a831872fe4bc9fbadf00b56328ec92f4484dad742"}
May 25 12:13:28.705002 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:28.704980    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vzv5c" event={"ID":"d776d0f6-f465-4279-81fc-33535ec8ea83","Type":"ContainerStarted","Data":"994d7ec2ea733ecdf2b66a77acd891c67f01d07d8370a2305eacd8ede42c61c3"}
May 25 12:13:28.706219 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:28.706202    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-brxmt" event={"ID":"b77364d1-80eb-4af1-9764-ca0f0858270a","Type":"ContainerStarted","Data":"38a2a865eda2e587a0c8ed602da30cbe208c4ee8df49f0f0291472b5811a28ef"}
May 25 12:13:28.727702 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:28.727624    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-dtbvj" podStartSLOduration=3.8845935430000003 podStartE2EDuration="21.727614432s" podCreationTimestamp="2026-05-25 12:13:07 +0000 UTC" firstStartedPulling="2026-05-25 12:13:10.263298044 +0000 UTC m=+3.223348427" lastFinishedPulling="2026-05-25 12:13:28.106318919 +0000 UTC m=+21.066369316" observedRunningTime="2026-05-25 12:13:28.710142339 +0000 UTC m=+21.670192745" watchObservedRunningTime="2026-05-25 12:13:28.727614432 +0000 UTC m=+21.687664865"
May 25 12:13:28.751049 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:28.751009    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-jpm95" podStartSLOduration=3.910103781 podStartE2EDuration="21.750999191s" podCreationTimestamp="2026-05-25 12:13:07 +0000 UTC" firstStartedPulling="2026-05-25 12:13:10.265548596 +0000 UTC m=+3.225598982" lastFinishedPulling="2026-05-25 12:13:28.106444 +0000 UTC m=+21.066494392" observedRunningTime="2026-05-25 12:13:28.739803808 +0000 UTC m=+21.699854217" watchObservedRunningTime="2026-05-25 12:13:28.750999191 +0000 UTC m=+21.711049587"
May 25 12:13:28.751676 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:28.751627    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-brxmt" podStartSLOduration=11.833786371 podStartE2EDuration="21.751618256s" podCreationTimestamp="2026-05-25 12:13:07 +0000 UTC" firstStartedPulling="2026-05-25 12:13:10.263595395 +0000 UTC m=+3.223645782" lastFinishedPulling="2026-05-25 12:13:20.181427271 +0000 UTC m=+13.141477667" observedRunningTime="2026-05-25 12:13:28.751133315 +0000 UTC m=+21.711183710" watchObservedRunningTime="2026-05-25 12:13:28.751618256 +0000 UTC m=+21.711668663"
May 25 12:13:28.764806 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:28.764769    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vzv5c" podStartSLOduration=3.911863288 podStartE2EDuration="21.764760238s" podCreationTimestamp="2026-05-25 12:13:07 +0000 UTC" firstStartedPulling="2026-05-25 12:13:10.254980352 +0000 UTC m=+3.215030752" lastFinishedPulling="2026-05-25 12:13:28.107877306 +0000 UTC m=+21.067927702" observedRunningTime="2026-05-25 12:13:28.764498923 +0000 UTC m=+21.724549331" watchObservedRunningTime="2026-05-25 12:13:28.764760238 +0000 UTC m=+21.724810642"
May 25 12:13:28.787557 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:28.787516    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-7hwgl" podStartSLOduration=3.796149122 podStartE2EDuration="21.787504634s" podCreationTimestamp="2026-05-25 12:13:07 +0000 UTC" firstStartedPulling="2026-05-25 12:13:10.255545391 +0000 UTC m=+3.215595794" lastFinishedPulling="2026-05-25 12:13:28.24690092 +0000 UTC m=+21.206951306" observedRunningTime="2026-05-25 12:13:28.787208673 +0000 UTC m=+21.747259077" watchObservedRunningTime="2026-05-25 12:13:28.787504634 +0000 UTC m=+21.747555020"
May 25 12:13:29.168138 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:29.168116    2555 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
May 25 12:13:29.543695 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:29.543522    2555 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-25T12:13:29.16813476Z","UUID":"38126f5b-8525-4cf5-9f93-136d47dc3a55","Handler":null,"Name":"","Endpoint":""}
May 25 12:13:29.545241 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:29.545214    2555 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
May 25 12:13:29.545241 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:29.545243    2555 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
May 25 12:13:29.593768 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:29.593741    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwlcx"
May 25 12:13:29.593933 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:29.593912    2555 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwlcx" podUID="30f9c156-1292-4927-9159-eba19cf76012"
May 25 12:13:29.710333 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:29.710301    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-49hs2" event={"ID":"53719c4f-a55b-4140-9d2b-1d690a6bc5bb","Type":"ContainerStarted","Data":"41ce05986a0146fc2aa05e0b1b764cf8b8a62facebcee91dd7a838a069a8fea5"}
May 25 12:13:29.712186 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:29.712151    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nqql" event={"ID":"3ba2b664-042c-4db1-92e7-bf94eb2a7802","Type":"ContainerStarted","Data":"6eac7e1e66ed8c748f290e51f3e4588e7ad8cacc95eb989c5f94800ba09e8bc8"}
May 25 12:13:30.594250 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:30.594218    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4wlq9"
May 25 12:13:30.594398 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:30.594327    2555 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4wlq9" podUID="2d835613-0b9d-4b52-bc2f-a40577b71208"
May 25 12:13:30.716505 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:30.716474    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nqql" event={"ID":"3ba2b664-042c-4db1-92e7-bf94eb2a7802","Type":"ContainerStarted","Data":"190e62ed44763bb6195358dab489f210e5e9f328c3a5680fb11d7d484b18ab67"}
May 25 12:13:30.731292 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:30.731242    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-49hs2" podStartSLOduration=5.886226834 podStartE2EDuration="23.731224184s" podCreationTimestamp="2026-05-25 12:13:07 +0000 UTC" firstStartedPulling="2026-05-25 12:13:10.261258733 +0000 UTC m=+3.221309130" lastFinishedPulling="2026-05-25 12:13:28.106256097 +0000 UTC m=+21.066306480" observedRunningTime="2026-05-25 12:13:29.724273561 +0000 UTC m=+22.684323967" watchObservedRunningTime="2026-05-25 12:13:30.731224184 +0000 UTC m=+23.691274594"
May 25 12:13:31.593953 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:31.593753    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwlcx"
May 25 12:13:31.594139 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:31.594070    2555 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwlcx" podUID="30f9c156-1292-4927-9159-eba19cf76012"
May 25 12:13:31.722970 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:31.722293    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm" event={"ID":"94f59572-7f6a-47f6-8091-56651d472fcf","Type":"ContainerStarted","Data":"76f99848a53e4f9fcc3402dc0173ccecf0deca087925f211548212150421a056"}
May 25 12:13:32.346549 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:32.346515    2555 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-jpm95"
May 25 12:13:32.347171 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:32.347145    2555 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-jpm95"
May 25 12:13:32.359561 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:32.359517    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nqql" podStartSLOduration=5.665141223 podStartE2EDuration="25.359499388s" podCreationTimestamp="2026-05-25 12:13:07 +0000 UTC" firstStartedPulling="2026-05-25 12:13:10.260709666 +0000 UTC m=+3.220760065" lastFinishedPulling="2026-05-25 12:13:29.955067847 +0000 UTC m=+22.915118230" observedRunningTime="2026-05-25 12:13:30.730846894 +0000 UTC m=+23.690897299" watchObservedRunningTime="2026-05-25 12:13:32.359499388 +0000 UTC m=+25.319549780"
May 25 12:13:32.593784 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:32.593755    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4wlq9"
May 25 12:13:32.593951 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:32.593868    2555 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4wlq9" podUID="2d835613-0b9d-4b52-bc2f-a40577b71208"
May 25 12:13:33.594425 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:33.594247    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwlcx"
May 25 12:13:33.595056 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:33.594499    2555 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwlcx" podUID="30f9c156-1292-4927-9159-eba19cf76012"
May 25 12:13:33.727032 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:33.726997    2555 generic.go:358] "Generic (PLEG): container finished" podID="798578bf-c83f-402e-97ad-b3944200e4f6" containerID="0a3ff667d6a0577de1eb7401fb0d5dec23690d58c5af679af7a30f2bdd644de9" exitCode=0
May 25 12:13:33.727190 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:33.727071    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6tbkc" event={"ID":"798578bf-c83f-402e-97ad-b3944200e4f6","Type":"ContainerDied","Data":"0a3ff667d6a0577de1eb7401fb0d5dec23690d58c5af679af7a30f2bdd644de9"}
May 25 12:13:33.730229 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:33.730200    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm" event={"ID":"94f59572-7f6a-47f6-8091-56651d472fcf","Type":"ContainerStarted","Data":"748d502573d1d7678230c002c8e8efed07df3961ff5a2032425a7cf61045eb36"}
May 25 12:13:33.730508 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:33.730479    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:33.730508 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:33.730507    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:33.746724 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:33.746636    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:33.768053 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:33.768020    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm" podStartSLOduration=8.878058525 podStartE2EDuration="26.768010049s" podCreationTimestamp="2026-05-25 12:13:07 +0000 UTC" firstStartedPulling="2026-05-25 12:13:10.257867801 +0000 UTC m=+3.217918184" lastFinishedPulling="2026-05-25 12:13:28.147819324 +0000 UTC m=+21.107869708" observedRunningTime="2026-05-25 12:13:33.76679816 +0000 UTC m=+26.726848564" watchObservedRunningTime="2026-05-25 12:13:33.768010049 +0000 UTC m=+26.728060453"
May 25 12:13:34.456954 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:34.456913    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jpm95"
May 25 12:13:34.457141 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:34.457083    2555 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
May 25 12:13:34.457764 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:34.457741    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jpm95"
May 25 12:13:34.594292 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:34.594264    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4wlq9"
May 25 12:13:34.594412 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:34.594379    2555 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4wlq9" podUID="2d835613-0b9d-4b52-bc2f-a40577b71208"
May 25 12:13:34.734488 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:34.734402    2555 generic.go:358] "Generic (PLEG): container finished" podID="798578bf-c83f-402e-97ad-b3944200e4f6" containerID="046e339e1bd2681683af4003631f2c9d156baef3ad0e670085652936be11be45" exitCode=0
May 25 12:13:34.734933 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:34.734487    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6tbkc" event={"ID":"798578bf-c83f-402e-97ad-b3944200e4f6","Type":"ContainerDied","Data":"046e339e1bd2681683af4003631f2c9d156baef3ad0e670085652936be11be45"}
May 25 12:13:34.735503 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:34.735485    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:34.741037 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:34.739393    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4wlq9"]
May 25 12:13:34.741037 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:34.739462    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4wlq9"
May 25 12:13:34.741037 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:34.739544    2555 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4wlq9" podUID="2d835613-0b9d-4b52-bc2f-a40577b71208"
May 25 12:13:34.741037 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:34.740772    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nwlcx"]
May 25 12:13:34.741037 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:34.740879    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwlcx"
May 25 12:13:34.741037 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:34.740975    2555 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwlcx" podUID="30f9c156-1292-4927-9159-eba19cf76012"
May 25 12:13:34.752361 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:34.752342    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:13:35.738380 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:35.738160    2555 generic.go:358] "Generic (PLEG): container finished" podID="798578bf-c83f-402e-97ad-b3944200e4f6" containerID="aab68ad9d22a41ea5f6c4c7b820baa8d92836bdc200655d3760e10ffec50adbf" exitCode=0
May 25 12:13:35.738788 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:35.738250    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6tbkc" event={"ID":"798578bf-c83f-402e-97ad-b3944200e4f6","Type":"ContainerDied","Data":"aab68ad9d22a41ea5f6c4c7b820baa8d92836bdc200655d3760e10ffec50adbf"}
May 25 12:13:36.593997 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:36.593953    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4wlq9"
May 25 12:13:36.594163 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:36.593953    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwlcx"
May 25 12:13:36.594163 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:36.594096    2555 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4wlq9" podUID="2d835613-0b9d-4b52-bc2f-a40577b71208"
May 25 12:13:36.594163 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:36.594142    2555 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwlcx" podUID="30f9c156-1292-4927-9159-eba19cf76012"
May 25 12:13:38.593962 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:38.593936    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwlcx"
May 25 12:13:38.594507 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:38.593935    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4wlq9"
May 25 12:13:38.594507 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:38.594050    2555 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwlcx" podUID="30f9c156-1292-4927-9159-eba19cf76012"
May 25 12:13:38.594507 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:38.594118    2555 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4wlq9" podUID="2d835613-0b9d-4b52-bc2f-a40577b71208"
May 25 12:13:40.594558 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:40.594492    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4wlq9"
May 25 12:13:40.594558 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:40.594500    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwlcx"
May 25 12:13:40.595138 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:40.594660    2555 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4wlq9" podUID="2d835613-0b9d-4b52-bc2f-a40577b71208"
May 25 12:13:40.595138 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:40.594769    2555 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nwlcx" podUID="30f9c156-1292-4927-9159-eba19cf76012"
May 25 12:13:40.889465 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:40.889398    2555 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-255.ec2.internal" event="NodeReady"
May 25 12:13:40.889705 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:40.889535    2555 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
May 25 12:13:40.931965 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:40.931928    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-gn5dh"]
May 25 12:13:40.968977 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:40.968944    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8mblp"]
May 25 12:13:40.969126 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:40.969099    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gn5dh"
May 25 12:13:40.971055 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:40.971027    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
May 25 12:13:40.971182 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:40.971160    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
May 25 12:13:40.971277 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:40.971250    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fp9z9\""
May 25 12:13:40.985360 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:40.985344    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gn5dh"]
May 25 12:13:40.985360 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:40.985361    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8mblp"]
May 25 12:13:40.985542 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:40.985445    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8mblp"
May 25 12:13:40.989084 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:40.989066    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
May 25 12:13:40.989369 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:40.989351    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-zkb72\""
May 25 12:13:40.990009 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:40.989836    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
May 25 12:13:40.990117 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:40.990042    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
May 25 12:13:41.145164 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:41.145095    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5b139bd-e9a1-45b9-9d31-11a98965a2e0-tmp-dir\") pod \"dns-default-gn5dh\" (UID: \"a5b139bd-e9a1-45b9-9d31-11a98965a2e0\") " pod="openshift-dns/dns-default-gn5dh"
May 25 12:13:41.145164 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:41.145142    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a5b139bd-e9a1-45b9-9d31-11a98965a2e0-config-volume\") pod \"dns-default-gn5dh\" (UID: \"a5b139bd-e9a1-45b9-9d31-11a98965a2e0\") " pod="openshift-dns/dns-default-gn5dh"
May 25 12:13:41.145164 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:41.145166    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5b139bd-e9a1-45b9-9d31-11a98965a2e0-metrics-tls\") pod \"dns-default-gn5dh\" (UID: \"a5b139bd-e9a1-45b9-9d31-11a98965a2e0\") " pod="openshift-dns/dns-default-gn5dh"
May 25 12:13:41.145406 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:41.145225    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26ec0201-e1d2-4816-944f-bbdb03159591-cert\") pod \"ingress-canary-8mblp\" (UID: \"26ec0201-e1d2-4816-944f-bbdb03159591\") " pod="openshift-ingress-canary/ingress-canary-8mblp"
May 25 12:13:41.145406 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:41.145272    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7tw5\" (UniqueName: \"kubernetes.io/projected/a5b139bd-e9a1-45b9-9d31-11a98965a2e0-kube-api-access-c7tw5\") pod \"dns-default-gn5dh\" (UID: \"a5b139bd-e9a1-45b9-9d31-11a98965a2e0\") " pod="openshift-dns/dns-default-gn5dh"
May 25 12:13:41.145406 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:41.145335    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jphzj\" (UniqueName: \"kubernetes.io/projected/26ec0201-e1d2-4816-944f-bbdb03159591-kube-api-access-jphzj\") pod \"ingress-canary-8mblp\" (UID: \"26ec0201-e1d2-4816-944f-bbdb03159591\") " pod="openshift-ingress-canary/ingress-canary-8mblp"
May 25 12:13:41.246213 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:41.246193    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jphzj\" (UniqueName: \"kubernetes.io/projected/26ec0201-e1d2-4816-944f-bbdb03159591-kube-api-access-jphzj\") pod \"ingress-canary-8mblp\" (UID: \"26ec0201-e1d2-4816-944f-bbdb03159591\") " pod="openshift-ingress-canary/ingress-canary-8mblp"
May 25 12:13:41.246325 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:41.246223    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5b139bd-e9a1-45b9-9d31-11a98965a2e0-tmp-dir\") pod \"dns-default-gn5dh\" (UID: \"a5b139bd-e9a1-45b9-9d31-11a98965a2e0\") " pod="openshift-dns/dns-default-gn5dh"
May 25 12:13:41.246325 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:41.246251    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a5b139bd-e9a1-45b9-9d31-11a98965a2e0-config-volume\") pod \"dns-default-gn5dh\" (UID: \"a5b139bd-e9a1-45b9-9d31-11a98965a2e0\") " pod="openshift-dns/dns-default-gn5dh"
May 25 12:13:41.246325 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:41.246270    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5b139bd-e9a1-45b9-9d31-11a98965a2e0-metrics-tls\") pod \"dns-default-gn5dh\" (UID: \"a5b139bd-e9a1-45b9-9d31-11a98965a2e0\") " pod="openshift-dns/dns-default-gn5dh"
May 25 12:13:41.246325 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:41.246294    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26ec0201-e1d2-4816-944f-bbdb03159591-cert\") pod \"ingress-canary-8mblp\" (UID: \"26ec0201-e1d2-4816-944f-bbdb03159591\") " pod="openshift-ingress-canary/ingress-canary-8mblp"
May 25 12:13:41.246325 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:41.246317    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-c7tw5\" (UniqueName: \"kubernetes.io/projected/a5b139bd-e9a1-45b9-9d31-11a98965a2e0-kube-api-access-c7tw5\") pod \"dns-default-gn5dh\" (UID: \"a5b139bd-e9a1-45b9-9d31-11a98965a2e0\") " pod="openshift-dns/dns-default-gn5dh"
May 25 12:13:41.246568 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:41.246375    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 12:13:41.246568 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:41.246431    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5b139bd-e9a1-45b9-9d31-11a98965a2e0-metrics-tls podName:a5b139bd-e9a1-45b9-9d31-11a98965a2e0 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:41.746413336 +0000 UTC m=+34.706463740 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a5b139bd-e9a1-45b9-9d31-11a98965a2e0-metrics-tls") pod "dns-default-gn5dh" (UID: "a5b139bd-e9a1-45b9-9d31-11a98965a2e0") : secret "dns-default-metrics-tls" not found
May 25 12:13:41.246568 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:41.246450    2555 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 12:13:41.246568 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:41.246506    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26ec0201-e1d2-4816-944f-bbdb03159591-cert podName:26ec0201-e1d2-4816-944f-bbdb03159591 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:41.746488282 +0000 UTC m=+34.706538682 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/26ec0201-e1d2-4816-944f-bbdb03159591-cert") pod "ingress-canary-8mblp" (UID: "26ec0201-e1d2-4816-944f-bbdb03159591") : secret "canary-serving-cert" not found
May 25 12:13:41.246568 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:41.246539    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5b139bd-e9a1-45b9-9d31-11a98965a2e0-tmp-dir\") pod \"dns-default-gn5dh\" (UID: \"a5b139bd-e9a1-45b9-9d31-11a98965a2e0\") " pod="openshift-dns/dns-default-gn5dh"
May 25 12:13:41.246799 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:41.246782    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a5b139bd-e9a1-45b9-9d31-11a98965a2e0-config-volume\") pod \"dns-default-gn5dh\" (UID: \"a5b139bd-e9a1-45b9-9d31-11a98965a2e0\") " pod="openshift-dns/dns-default-gn5dh"
May 25 12:13:41.255731 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:41.255712    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7tw5\" (UniqueName: \"kubernetes.io/projected/a5b139bd-e9a1-45b9-9d31-11a98965a2e0-kube-api-access-c7tw5\") pod \"dns-default-gn5dh\" (UID: \"a5b139bd-e9a1-45b9-9d31-11a98965a2e0\") " pod="openshift-dns/dns-default-gn5dh"
May 25 12:13:41.255953 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:41.255936    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jphzj\" (UniqueName: \"kubernetes.io/projected/26ec0201-e1d2-4816-944f-bbdb03159591-kube-api-access-jphzj\") pod \"ingress-canary-8mblp\" (UID: \"26ec0201-e1d2-4816-944f-bbdb03159591\") " pod="openshift-ingress-canary/ingress-canary-8mblp"
May 25 12:13:41.347194 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:41.347171    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30f9c156-1292-4927-9159-eba19cf76012-metrics-certs\") pod \"network-metrics-daemon-nwlcx\" (UID: \"30f9c156-1292-4927-9159-eba19cf76012\") " pod="openshift-multus/network-metrics-daemon-nwlcx"
May 25 12:13:41.347285 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:41.347275    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:13:41.347332 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:41.347321    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30f9c156-1292-4927-9159-eba19cf76012-metrics-certs podName:30f9c156-1292-4927-9159-eba19cf76012 nodeName:}" failed. No retries permitted until 2026-05-25 12:14:13.347310424 +0000 UTC m=+66.307360808 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30f9c156-1292-4927-9159-eba19cf76012-metrics-certs") pod "network-metrics-daemon-nwlcx" (UID: "30f9c156-1292-4927-9159-eba19cf76012") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:13:41.448401 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:41.448361    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-45vpt\" (UniqueName: \"kubernetes.io/projected/2d835613-0b9d-4b52-bc2f-a40577b71208-kube-api-access-45vpt\") pod \"network-check-target-4wlq9\" (UID: \"2d835613-0b9d-4b52-bc2f-a40577b71208\") " pod="openshift-network-diagnostics/network-check-target-4wlq9"
May 25 12:13:41.448485 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:41.448477    2555 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 12:13:41.448522 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:41.448488    2555 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 12:13:41.448522 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:41.448497    2555 projected.go:194] Error preparing data for projected volume kube-api-access-45vpt for pod openshift-network-diagnostics/network-check-target-4wlq9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:13:41.448586 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:41.448530    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d835613-0b9d-4b52-bc2f-a40577b71208-kube-api-access-45vpt podName:2d835613-0b9d-4b52-bc2f-a40577b71208 nodeName:}" failed. No retries permitted until 2026-05-25 12:14:13.44852073 +0000 UTC m=+66.408571113 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-45vpt" (UniqueName: "kubernetes.io/projected/2d835613-0b9d-4b52-bc2f-a40577b71208-kube-api-access-45vpt") pod "network-check-target-4wlq9" (UID: "2d835613-0b9d-4b52-bc2f-a40577b71208") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:13:41.750394 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:41.750367    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5b139bd-e9a1-45b9-9d31-11a98965a2e0-metrics-tls\") pod \"dns-default-gn5dh\" (UID: \"a5b139bd-e9a1-45b9-9d31-11a98965a2e0\") " pod="openshift-dns/dns-default-gn5dh"
May 25 12:13:41.751046 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:41.750412    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26ec0201-e1d2-4816-944f-bbdb03159591-cert\") pod \"ingress-canary-8mblp\" (UID: \"26ec0201-e1d2-4816-944f-bbdb03159591\") " pod="openshift-ingress-canary/ingress-canary-8mblp"
May 25 12:13:41.751046 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:41.750514    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 12:13:41.751046 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:41.750564    2555 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 12:13:41.751046 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:41.750586    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5b139bd-e9a1-45b9-9d31-11a98965a2e0-metrics-tls podName:a5b139bd-e9a1-45b9-9d31-11a98965a2e0 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:42.750565568 +0000 UTC m=+35.710615971 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a5b139bd-e9a1-45b9-9d31-11a98965a2e0-metrics-tls") pod "dns-default-gn5dh" (UID: "a5b139bd-e9a1-45b9-9d31-11a98965a2e0") : secret "dns-default-metrics-tls" not found
May 25 12:13:41.751046 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:41.750603    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26ec0201-e1d2-4816-944f-bbdb03159591-cert podName:26ec0201-e1d2-4816-944f-bbdb03159591 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:42.750594811 +0000 UTC m=+35.710645198 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/26ec0201-e1d2-4816-944f-bbdb03159591-cert") pod "ingress-canary-8mblp" (UID: "26ec0201-e1d2-4816-944f-bbdb03159591") : secret "canary-serving-cert" not found
May 25 12:13:41.754986 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:41.754964    2555 generic.go:358] "Generic (PLEG): container finished" podID="798578bf-c83f-402e-97ad-b3944200e4f6" containerID="4d0c38a43c7c0e472697e97906e12ef84a4b21d5df273ca3aff3386509fac548" exitCode=0
May 25 12:13:41.755074 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:41.755022    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6tbkc" event={"ID":"798578bf-c83f-402e-97ad-b3944200e4f6","Type":"ContainerDied","Data":"4d0c38a43c7c0e472697e97906e12ef84a4b21d5df273ca3aff3386509fac548"}
May 25 12:13:42.594445 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:42.594380    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4wlq9"
May 25 12:13:42.594582 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:42.594386    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwlcx"
May 25 12:13:42.596943 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:42.596920    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 25 12:13:42.597460 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:42.597443    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nlm6z\""
May 25 12:13:42.597460 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:42.597454    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 25 12:13:42.597597 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:42.597551    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wxdt4\""
May 25 12:13:42.597597 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:42.597568    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 25 12:13:42.756941 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:42.756916    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5b139bd-e9a1-45b9-9d31-11a98965a2e0-metrics-tls\") pod \"dns-default-gn5dh\" (UID: \"a5b139bd-e9a1-45b9-9d31-11a98965a2e0\") " pod="openshift-dns/dns-default-gn5dh"
May 25 12:13:42.756941 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:42.756944    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26ec0201-e1d2-4816-944f-bbdb03159591-cert\") pod \"ingress-canary-8mblp\" (UID: \"26ec0201-e1d2-4816-944f-bbdb03159591\") " pod="openshift-ingress-canary/ingress-canary-8mblp"
May 25 12:13:42.757350 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:42.757053    2555 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 12:13:42.757350 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:42.757055    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 12:13:42.757350 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:42.757099    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26ec0201-e1d2-4816-944f-bbdb03159591-cert podName:26ec0201-e1d2-4816-944f-bbdb03159591 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:44.757081754 +0000 UTC m=+37.717132137 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/26ec0201-e1d2-4816-944f-bbdb03159591-cert") pod "ingress-canary-8mblp" (UID: "26ec0201-e1d2-4816-944f-bbdb03159591") : secret "canary-serving-cert" not found
May 25 12:13:42.757350 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:42.757118    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5b139bd-e9a1-45b9-9d31-11a98965a2e0-metrics-tls podName:a5b139bd-e9a1-45b9-9d31-11a98965a2e0 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:44.757107631 +0000 UTC m=+37.717158017 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a5b139bd-e9a1-45b9-9d31-11a98965a2e0-metrics-tls") pod "dns-default-gn5dh" (UID: "a5b139bd-e9a1-45b9-9d31-11a98965a2e0") : secret "dns-default-metrics-tls" not found
May 25 12:13:42.759592 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:42.759569    2555 generic.go:358] "Generic (PLEG): container finished" podID="798578bf-c83f-402e-97ad-b3944200e4f6" containerID="a40e3ff26a57c08a678999149674a8978392e3a1231480b5a7e4efe1a4bed94f" exitCode=0
May 25 12:13:42.759701 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:42.759623    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6tbkc" event={"ID":"798578bf-c83f-402e-97ad-b3944200e4f6","Type":"ContainerDied","Data":"a40e3ff26a57c08a678999149674a8978392e3a1231480b5a7e4efe1a4bed94f"}
May 25 12:13:43.764117 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:43.763956    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6tbkc" event={"ID":"798578bf-c83f-402e-97ad-b3944200e4f6","Type":"ContainerStarted","Data":"69dbdeab4a27ff2037f85f7fe24e158e26db28dcee72670ad662309ddec379c0"}
May 25 12:13:43.783080 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:43.783035    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6tbkc" podStartSLOduration=5.568973327 podStartE2EDuration="36.783023086s" podCreationTimestamp="2026-05-25 12:13:07 +0000 UTC" firstStartedPulling="2026-05-25 12:13:10.262021742 +0000 UTC m=+3.222072139" lastFinishedPulling="2026-05-25 12:13:41.476071498 +0000 UTC m=+34.436121898" observedRunningTime="2026-05-25 12:13:43.781830422 +0000 UTC m=+36.741880826" watchObservedRunningTime="2026-05-25 12:13:43.783023086 +0000 UTC m=+36.743073491"
May 25 12:13:44.769062 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:44.768991    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5b139bd-e9a1-45b9-9d31-11a98965a2e0-metrics-tls\") pod \"dns-default-gn5dh\" (UID: \"a5b139bd-e9a1-45b9-9d31-11a98965a2e0\") " pod="openshift-dns/dns-default-gn5dh"
May 25 12:13:44.769062 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:44.769022    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26ec0201-e1d2-4816-944f-bbdb03159591-cert\") pod \"ingress-canary-8mblp\" (UID: \"26ec0201-e1d2-4816-944f-bbdb03159591\") " pod="openshift-ingress-canary/ingress-canary-8mblp"
May 25 12:13:44.769393 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:44.769100    2555 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 12:13:44.769393 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:44.769109    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 12:13:44.769393 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:44.769150    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26ec0201-e1d2-4816-944f-bbdb03159591-cert podName:26ec0201-e1d2-4816-944f-bbdb03159591 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:48.769137128 +0000 UTC m=+41.729187511 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/26ec0201-e1d2-4816-944f-bbdb03159591-cert") pod "ingress-canary-8mblp" (UID: "26ec0201-e1d2-4816-944f-bbdb03159591") : secret "canary-serving-cert" not found
May 25 12:13:44.769393 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:44.769163    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5b139bd-e9a1-45b9-9d31-11a98965a2e0-metrics-tls podName:a5b139bd-e9a1-45b9-9d31-11a98965a2e0 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:48.7691573 +0000 UTC m=+41.729207683 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a5b139bd-e9a1-45b9-9d31-11a98965a2e0-metrics-tls") pod "dns-default-gn5dh" (UID: "a5b139bd-e9a1-45b9-9d31-11a98965a2e0") : secret "dns-default-metrics-tls" not found
May 25 12:13:48.794915 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:48.794885    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5b139bd-e9a1-45b9-9d31-11a98965a2e0-metrics-tls\") pod \"dns-default-gn5dh\" (UID: \"a5b139bd-e9a1-45b9-9d31-11a98965a2e0\") " pod="openshift-dns/dns-default-gn5dh"
May 25 12:13:48.794915 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:48.794917    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26ec0201-e1d2-4816-944f-bbdb03159591-cert\") pod \"ingress-canary-8mblp\" (UID: \"26ec0201-e1d2-4816-944f-bbdb03159591\") " pod="openshift-ingress-canary/ingress-canary-8mblp"
May 25 12:13:48.795320 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:48.795023    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 12:13:48.795320 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:48.795065    2555 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 12:13:48.795320 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:48.795092    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5b139bd-e9a1-45b9-9d31-11a98965a2e0-metrics-tls podName:a5b139bd-e9a1-45b9-9d31-11a98965a2e0 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:56.795076491 +0000 UTC m=+49.755126874 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a5b139bd-e9a1-45b9-9d31-11a98965a2e0-metrics-tls") pod "dns-default-gn5dh" (UID: "a5b139bd-e9a1-45b9-9d31-11a98965a2e0") : secret "dns-default-metrics-tls" not found
May 25 12:13:48.795320 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:48.795109    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26ec0201-e1d2-4816-944f-bbdb03159591-cert podName:26ec0201-e1d2-4816-944f-bbdb03159591 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:56.795098272 +0000 UTC m=+49.755148655 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/26ec0201-e1d2-4816-944f-bbdb03159591-cert") pod "ingress-canary-8mblp" (UID: "26ec0201-e1d2-4816-944f-bbdb03159591") : secret "canary-serving-cert" not found
May 25 12:13:56.845474 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:56.845442    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5b139bd-e9a1-45b9-9d31-11a98965a2e0-metrics-tls\") pod \"dns-default-gn5dh\" (UID: \"a5b139bd-e9a1-45b9-9d31-11a98965a2e0\") " pod="openshift-dns/dns-default-gn5dh"
May 25 12:13:56.845474 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:13:56.845475    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26ec0201-e1d2-4816-944f-bbdb03159591-cert\") pod \"ingress-canary-8mblp\" (UID: \"26ec0201-e1d2-4816-944f-bbdb03159591\") " pod="openshift-ingress-canary/ingress-canary-8mblp"
May 25 12:13:56.845966 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:56.845570    2555 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 12:13:56.845966 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:56.845571    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 12:13:56.845966 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:56.845614    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26ec0201-e1d2-4816-944f-bbdb03159591-cert podName:26ec0201-e1d2-4816-944f-bbdb03159591 nodeName:}" failed. No retries permitted until 2026-05-25 12:14:12.845602086 +0000 UTC m=+65.805652469 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/26ec0201-e1d2-4816-944f-bbdb03159591-cert") pod "ingress-canary-8mblp" (UID: "26ec0201-e1d2-4816-944f-bbdb03159591") : secret "canary-serving-cert" not found
May 25 12:13:56.845966 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:13:56.845626    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5b139bd-e9a1-45b9-9d31-11a98965a2e0-metrics-tls podName:a5b139bd-e9a1-45b9-9d31-11a98965a2e0 nodeName:}" failed. No retries permitted until 2026-05-25 12:14:12.845620044 +0000 UTC m=+65.805670427 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a5b139bd-e9a1-45b9-9d31-11a98965a2e0-metrics-tls") pod "dns-default-gn5dh" (UID: "a5b139bd-e9a1-45b9-9d31-11a98965a2e0") : secret "dns-default-metrics-tls" not found
May 25 12:14:06.750795 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:14:06.750763    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jxpcm"
May 25 12:14:12.940522 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:14:12.940488    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5b139bd-e9a1-45b9-9d31-11a98965a2e0-metrics-tls\") pod \"dns-default-gn5dh\" (UID: \"a5b139bd-e9a1-45b9-9d31-11a98965a2e0\") " pod="openshift-dns/dns-default-gn5dh"
May 25 12:14:12.940522 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:14:12.940523    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26ec0201-e1d2-4816-944f-bbdb03159591-cert\") pod \"ingress-canary-8mblp\" (UID: \"26ec0201-e1d2-4816-944f-bbdb03159591\") " pod="openshift-ingress-canary/ingress-canary-8mblp"
May 25 12:14:12.941014 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:14:12.940596    2555 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 12:14:12.941014 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:14:12.940624    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 12:14:12.941014 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:14:12.940656    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26ec0201-e1d2-4816-944f-bbdb03159591-cert podName:26ec0201-e1d2-4816-944f-bbdb03159591 nodeName:}" failed. No retries permitted until 2026-05-25 12:14:44.940627665 +0000 UTC m=+97.900678048 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/26ec0201-e1d2-4816-944f-bbdb03159591-cert") pod "ingress-canary-8mblp" (UID: "26ec0201-e1d2-4816-944f-bbdb03159591") : secret "canary-serving-cert" not found
May 25 12:14:12.941014 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:14:12.940696    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5b139bd-e9a1-45b9-9d31-11a98965a2e0-metrics-tls podName:a5b139bd-e9a1-45b9-9d31-11a98965a2e0 nodeName:}" failed. No retries permitted until 2026-05-25 12:14:44.940683534 +0000 UTC m=+97.900733918 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a5b139bd-e9a1-45b9-9d31-11a98965a2e0-metrics-tls") pod "dns-default-gn5dh" (UID: "a5b139bd-e9a1-45b9-9d31-11a98965a2e0") : secret "dns-default-metrics-tls" not found
May 25 12:14:13.444081 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:14:13.444056    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30f9c156-1292-4927-9159-eba19cf76012-metrics-certs\") pod \"network-metrics-daemon-nwlcx\" (UID: \"30f9c156-1292-4927-9159-eba19cf76012\") " pod="openshift-multus/network-metrics-daemon-nwlcx"
May 25 12:14:13.446190 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:14:13.446172    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 25 12:14:13.454786 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:14:13.454770    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 25 12:14:13.454848 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:14:13.454818    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30f9c156-1292-4927-9159-eba19cf76012-metrics-certs podName:30f9c156-1292-4927-9159-eba19cf76012 nodeName:}" failed. No retries permitted until 2026-05-25 12:15:17.454805505 +0000 UTC m=+130.414855893 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30f9c156-1292-4927-9159-eba19cf76012-metrics-certs") pod "network-metrics-daemon-nwlcx" (UID: "30f9c156-1292-4927-9159-eba19cf76012") : secret "metrics-daemon-secret" not found
May 25 12:14:13.544788 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:14:13.544762    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-45vpt\" (UniqueName: \"kubernetes.io/projected/2d835613-0b9d-4b52-bc2f-a40577b71208-kube-api-access-45vpt\") pod \"network-check-target-4wlq9\" (UID: \"2d835613-0b9d-4b52-bc2f-a40577b71208\") " pod="openshift-network-diagnostics/network-check-target-4wlq9"
May 25 12:14:13.546798 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:14:13.546782    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 25 12:14:13.556774 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:14:13.556756    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 25 12:14:13.568050 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:14:13.568032    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45vpt\" (UniqueName: \"kubernetes.io/projected/2d835613-0b9d-4b52-bc2f-a40577b71208-kube-api-access-45vpt\") pod \"network-check-target-4wlq9\" (UID: \"2d835613-0b9d-4b52-bc2f-a40577b71208\") " pod="openshift-network-diagnostics/network-check-target-4wlq9"
May 25 12:14:13.806063 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:14:13.806006    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wxdt4\""
May 25 12:14:13.814247 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:14:13.814230    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4wlq9"
May 25 12:14:13.983621 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:14:13.983591    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4wlq9"]
May 25 12:14:13.987509 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:14:13.987479    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d835613_0b9d_4b52_bc2f_a40577b71208.slice/crio-99a362c4bfff785c62ba1d75aea13d5f4d6d94f030c2e3cd5a8107378c978666 WatchSource:0}: Error finding container 99a362c4bfff785c62ba1d75aea13d5f4d6d94f030c2e3cd5a8107378c978666: Status 404 returned error can't find the container with id 99a362c4bfff785c62ba1d75aea13d5f4d6d94f030c2e3cd5a8107378c978666
May 25 12:14:14.817498 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:14:14.817461    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4wlq9" event={"ID":"2d835613-0b9d-4b52-bc2f-a40577b71208","Type":"ContainerStarted","Data":"99a362c4bfff785c62ba1d75aea13d5f4d6d94f030c2e3cd5a8107378c978666"}
May 25 12:14:17.823751 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:14:17.823715    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4wlq9" event={"ID":"2d835613-0b9d-4b52-bc2f-a40577b71208","Type":"ContainerStarted","Data":"89b8b6ae8cb982ea3d036049315246d0bfe75c8dbffbdf57998d92ea2636c45e"}
May 25 12:14:17.824142 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:14:17.823861    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4wlq9"
May 25 12:14:17.837557 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:14:17.837518    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-4wlq9" podStartSLOduration=68.001303219 podStartE2EDuration="1m10.837505784s" podCreationTimestamp="2026-05-25 12:13:07 +0000 UTC" firstStartedPulling="2026-05-25 12:14:13.989296964 +0000 UTC m=+66.949347351" lastFinishedPulling="2026-05-25 12:14:16.825499521 +0000 UTC m=+69.785549916" observedRunningTime="2026-05-25 12:14:17.836850592 +0000 UTC m=+70.796900994" watchObservedRunningTime="2026-05-25 12:14:17.837505784 +0000 UTC m=+70.797556493"
May 25 12:14:45.040419 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:14:45.040381    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5b139bd-e9a1-45b9-9d31-11a98965a2e0-metrics-tls\") pod \"dns-default-gn5dh\" (UID: \"a5b139bd-e9a1-45b9-9d31-11a98965a2e0\") " pod="openshift-dns/dns-default-gn5dh"
May 25 12:14:45.040419 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:14:45.040419    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26ec0201-e1d2-4816-944f-bbdb03159591-cert\") pod \"ingress-canary-8mblp\" (UID: \"26ec0201-e1d2-4816-944f-bbdb03159591\") " pod="openshift-ingress-canary/ingress-canary-8mblp"
May 25 12:14:45.040841 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:14:45.040494    2555 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 12:14:45.040841 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:14:45.040501    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 12:14:45.040841 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:14:45.040549    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26ec0201-e1d2-4816-944f-bbdb03159591-cert podName:26ec0201-e1d2-4816-944f-bbdb03159591 nodeName:}" failed. No retries permitted until 2026-05-25 12:15:49.040535972 +0000 UTC m=+162.000586356 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/26ec0201-e1d2-4816-944f-bbdb03159591-cert") pod "ingress-canary-8mblp" (UID: "26ec0201-e1d2-4816-944f-bbdb03159591") : secret "canary-serving-cert" not found
May 25 12:14:45.040841 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:14:45.040561    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5b139bd-e9a1-45b9-9d31-11a98965a2e0-metrics-tls podName:a5b139bd-e9a1-45b9-9d31-11a98965a2e0 nodeName:}" failed. No retries permitted until 2026-05-25 12:15:49.040555521 +0000 UTC m=+162.000605904 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a5b139bd-e9a1-45b9-9d31-11a98965a2e0-metrics-tls") pod "dns-default-gn5dh" (UID: "a5b139bd-e9a1-45b9-9d31-11a98965a2e0") : secret "dns-default-metrics-tls" not found
May 25 12:14:48.827861 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:14:48.827834    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-4wlq9"
May 25 12:15:08.750205 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.750172    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-cz2wm"]
May 25 12:15:08.751987 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.751968    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-cz2wm"
May 25 12:15:08.753761 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.753736    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
May 25 12:15:08.753903 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.753802    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
May 25 12:15:08.753903 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.753865    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
May 25 12:15:08.754298 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.754282    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
May 25 12:15:08.754351 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.754306    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-scljv\""
May 25 12:15:08.758374 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.758232    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-4jhg4"]
May 25 12:15:08.759949 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.759926    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-j92gb"]
May 25 12:15:08.760139 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.760119    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-4jhg4"
May 25 12:15:08.761916 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.761900    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-cz2wm"]
May 25 12:15:08.761990 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.761981    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-j92gb"
May 25 12:15:08.762436 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.762420    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
May 25 12:15:08.762502 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.762451    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-p6wgg\""
May 25 12:15:08.763234 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.763217    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
May 25 12:15:08.764315 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.764296    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
May 25 12:15:08.764611 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.764570    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
May 25 12:15:08.764735 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.764575    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
May 25 12:15:08.764904 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.764887    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-x856t\""
May 25 12:15:08.773718 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.773701    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-j92gb"]
May 25 12:15:08.780063 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.780044    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-4jhg4"]
May 25 12:15:08.790506 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.790488    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/01c35034-4da3-4712-b62f-77b9082e49fd-serving-cert\") pod \"kube-storage-version-migrator-operator-565644f49-cz2wm\" (UID: \"01c35034-4da3-4712-b62f-77b9082e49fd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-cz2wm"
May 25 12:15:08.790591 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.790521    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/01c35034-4da3-4712-b62f-77b9082e49fd-config\") pod \"kube-storage-version-migrator-operator-565644f49-cz2wm\" (UID: \"01c35034-4da3-4712-b62f-77b9082e49fd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-cz2wm"
May 25 12:15:08.790591 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.790542    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw7gh\" (UniqueName: \"kubernetes.io/projected/01c35034-4da3-4712-b62f-77b9082e49fd-kube-api-access-nw7gh\") pod \"kube-storage-version-migrator-operator-565644f49-cz2wm\" (UID: \"01c35034-4da3-4712-b62f-77b9082e49fd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-cz2wm"
May 25 12:15:08.870860 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.870827    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-78bdfb888c-sp5bh"]
May 25 12:15:08.872528 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.872514    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-78bdfb888c-sp5bh"
May 25 12:15:08.875509 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.875492    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57b5b68b8-sch65"]
May 25 12:15:08.877999 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.877975    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-sch65"
May 25 12:15:08.879812 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.879795    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
May 25 12:15:08.879901 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.879803    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
May 25 12:15:08.879901 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.879883    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-4fcnx\""
May 25 12:15:08.880753 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.880736    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
May 25 12:15:08.880753 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.880736    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
May 25 12:15:08.885833 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.885814    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
May 25 12:15:08.887599 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.887581    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7f44b6dd87-t97h9"]
May 25 12:15:08.889237 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.889222    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-t97h9"
May 25 12:15:08.890868 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.890851    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/01c35034-4da3-4712-b62f-77b9082e49fd-serving-cert\") pod \"kube-storage-version-migrator-operator-565644f49-cz2wm\" (UID: \"01c35034-4da3-4712-b62f-77b9082e49fd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-cz2wm"
May 25 12:15:08.890936 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.890883    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkm85\" (UniqueName: \"kubernetes.io/projected/c797c7c5-cc61-40ff-b27e-c28f72dcf80e-kube-api-access-nkm85\") pod \"cluster-samples-operator-697587599-j92gb\" (UID: \"c797c7c5-cc61-40ff-b27e-c28f72dcf80e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-j92gb"
May 25 12:15:08.890936 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.890902    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/01c35034-4da3-4712-b62f-77b9082e49fd-config\") pod \"kube-storage-version-migrator-operator-565644f49-cz2wm\" (UID: \"01c35034-4da3-4712-b62f-77b9082e49fd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-cz2wm"
May 25 12:15:08.890998 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.890949    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nw7gh\" (UniqueName: \"kubernetes.io/projected/01c35034-4da3-4712-b62f-77b9082e49fd-kube-api-access-nw7gh\") pod \"kube-storage-version-migrator-operator-565644f49-cz2wm\" (UID: \"01c35034-4da3-4712-b62f-77b9082e49fd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-cz2wm"
May 25 12:15:08.891042 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.890998    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjnq6\" (UniqueName: \"kubernetes.io/projected/ad5bd493-e25b-42d8-b008-df2b89dda3f9-kube-api-access-zjnq6\") pod \"volume-data-source-validator-5cf8c89d64-4jhg4\" (UID: \"ad5bd493-e25b-42d8-b008-df2b89dda3f9\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-4jhg4"
May 25 12:15:08.891042 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.891024    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c797c7c5-cc61-40ff-b27e-c28f72dcf80e-samples-operator-tls\") pod \"cluster-samples-operator-697587599-j92gb\" (UID: \"c797c7c5-cc61-40ff-b27e-c28f72dcf80e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-j92gb"
May 25 12:15:08.891391 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.891375    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/01c35034-4da3-4712-b62f-77b9082e49fd-config\") pod \"kube-storage-version-migrator-operator-565644f49-cz2wm\" (UID: \"01c35034-4da3-4712-b62f-77b9082e49fd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-cz2wm"
May 25 12:15:08.892914 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.892896    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/01c35034-4da3-4712-b62f-77b9082e49fd-serving-cert\") pod \"kube-storage-version-migrator-operator-565644f49-cz2wm\" (UID: \"01c35034-4da3-4712-b62f-77b9082e49fd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-cz2wm"
May 25 12:15:08.893518 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.893503    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
May 25 12:15:08.893717 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.893704    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
May 25 12:15:08.894670 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.894631    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-2tqtm"]
May 25 12:15:08.895305 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.895292    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
May 25 12:15:08.896774 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.896760    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-76dff48d59-2tqtm"
May 25 12:15:08.899459 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.899441    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-4wbdr\""
May 25 12:15:08.904011 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.903984    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
May 25 12:15:08.905246 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.905225    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
May 25 12:15:08.905951 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.905936    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
May 25 12:15:08.906719 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.906703    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-qnzns\""
May 25 12:15:08.908994 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.908979    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-g9sb4\""
May 25 12:15:08.909456 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.909442    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
May 25 12:15:08.918245 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.918227    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-78bdfb888c-sp5bh"]
May 25 12:15:08.918904 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.918889    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57b5b68b8-sch65"]
May 25 12:15:08.919504 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.919487    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7f44b6dd87-t97h9"]
May 25 12:15:08.946243 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.945930    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw7gh\" (UniqueName: \"kubernetes.io/projected/01c35034-4da3-4712-b62f-77b9082e49fd-kube-api-access-nw7gh\") pod \"kube-storage-version-migrator-operator-565644f49-cz2wm\" (UID: \"01c35034-4da3-4712-b62f-77b9082e49fd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-cz2wm"
May 25 12:15:08.947522 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.947500    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
May 25 12:15:08.948007 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.947986    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-2tqtm"]
May 25 12:15:08.948913 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.948898    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
May 25 12:15:08.991739 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.991712    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/94423b0c-4155-43d2-8037-2054e0f36255-config\") pod \"service-ca-operator-57b5b68b8-sch65\" (UID: \"94423b0c-4155-43d2-8037-2054e0f36255\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-sch65"
May 25 12:15:08.991865 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.991743    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b30b319-e998-4354-99b7-a46815d537cf-trusted-ca-bundle\") pod \"insights-operator-78bdfb888c-sp5bh\" (UID: \"3b30b319-e998-4354-99b7-a46815d537cf\") " pod="openshift-insights/insights-operator-78bdfb888c-sp5bh"
May 25 12:15:08.991865 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.991765    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3b30b319-e998-4354-99b7-a46815d537cf-snapshots\") pod \"insights-operator-78bdfb888c-sp5bh\" (UID: \"3b30b319-e998-4354-99b7-a46815d537cf\") " pod="openshift-insights/insights-operator-78bdfb888c-sp5bh"
May 25 12:15:08.991865 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.991792    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zjnq6\" (UniqueName: \"kubernetes.io/projected/ad5bd493-e25b-42d8-b008-df2b89dda3f9-kube-api-access-zjnq6\") pod \"volume-data-source-validator-5cf8c89d64-4jhg4\" (UID: \"ad5bd493-e25b-42d8-b008-df2b89dda3f9\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-4jhg4"
May 25 12:15:08.991865 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.991809    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c797c7c5-cc61-40ff-b27e-c28f72dcf80e-samples-operator-tls\") pod \"cluster-samples-operator-697587599-j92gb\" (UID: \"c797c7c5-cc61-40ff-b27e-c28f72dcf80e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-j92gb"
May 25 12:15:08.992058 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.991894    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxl5c\" (UniqueName: \"kubernetes.io/projected/94423b0c-4155-43d2-8037-2054e0f36255-kube-api-access-mxl5c\") pod \"service-ca-operator-57b5b68b8-sch65\" (UID: \"94423b0c-4155-43d2-8037-2054e0f36255\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-sch65"
May 25 12:15:08.992058 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:15:08.991937    2555 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 25 12:15:08.992058 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.991977    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b30b319-e998-4354-99b7-a46815d537cf-tmp\") pod \"insights-operator-78bdfb888c-sp5bh\" (UID: \"3b30b319-e998-4354-99b7-a46815d537cf\") " pod="openshift-insights/insights-operator-78bdfb888c-sp5bh"
May 25 12:15:08.992058 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:15:08.991990    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c797c7c5-cc61-40ff-b27e-c28f72dcf80e-samples-operator-tls podName:c797c7c5-cc61-40ff-b27e-c28f72dcf80e nodeName:}" failed. No retries permitted until 2026-05-25 12:15:09.491975753 +0000 UTC m=+122.452026137 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c797c7c5-cc61-40ff-b27e-c28f72dcf80e-samples-operator-tls") pod "cluster-samples-operator-697587599-j92gb" (UID: "c797c7c5-cc61-40ff-b27e-c28f72dcf80e") : secret "samples-operator-tls" not found
May 25 12:15:08.992058 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.992016    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6g5b\" (UniqueName: \"kubernetes.io/projected/3b30b319-e998-4354-99b7-a46815d537cf-kube-api-access-j6g5b\") pod \"insights-operator-78bdfb888c-sp5bh\" (UID: \"3b30b319-e998-4354-99b7-a46815d537cf\") " pod="openshift-insights/insights-operator-78bdfb888c-sp5bh"
May 25 12:15:08.992058 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.992035    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/00e6f79c-8f05-4330-aa16-1ada2ae9e9cc-config\") pod \"console-operator-76dff48d59-2tqtm\" (UID: \"00e6f79c-8f05-4330-aa16-1ada2ae9e9cc\") " pod="openshift-console-operator/console-operator-76dff48d59-2tqtm"
May 25 12:15:08.992298 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.992089    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/00e6f79c-8f05-4330-aa16-1ada2ae9e9cc-serving-cert\") pod \"console-operator-76dff48d59-2tqtm\" (UID: \"00e6f79c-8f05-4330-aa16-1ada2ae9e9cc\") " pod="openshift-console-operator/console-operator-76dff48d59-2tqtm"
May 25 12:15:08.992298 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.992127    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bctl\" (UniqueName: \"kubernetes.io/projected/0feec15a-bfbe-4544-b178-9889ca727a87-kube-api-access-7bctl\") pod \"network-check-source-7f44b6dd87-t97h9\" (UID: \"0feec15a-bfbe-4544-b178-9889ca727a87\") " pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-t97h9"
May 25 12:15:08.992298 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.992179    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b30b319-e998-4354-99b7-a46815d537cf-service-ca-bundle\") pod \"insights-operator-78bdfb888c-sp5bh\" (UID: \"3b30b319-e998-4354-99b7-a46815d537cf\") " pod="openshift-insights/insights-operator-78bdfb888c-sp5bh"
May 25 12:15:08.992298 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.992199    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3b30b319-e998-4354-99b7-a46815d537cf-serving-cert\") pod \"insights-operator-78bdfb888c-sp5bh\" (UID: \"3b30b319-e998-4354-99b7-a46815d537cf\") " pod="openshift-insights/insights-operator-78bdfb888c-sp5bh"
May 25 12:15:08.992298 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.992236    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nkm85\" (UniqueName: \"kubernetes.io/projected/c797c7c5-cc61-40ff-b27e-c28f72dcf80e-kube-api-access-nkm85\") pod \"cluster-samples-operator-697587599-j92gb\" (UID: \"c797c7c5-cc61-40ff-b27e-c28f72dcf80e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-j92gb"
May 25 12:15:08.992298 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.992252    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmtz8\" (UniqueName: \"kubernetes.io/projected/00e6f79c-8f05-4330-aa16-1ada2ae9e9cc-kube-api-access-cmtz8\") pod \"console-operator-76dff48d59-2tqtm\" (UID: \"00e6f79c-8f05-4330-aa16-1ada2ae9e9cc\") " pod="openshift-console-operator/console-operator-76dff48d59-2tqtm"
May 25 12:15:08.992298 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.992277    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/94423b0c-4155-43d2-8037-2054e0f36255-serving-cert\") pod \"service-ca-operator-57b5b68b8-sch65\" (UID: \"94423b0c-4155-43d2-8037-2054e0f36255\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-sch65"
May 25 12:15:08.992561 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.992356    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/00e6f79c-8f05-4330-aa16-1ada2ae9e9cc-trusted-ca\") pod \"console-operator-76dff48d59-2tqtm\" (UID: \"00e6f79c-8f05-4330-aa16-1ada2ae9e9cc\") " pod="openshift-console-operator/console-operator-76dff48d59-2tqtm"
May 25 12:15:08.999767 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.999738    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjnq6\" (UniqueName: \"kubernetes.io/projected/ad5bd493-e25b-42d8-b008-df2b89dda3f9-kube-api-access-zjnq6\") pod \"volume-data-source-validator-5cf8c89d64-4jhg4\" (UID: \"ad5bd493-e25b-42d8-b008-df2b89dda3f9\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-4jhg4"
May 25 12:15:08.999867 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:08.999738    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkm85\" (UniqueName: \"kubernetes.io/projected/c797c7c5-cc61-40ff-b27e-c28f72dcf80e-kube-api-access-nkm85\") pod \"cluster-samples-operator-697587599-j92gb\" (UID: \"c797c7c5-cc61-40ff-b27e-c28f72dcf80e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-j92gb"
May 25 12:15:09.060565 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:09.060512    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-cz2wm"
May 25 12:15:09.069732 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:09.069710    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-4jhg4"
May 25 12:15:09.093919 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:09.093880    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b30b319-e998-4354-99b7-a46815d537cf-service-ca-bundle\") pod \"insights-operator-78bdfb888c-sp5bh\" (UID: \"3b30b319-e998-4354-99b7-a46815d537cf\") " pod="openshift-insights/insights-operator-78bdfb888c-sp5bh"
May 25 12:15:09.094058 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:09.093927    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3b30b319-e998-4354-99b7-a46815d537cf-serving-cert\") pod \"insights-operator-78bdfb888c-sp5bh\" (UID: \"3b30b319-e998-4354-99b7-a46815d537cf\") " pod="openshift-insights/insights-operator-78bdfb888c-sp5bh"
May 25 12:15:09.094058 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:09.093972    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cmtz8\" (UniqueName: \"kubernetes.io/projected/00e6f79c-8f05-4330-aa16-1ada2ae9e9cc-kube-api-access-cmtz8\") pod \"console-operator-76dff48d59-2tqtm\" (UID: \"00e6f79c-8f05-4330-aa16-1ada2ae9e9cc\") " pod="openshift-console-operator/console-operator-76dff48d59-2tqtm"
May 25 12:15:09.094058 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:09.094001    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/94423b0c-4155-43d2-8037-2054e0f36255-serving-cert\") pod \"service-ca-operator-57b5b68b8-sch65\" (UID: \"94423b0c-4155-43d2-8037-2054e0f36255\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-sch65"
May 25 12:15:09.094058 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:09.094025    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/00e6f79c-8f05-4330-aa16-1ada2ae9e9cc-trusted-ca\") pod \"console-operator-76dff48d59-2tqtm\" (UID: \"00e6f79c-8f05-4330-aa16-1ada2ae9e9cc\") " pod="openshift-console-operator/console-operator-76dff48d59-2tqtm"
May 25 12:15:09.094261 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:09.094152    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/94423b0c-4155-43d2-8037-2054e0f36255-config\") pod \"service-ca-operator-57b5b68b8-sch65\" (UID: \"94423b0c-4155-43d2-8037-2054e0f36255\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-sch65"
May 25 12:15:09.094261 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:09.094193    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b30b319-e998-4354-99b7-a46815d537cf-trusted-ca-bundle\") pod \"insights-operator-78bdfb888c-sp5bh\" (UID: \"3b30b319-e998-4354-99b7-a46815d537cf\") " pod="openshift-insights/insights-operator-78bdfb888c-sp5bh"
May 25 12:15:09.094261 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:09.094230    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3b30b319-e998-4354-99b7-a46815d537cf-snapshots\") pod \"insights-operator-78bdfb888c-sp5bh\" (UID: \"3b30b319-e998-4354-99b7-a46815d537cf\") " pod="openshift-insights/insights-operator-78bdfb888c-sp5bh"
May 25 12:15:09.094412 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:09.094268    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mxl5c\" (UniqueName: \"kubernetes.io/projected/94423b0c-4155-43d2-8037-2054e0f36255-kube-api-access-mxl5c\") pod \"service-ca-operator-57b5b68b8-sch65\" (UID: \"94423b0c-4155-43d2-8037-2054e0f36255\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-sch65"
May 25 12:15:09.094412 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:09.094294    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b30b319-e998-4354-99b7-a46815d537cf-tmp\") pod \"insights-operator-78bdfb888c-sp5bh\" (UID: \"3b30b319-e998-4354-99b7-a46815d537cf\") " pod="openshift-insights/insights-operator-78bdfb888c-sp5bh"
May 25 12:15:09.094412 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:09.094321    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-j6g5b\" (UniqueName: \"kubernetes.io/projected/3b30b319-e998-4354-99b7-a46815d537cf-kube-api-access-j6g5b\") pod \"insights-operator-78bdfb888c-sp5bh\" (UID: \"3b30b319-e998-4354-99b7-a46815d537cf\") " pod="openshift-insights/insights-operator-78bdfb888c-sp5bh"
May 25 12:15:09.094412 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:09.094346    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/00e6f79c-8f05-4330-aa16-1ada2ae9e9cc-config\") pod \"console-operator-76dff48d59-2tqtm\" (UID: \"00e6f79c-8f05-4330-aa16-1ada2ae9e9cc\") " pod="openshift-console-operator/console-operator-76dff48d59-2tqtm"
May 25 12:15:09.094412 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:09.094376    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/00e6f79c-8f05-4330-aa16-1ada2ae9e9cc-serving-cert\") pod \"console-operator-76dff48d59-2tqtm\" (UID: \"00e6f79c-8f05-4330-aa16-1ada2ae9e9cc\") " pod="openshift-console-operator/console-operator-76dff48d59-2tqtm"
May 25 12:15:09.094412 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:09.094403    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7bctl\" (UniqueName: \"kubernetes.io/projected/0feec15a-bfbe-4544-b178-9889ca727a87-kube-api-access-7bctl\") pod \"network-check-source-7f44b6dd87-t97h9\" (UID: \"0feec15a-bfbe-4544-b178-9889ca727a87\") " pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-t97h9"
May 25 12:15:09.094722 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:09.094610    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b30b319-e998-4354-99b7-a46815d537cf-service-ca-bundle\") pod \"insights-operator-78bdfb888c-sp5bh\" (UID: \"3b30b319-e998-4354-99b7-a46815d537cf\") " pod="openshift-insights/insights-operator-78bdfb888c-sp5bh"
May 25 12:15:09.094796 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:09.094773    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b30b319-e998-4354-99b7-a46815d537cf-tmp\") pod \"insights-operator-78bdfb888c-sp5bh\" (UID: \"3b30b319-e998-4354-99b7-a46815d537cf\") " pod="openshift-insights/insights-operator-78bdfb888c-sp5bh"
May 25 12:15:09.094887 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:09.094775    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/94423b0c-4155-43d2-8037-2054e0f36255-config\") pod \"service-ca-operator-57b5b68b8-sch65\" (UID: \"94423b0c-4155-43d2-8037-2054e0f36255\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-sch65"
May 25 12:15:09.095146 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:09.095117    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/00e6f79c-8f05-4330-aa16-1ada2ae9e9cc-config\") pod \"console-operator-76dff48d59-2tqtm\" (UID: \"00e6f79c-8f05-4330-aa16-1ada2ae9e9cc\") " pod="openshift-console-operator/console-operator-76dff48d59-2tqtm"
May 25 12:15:09.095359 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:09.095336    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3b30b319-e998-4354-99b7-a46815d537cf-snapshots\") pod \"insights-operator-78bdfb888c-sp5bh\" (UID: \"3b30b319-e998-4354-99b7-a46815d537cf\") " pod="openshift-insights/insights-operator-78bdfb888c-sp5bh"
May 25 12:15:09.096007 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:09.095890    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b30b319-e998-4354-99b7-a46815d537cf-trusted-ca-bundle\") pod \"insights-operator-78bdfb888c-sp5bh\" (UID: \"3b30b319-e998-4354-99b7-a46815d537cf\") " pod="openshift-insights/insights-operator-78bdfb888c-sp5bh"
May 25 12:15:09.096378 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:09.096358    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/00e6f79c-8f05-4330-aa16-1ada2ae9e9cc-trusted-ca\") pod \"console-operator-76dff48d59-2tqtm\" (UID: \"00e6f79c-8f05-4330-aa16-1ada2ae9e9cc\") " pod="openshift-console-operator/console-operator-76dff48d59-2tqtm"
May 25 12:15:09.097232 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:09.097207    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/94423b0c-4155-43d2-8037-2054e0f36255-serving-cert\") pod \"service-ca-operator-57b5b68b8-sch65\" (UID: \"94423b0c-4155-43d2-8037-2054e0f36255\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-sch65"
May 25 12:15:09.097383 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:09.097363    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3b30b319-e998-4354-99b7-a46815d537cf-serving-cert\") pod \"insights-operator-78bdfb888c-sp5bh\" (UID: \"3b30b319-e998-4354-99b7-a46815d537cf\") " pod="openshift-insights/insights-operator-78bdfb888c-sp5bh"
May 25 12:15:09.097468 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:09.097449    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/00e6f79c-8f05-4330-aa16-1ada2ae9e9cc-serving-cert\") pod \"console-operator-76dff48d59-2tqtm\" (UID: \"00e6f79c-8f05-4330-aa16-1ada2ae9e9cc\") " pod="openshift-console-operator/console-operator-76dff48d59-2tqtm"
May 25 12:15:09.105157 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:09.105113    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6g5b\" (UniqueName: \"kubernetes.io/projected/3b30b319-e998-4354-99b7-a46815d537cf-kube-api-access-j6g5b\") pod \"insights-operator-78bdfb888c-sp5bh\" (UID: \"3b30b319-e998-4354-99b7-a46815d537cf\") " pod="openshift-insights/insights-operator-78bdfb888c-sp5bh"
May 25 12:15:09.105157 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:09.105143    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxl5c\" (UniqueName: \"kubernetes.io/projected/94423b0c-4155-43d2-8037-2054e0f36255-kube-api-access-mxl5c\") pod \"service-ca-operator-57b5b68b8-sch65\" (UID: \"94423b0c-4155-43d2-8037-2054e0f36255\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-sch65"
May 25 12:15:09.107827 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:09.107786    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmtz8\" (UniqueName: \"kubernetes.io/projected/00e6f79c-8f05-4330-aa16-1ada2ae9e9cc-kube-api-access-cmtz8\") pod \"console-operator-76dff48d59-2tqtm\" (UID: \"00e6f79c-8f05-4330-aa16-1ada2ae9e9cc\") " pod="openshift-console-operator/console-operator-76dff48d59-2tqtm"
May 25 12:15:09.109670 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:09.109231    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bctl\" (UniqueName: \"kubernetes.io/projected/0feec15a-bfbe-4544-b178-9889ca727a87-kube-api-access-7bctl\") pod \"network-check-source-7f44b6dd87-t97h9\" (UID: \"0feec15a-bfbe-4544-b178-9889ca727a87\") " pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-t97h9"
May 25 12:15:09.178243 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:09.178209    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-cz2wm"]
May 25 12:15:09.181492 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:15:09.181454    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01c35034_4da3_4712_b62f_77b9082e49fd.slice/crio-5d5609c343f81a6a603d6de2dd1909e8d75ccc50e05f4a19e9f32a83b52d6f39 WatchSource:0}: Error finding container 5d5609c343f81a6a603d6de2dd1909e8d75ccc50e05f4a19e9f32a83b52d6f39: Status 404 returned error can't find the container with id 5d5609c343f81a6a603d6de2dd1909e8d75ccc50e05f4a19e9f32a83b52d6f39
May 25 12:15:09.183160 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:09.183143    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-78bdfb888c-sp5bh"
May 25 12:15:09.188676 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:09.188656    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-sch65"
May 25 12:15:09.193547 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:09.193260    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-4jhg4"]
May 25 12:15:09.197816 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:15:09.197791    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad5bd493_e25b_42d8_b008_df2b89dda3f9.slice/crio-35097e5cbf5d231611160b088f03e38bf1583c7ae014bc6bd105283a2e3e36c5 WatchSource:0}: Error finding container 35097e5cbf5d231611160b088f03e38bf1583c7ae014bc6bd105283a2e3e36c5: Status 404 returned error can't find the container with id 35097e5cbf5d231611160b088f03e38bf1583c7ae014bc6bd105283a2e3e36c5
May 25 12:15:09.202665 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:09.202630    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-t97h9"
May 25 12:15:09.207966 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:09.207661    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-76dff48d59-2tqtm"
May 25 12:15:09.322814 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:09.322720    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-78bdfb888c-sp5bh"]
May 25 12:15:09.327309 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:15:09.327276    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b30b319_e998_4354_99b7_a46815d537cf.slice/crio-f271c0bb17263e3abd1f948dd377dcfc3b27d995ace3ec9bab7388efa3eee4fa WatchSource:0}: Error finding container f271c0bb17263e3abd1f948dd377dcfc3b27d995ace3ec9bab7388efa3eee4fa: Status 404 returned error can't find the container with id f271c0bb17263e3abd1f948dd377dcfc3b27d995ace3ec9bab7388efa3eee4fa
May 25 12:15:09.339236 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:09.339210    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57b5b68b8-sch65"]
May 25 12:15:09.342505 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:15:09.342484    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94423b0c_4155_43d2_8037_2054e0f36255.slice/crio-e58edff217773c60c77476151f5de33cd8a0688d6a16d6738b788fe3987dfd76 WatchSource:0}: Error finding container e58edff217773c60c77476151f5de33cd8a0688d6a16d6738b788fe3987dfd76: Status 404 returned error can't find the container with id e58edff217773c60c77476151f5de33cd8a0688d6a16d6738b788fe3987dfd76
May 25 12:15:09.497637 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:09.497602    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c797c7c5-cc61-40ff-b27e-c28f72dcf80e-samples-operator-tls\") pod \"cluster-samples-operator-697587599-j92gb\" (UID: \"c797c7c5-cc61-40ff-b27e-c28f72dcf80e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-j92gb"
May 25 12:15:09.497784 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:15:09.497765    2555 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 25 12:15:09.497848 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:15:09.497838    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c797c7c5-cc61-40ff-b27e-c28f72dcf80e-samples-operator-tls podName:c797c7c5-cc61-40ff-b27e-c28f72dcf80e nodeName:}" failed. No retries permitted until 2026-05-25 12:15:10.497821201 +0000 UTC m=+123.457871608 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c797c7c5-cc61-40ff-b27e-c28f72dcf80e-samples-operator-tls") pod "cluster-samples-operator-697587599-j92gb" (UID: "c797c7c5-cc61-40ff-b27e-c28f72dcf80e") : secret "samples-operator-tls" not found
May 25 12:15:09.558342 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:09.558310    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7f44b6dd87-t97h9"]
May 25 12:15:09.561003 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:09.560980    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-2tqtm"]
May 25 12:15:09.561683 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:15:09.561629    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0feec15a_bfbe_4544_b178_9889ca727a87.slice/crio-554032457eaa2ca72174d4a357269b711cbb586ce18f5bed937f72457b0536b2 WatchSource:0}: Error finding container 554032457eaa2ca72174d4a357269b711cbb586ce18f5bed937f72457b0536b2: Status 404 returned error can't find the container with id 554032457eaa2ca72174d4a357269b711cbb586ce18f5bed937f72457b0536b2
May 25 12:15:09.567286 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:15:09.567263    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00e6f79c_8f05_4330_aa16_1ada2ae9e9cc.slice/crio-695c3335f48f0acbb0e421cdadd9b2382d850b4b1837c3c4f7de2b3ab0242ea5 WatchSource:0}: Error finding container 695c3335f48f0acbb0e421cdadd9b2382d850b4b1837c3c4f7de2b3ab0242ea5: Status 404 returned error can't find the container with id 695c3335f48f0acbb0e421cdadd9b2382d850b4b1837c3c4f7de2b3ab0242ea5
May 25 12:15:09.915226 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:09.915126    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-2tqtm" event={"ID":"00e6f79c-8f05-4330-aa16-1ada2ae9e9cc","Type":"ContainerStarted","Data":"695c3335f48f0acbb0e421cdadd9b2382d850b4b1837c3c4f7de2b3ab0242ea5"}
May 25 12:15:09.917724 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:09.917673    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-4jhg4" event={"ID":"ad5bd493-e25b-42d8-b008-df2b89dda3f9","Type":"ContainerStarted","Data":"35097e5cbf5d231611160b088f03e38bf1583c7ae014bc6bd105283a2e3e36c5"}
May 25 12:15:09.919358 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:09.919318    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-cz2wm" event={"ID":"01c35034-4da3-4712-b62f-77b9082e49fd","Type":"ContainerStarted","Data":"5d5609c343f81a6a603d6de2dd1909e8d75ccc50e05f4a19e9f32a83b52d6f39"}
May 25 12:15:09.920629 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:09.920587    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-sch65" event={"ID":"94423b0c-4155-43d2-8037-2054e0f36255","Type":"ContainerStarted","Data":"e58edff217773c60c77476151f5de33cd8a0688d6a16d6738b788fe3987dfd76"}
May 25 12:15:09.921586 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:09.921523    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-sp5bh" event={"ID":"3b30b319-e998-4354-99b7-a46815d537cf","Type":"ContainerStarted","Data":"f271c0bb17263e3abd1f948dd377dcfc3b27d995ace3ec9bab7388efa3eee4fa"}
May 25 12:15:09.923254 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:09.923232    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-t97h9" event={"ID":"0feec15a-bfbe-4544-b178-9889ca727a87","Type":"ContainerStarted","Data":"56d7c91c7a17917e35589f0283555632d703fbb54b44ade4e8076449bfc1a411"}
May 25 12:15:09.923345 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:09.923261    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-t97h9" event={"ID":"0feec15a-bfbe-4544-b178-9889ca727a87","Type":"ContainerStarted","Data":"554032457eaa2ca72174d4a357269b711cbb586ce18f5bed937f72457b0536b2"}
May 25 12:15:09.937959 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:09.937238    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-t97h9" podStartSLOduration=1.9372236919999999 podStartE2EDuration="1.937223692s" podCreationTimestamp="2026-05-25 12:15:08 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:15:09.936501301 +0000 UTC m=+122.896551697" watchObservedRunningTime="2026-05-25 12:15:09.937223692 +0000 UTC m=+122.897274097"
May 25 12:15:10.507492 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:10.507443    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c797c7c5-cc61-40ff-b27e-c28f72dcf80e-samples-operator-tls\") pod \"cluster-samples-operator-697587599-j92gb\" (UID: \"c797c7c5-cc61-40ff-b27e-c28f72dcf80e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-j92gb"
May 25 12:15:10.507703 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:15:10.507687    2555 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 25 12:15:10.507839 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:15:10.507774    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c797c7c5-cc61-40ff-b27e-c28f72dcf80e-samples-operator-tls podName:c797c7c5-cc61-40ff-b27e-c28f72dcf80e nodeName:}" failed. No retries permitted until 2026-05-25 12:15:12.507754807 +0000 UTC m=+125.467805192 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c797c7c5-cc61-40ff-b27e-c28f72dcf80e-samples-operator-tls") pod "cluster-samples-operator-697587599-j92gb" (UID: "c797c7c5-cc61-40ff-b27e-c28f72dcf80e") : secret "samples-operator-tls" not found
May 25 12:15:10.927962 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:10.927926    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-4jhg4" event={"ID":"ad5bd493-e25b-42d8-b008-df2b89dda3f9","Type":"ContainerStarted","Data":"f352409365590f7b1d6bf673f58c496e10b3e8c6fb6d3367a7f41835a601f6eb"}
May 25 12:15:10.941379 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:10.941237    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-4jhg4" podStartSLOduration=1.399519794 podStartE2EDuration="2.941219987s" podCreationTimestamp="2026-05-25 12:15:08 +0000 UTC" firstStartedPulling="2026-05-25 12:15:09.199960434 +0000 UTC m=+122.160010818" lastFinishedPulling="2026-05-25 12:15:10.741660622 +0000 UTC m=+123.701711011" observedRunningTime="2026-05-25 12:15:10.940968347 +0000 UTC m=+123.901018773" watchObservedRunningTime="2026-05-25 12:15:10.941219987 +0000 UTC m=+123.901270394"
May 25 12:15:12.521986 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:12.521946    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c797c7c5-cc61-40ff-b27e-c28f72dcf80e-samples-operator-tls\") pod \"cluster-samples-operator-697587599-j92gb\" (UID: \"c797c7c5-cc61-40ff-b27e-c28f72dcf80e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-j92gb"
May 25 12:15:12.522373 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:15:12.522119    2555 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 25 12:15:12.522373 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:15:12.522186    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c797c7c5-cc61-40ff-b27e-c28f72dcf80e-samples-operator-tls podName:c797c7c5-cc61-40ff-b27e-c28f72dcf80e nodeName:}" failed. No retries permitted until 2026-05-25 12:15:16.522171077 +0000 UTC m=+129.482221460 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c797c7c5-cc61-40ff-b27e-c28f72dcf80e-samples-operator-tls") pod "cluster-samples-operator-697587599-j92gb" (UID: "c797c7c5-cc61-40ff-b27e-c28f72dcf80e") : secret "samples-operator-tls" not found
May 25 12:15:13.938581 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:13.938544    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-cz2wm" event={"ID":"01c35034-4da3-4712-b62f-77b9082e49fd","Type":"ContainerStarted","Data":"f42b8f4a91787448e3cdb13a75ba7dcf5a1f173ee0c5a09c6461f1be03c963b6"}
May 25 12:15:13.939968 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:13.939944    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-sch65" event={"ID":"94423b0c-4155-43d2-8037-2054e0f36255","Type":"ContainerStarted","Data":"17ede3dccb11988def47ffe015750142b59a03893cb79a90f8d356ed5ceef0bf"}
May 25 12:15:13.941310 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:13.941289    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-sp5bh" event={"ID":"3b30b319-e998-4354-99b7-a46815d537cf","Type":"ContainerStarted","Data":"aba3f4dffe8aec2704832b310eb571be150a50e5364bd4e835f79aa0375303c2"}
May 25 12:15:13.942840 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:13.942822    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-2tqtm_00e6f79c-8f05-4330-aa16-1ada2ae9e9cc/console-operator/0.log"
May 25 12:15:13.942944 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:13.942853    2555 generic.go:358] "Generic (PLEG): container finished" podID="00e6f79c-8f05-4330-aa16-1ada2ae9e9cc" containerID="e548614f97491cd45fe5599efacb38e5b317a42b3d698b696bbc19441e4582b2" exitCode=255
May 25 12:15:13.942944 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:13.942876    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-2tqtm" event={"ID":"00e6f79c-8f05-4330-aa16-1ada2ae9e9cc","Type":"ContainerDied","Data":"e548614f97491cd45fe5599efacb38e5b317a42b3d698b696bbc19441e4582b2"}
May 25 12:15:13.943130 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:13.943114    2555 scope.go:117] "RemoveContainer" containerID="e548614f97491cd45fe5599efacb38e5b317a42b3d698b696bbc19441e4582b2"
May 25 12:15:13.953159 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:13.953111    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-cz2wm" podStartSLOduration=2.186931412 podStartE2EDuration="5.953098203s" podCreationTimestamp="2026-05-25 12:15:08 +0000 UTC" firstStartedPulling="2026-05-25 12:15:09.18330852 +0000 UTC m=+122.143358906" lastFinishedPulling="2026-05-25 12:15:12.949475301 +0000 UTC m=+125.909525697" observedRunningTime="2026-05-25 12:15:13.952661976 +0000 UTC m=+126.912712386" watchObservedRunningTime="2026-05-25 12:15:13.953098203 +0000 UTC m=+126.913148608"
May 25 12:15:13.966674 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:13.965943    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-sch65" podStartSLOduration=2.3573580290000002 podStartE2EDuration="5.965928678s" podCreationTimestamp="2026-05-25 12:15:08 +0000 UTC" firstStartedPulling="2026-05-25 12:15:09.344165616 +0000 UTC m=+122.304216001" lastFinishedPulling="2026-05-25 12:15:12.952736251 +0000 UTC m=+125.912786650" observedRunningTime="2026-05-25 12:15:13.965343806 +0000 UTC m=+126.925394211" watchObservedRunningTime="2026-05-25 12:15:13.965928678 +0000 UTC m=+126.925979085"
May 25 12:15:13.981550 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:13.981506    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-operator-78bdfb888c-sp5bh" podStartSLOduration=2.356228976 podStartE2EDuration="5.981490741s" podCreationTimestamp="2026-05-25 12:15:08 +0000 UTC" firstStartedPulling="2026-05-25 12:15:09.329708625 +0000 UTC m=+122.289759023" lastFinishedPulling="2026-05-25 12:15:12.954970403 +0000 UTC m=+125.915020788" observedRunningTime="2026-05-25 12:15:13.981407509 +0000 UTC m=+126.941457911" watchObservedRunningTime="2026-05-25 12:15:13.981490741 +0000 UTC m=+126.941541146"
May 25 12:15:14.946375 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:14.946343    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-2tqtm_00e6f79c-8f05-4330-aa16-1ada2ae9e9cc/console-operator/1.log"
May 25 12:15:14.946874 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:14.946694    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-2tqtm_00e6f79c-8f05-4330-aa16-1ada2ae9e9cc/console-operator/0.log"
May 25 12:15:14.946874 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:14.946724    2555 generic.go:358] "Generic (PLEG): container finished" podID="00e6f79c-8f05-4330-aa16-1ada2ae9e9cc" containerID="fa42612a1438e892b2c215633709de7feed2ffc566cc8f2a7ad51bbce6d609db" exitCode=255
May 25 12:15:14.946874 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:14.946769    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-2tqtm" event={"ID":"00e6f79c-8f05-4330-aa16-1ada2ae9e9cc","Type":"ContainerDied","Data":"fa42612a1438e892b2c215633709de7feed2ffc566cc8f2a7ad51bbce6d609db"}
May 25 12:15:14.946874 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:14.946794    2555 scope.go:117] "RemoveContainer" containerID="e548614f97491cd45fe5599efacb38e5b317a42b3d698b696bbc19441e4582b2"
May 25 12:15:14.947480 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:14.947459    2555 scope.go:117] "RemoveContainer" containerID="fa42612a1438e892b2c215633709de7feed2ffc566cc8f2a7ad51bbce6d609db"
May 25 12:15:14.947676 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:15:14.947654    2555 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-2tqtm_openshift-console-operator(00e6f79c-8f05-4330-aa16-1ada2ae9e9cc)\"" pod="openshift-console-operator/console-operator-76dff48d59-2tqtm" podUID="00e6f79c-8f05-4330-aa16-1ada2ae9e9cc"
May 25 12:15:15.950204 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:15.950175    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-2tqtm_00e6f79c-8f05-4330-aa16-1ada2ae9e9cc/console-operator/1.log"
May 25 12:15:15.950569 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:15.950460    2555 scope.go:117] "RemoveContainer" containerID="fa42612a1438e892b2c215633709de7feed2ffc566cc8f2a7ad51bbce6d609db"
May 25 12:15:15.950612 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:15:15.950598    2555 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-2tqtm_openshift-console-operator(00e6f79c-8f05-4330-aa16-1ada2ae9e9cc)\"" pod="openshift-console-operator/console-operator-76dff48d59-2tqtm" podUID="00e6f79c-8f05-4330-aa16-1ada2ae9e9cc"
May 25 12:15:16.550849 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:16.550809    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c797c7c5-cc61-40ff-b27e-c28f72dcf80e-samples-operator-tls\") pod \"cluster-samples-operator-697587599-j92gb\" (UID: \"c797c7c5-cc61-40ff-b27e-c28f72dcf80e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-j92gb"
May 25 12:15:16.551031 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:15:16.550955    2555 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 25 12:15:16.551031 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:15:16.551019    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c797c7c5-cc61-40ff-b27e-c28f72dcf80e-samples-operator-tls podName:c797c7c5-cc61-40ff-b27e-c28f72dcf80e nodeName:}" failed. No retries permitted until 2026-05-25 12:15:24.551004735 +0000 UTC m=+137.511055118 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c797c7c5-cc61-40ff-b27e-c28f72dcf80e-samples-operator-tls") pod "cluster-samples-operator-697587599-j92gb" (UID: "c797c7c5-cc61-40ff-b27e-c28f72dcf80e") : secret "samples-operator-tls" not found
May 25 12:15:17.400611 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:17.400582    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dtbvj_ab76ee4f-2142-41d5-99e2-0de355f6564d/dns-node-resolver/0.log"
May 25 12:15:17.458305 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:17.458280    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30f9c156-1292-4927-9159-eba19cf76012-metrics-certs\") pod \"network-metrics-daemon-nwlcx\" (UID: \"30f9c156-1292-4927-9159-eba19cf76012\") " pod="openshift-multus/network-metrics-daemon-nwlcx"
May 25 12:15:17.458424 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:15:17.458390    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 25 12:15:17.458462 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:15:17.458438    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30f9c156-1292-4927-9159-eba19cf76012-metrics-certs podName:30f9c156-1292-4927-9159-eba19cf76012 nodeName:}" failed. No retries permitted until 2026-05-25 12:17:19.458425308 +0000 UTC m=+252.418475691 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30f9c156-1292-4927-9159-eba19cf76012-metrics-certs") pod "network-metrics-daemon-nwlcx" (UID: "30f9c156-1292-4927-9159-eba19cf76012") : secret "metrics-daemon-secret" not found
May 25 12:15:18.201392 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:18.201361    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-brxmt_b77364d1-80eb-4af1-9764-ca0f0858270a/node-ca/0.log"
May 25 12:15:19.208309 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:19.208279    2555 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-76dff48d59-2tqtm"
May 25 12:15:19.208674 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:19.208322    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-76dff48d59-2tqtm"
May 25 12:15:19.208674 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:19.208624    2555 scope.go:117] "RemoveContainer" containerID="fa42612a1438e892b2c215633709de7feed2ffc566cc8f2a7ad51bbce6d609db"
May 25 12:15:19.208809 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:15:19.208792    2555 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-2tqtm_openshift-console-operator(00e6f79c-8f05-4330-aa16-1ada2ae9e9cc)\"" pod="openshift-console-operator/console-operator-76dff48d59-2tqtm" podUID="00e6f79c-8f05-4330-aa16-1ada2ae9e9cc"
May 25 12:15:19.958546 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:19.958521    2555 scope.go:117] "RemoveContainer" containerID="fa42612a1438e892b2c215633709de7feed2ffc566cc8f2a7ad51bbce6d609db"
May 25 12:15:19.958712 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:15:19.958696    2555 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-2tqtm_openshift-console-operator(00e6f79c-8f05-4330-aa16-1ada2ae9e9cc)\"" pod="openshift-console-operator/console-operator-76dff48d59-2tqtm" podUID="00e6f79c-8f05-4330-aa16-1ada2ae9e9cc"
May 25 12:15:20.002362 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:20.002340    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-565644f49-cz2wm_01c35034-4da3-4712-b62f-77b9082e49fd/kube-storage-version-migrator-operator/0.log"
May 25 12:15:24.609937 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:24.609898    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c797c7c5-cc61-40ff-b27e-c28f72dcf80e-samples-operator-tls\") pod \"cluster-samples-operator-697587599-j92gb\" (UID: \"c797c7c5-cc61-40ff-b27e-c28f72dcf80e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-j92gb"
May 25 12:15:24.612151 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:24.612124    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c797c7c5-cc61-40ff-b27e-c28f72dcf80e-samples-operator-tls\") pod \"cluster-samples-operator-697587599-j92gb\" (UID: \"c797c7c5-cc61-40ff-b27e-c28f72dcf80e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-j92gb"
May 25 12:15:24.673603 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:24.673575    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-j92gb"
May 25 12:15:24.786082 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:24.786057    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-j92gb"]
May 25 12:15:24.975129 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:24.975104    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-j92gb" event={"ID":"c797c7c5-cc61-40ff-b27e-c28f72dcf80e","Type":"ContainerStarted","Data":"a1094b3a8c54be3314222ad7952ce171a15becce6ef4fb0fce7fb348b5350fb0"}
May 25 12:15:26.981507 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:26.981473    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-j92gb" event={"ID":"c797c7c5-cc61-40ff-b27e-c28f72dcf80e","Type":"ContainerStarted","Data":"6f0ab9c6fe7cc79831ac06c7c739dfa115a413d7418afc5d51db968eb7770056"}
May 25 12:15:26.981507 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:26.981511    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-j92gb" event={"ID":"c797c7c5-cc61-40ff-b27e-c28f72dcf80e","Type":"ContainerStarted","Data":"c3c10ad1cd33ff7683064d5beca238495a905db0030d2c71c3f69c2d22069fc0"}
May 25 12:15:26.996064 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:26.996018    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-j92gb" podStartSLOduration=16.984066483 podStartE2EDuration="18.996005011s" podCreationTimestamp="2026-05-25 12:15:08 +0000 UTC" firstStartedPulling="2026-05-25 12:15:24.827273111 +0000 UTC m=+137.787323495" lastFinishedPulling="2026-05-25 12:15:26.839211636 +0000 UTC m=+139.799262023" observedRunningTime="2026-05-25 12:15:26.995065299 +0000 UTC m=+139.955115705" watchObservedRunningTime="2026-05-25 12:15:26.996005011 +0000 UTC m=+139.956055415"
May 25 12:15:33.594300 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:33.594262    2555 scope.go:117] "RemoveContainer" containerID="fa42612a1438e892b2c215633709de7feed2ffc566cc8f2a7ad51bbce6d609db"
May 25 12:15:33.998681 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:33.998658    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-2tqtm_00e6f79c-8f05-4330-aa16-1ada2ae9e9cc/console-operator/1.log"
May 25 12:15:33.998837 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:33.998738    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-2tqtm" event={"ID":"00e6f79c-8f05-4330-aa16-1ada2ae9e9cc","Type":"ContainerStarted","Data":"70339f10738b479fd65d709f6e9e85f61d527476b78d8686c28a1877386f9211"}
May 25 12:15:33.999000 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:33.998972    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-76dff48d59-2tqtm"
May 25 12:15:34.013842 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:34.013802    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console-operator/console-operator-76dff48d59-2tqtm" podStartSLOduration=22.630247703 podStartE2EDuration="26.013791679s" podCreationTimestamp="2026-05-25 12:15:08 +0000 UTC" firstStartedPulling="2026-05-25 12:15:09.56903508 +0000 UTC m=+122.529085464" lastFinishedPulling="2026-05-25 12:15:12.952579057 +0000 UTC m=+125.912629440" observedRunningTime="2026-05-25 12:15:34.01355661 +0000 UTC m=+146.973607022" watchObservedRunningTime="2026-05-25 12:15:34.013791679 +0000 UTC m=+146.973842083"
May 25 12:15:34.473259 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:34.473231    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-76dff48d59-2tqtm"
May 25 12:15:36.783972 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:36.783941    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-bmlx5"]
May 25 12:15:36.786753 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:36.786731    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bmlx5"
May 25 12:15:36.789108 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:36.789085    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
May 25 12:15:36.789223 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:36.789204    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
May 25 12:15:36.789698 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:36.789683    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-m27gk\""
May 25 12:15:36.804693 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:36.804667    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bmlx5"]
May 25 12:15:36.809239 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:36.809221    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-8c795b88c-jzr6x"]
May 25 12:15:36.811111 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:36.811098    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8c795b88c-jzr6x"
May 25 12:15:36.813361 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:36.813343    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
May 25 12:15:36.813678 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:36.813662    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
May 25 12:15:36.813796 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:36.813783    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-k9dhz\""
May 25 12:15:36.814765 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:36.814750    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
May 25 12:15:36.819632 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:36.819612    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
May 25 12:15:36.828622 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:36.828603    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8c795b88c-jzr6x"]
May 25 12:15:36.886726 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:36.886702    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-57bd96b888-jmzg5"]
May 25 12:15:36.888751 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:36.888735    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-57bd96b888-jmzg5"
May 25 12:15:36.890384 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:36.890363    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-8c795b88c-jzr6x"]
May 25 12:15:36.890532 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:15:36.890513    2555 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[bound-sa-token ca-trust-extracted image-registry-private-configuration installation-pull-secrets kube-api-access-2pdqh registry-certificates registry-tls trusted-ca], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-8c795b88c-jzr6x" podUID="ba09aab0-85f9-424b-b19f-5c930df4fe5c"
May 25 12:15:36.891005 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:36.890990    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
May 25 12:15:36.891060 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:36.891047    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
May 25 12:15:36.891239 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:36.891228    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-p65qq\""
May 25 12:15:36.899634 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:36.899607    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dqwr\" (UniqueName: \"kubernetes.io/projected/486a5c2b-34d6-4041-a7aa-f4bce7b4d1db-kube-api-access-8dqwr\") pod \"insights-runtime-extractor-bmlx5\" (UID: \"486a5c2b-34d6-4041-a7aa-f4bce7b4d1db\") " pod="openshift-insights/insights-runtime-extractor-bmlx5"
May 25 12:15:36.899751 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:36.899687    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ba09aab0-85f9-424b-b19f-5c930df4fe5c-trusted-ca\") pod \"image-registry-8c795b88c-jzr6x\" (UID: \"ba09aab0-85f9-424b-b19f-5c930df4fe5c\") " pod="openshift-image-registry/image-registry-8c795b88c-jzr6x"
May 25 12:15:36.899751 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:36.899723    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/486a5c2b-34d6-4041-a7aa-f4bce7b4d1db-crio-socket\") pod \"insights-runtime-extractor-bmlx5\" (UID: \"486a5c2b-34d6-4041-a7aa-f4bce7b4d1db\") " pod="openshift-insights/insights-runtime-extractor-bmlx5"
May 25 12:15:36.899855 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:36.899768    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ba09aab0-85f9-424b-b19f-5c930df4fe5c-registry-tls\") pod \"image-registry-8c795b88c-jzr6x\" (UID: \"ba09aab0-85f9-424b-b19f-5c930df4fe5c\") " pod="openshift-image-registry/image-registry-8c795b88c-jzr6x"
May 25 12:15:36.899855 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:36.899819    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ba09aab0-85f9-424b-b19f-5c930df4fe5c-registry-certificates\") pod \"image-registry-8c795b88c-jzr6x\" (UID: \"ba09aab0-85f9-424b-b19f-5c930df4fe5c\") " pod="openshift-image-registry/image-registry-8c795b88c-jzr6x"
May 25 12:15:36.899855 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:36.899839    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ba09aab0-85f9-424b-b19f-5c930df4fe5c-image-registry-private-configuration\") pod \"image-registry-8c795b88c-jzr6x\" (UID: \"ba09aab0-85f9-424b-b19f-5c930df4fe5c\") " pod="openshift-image-registry/image-registry-8c795b88c-jzr6x"
May 25 12:15:36.899995 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:36.899854    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ba09aab0-85f9-424b-b19f-5c930df4fe5c-installation-pull-secrets\") pod \"image-registry-8c795b88c-jzr6x\" (UID: \"ba09aab0-85f9-424b-b19f-5c930df4fe5c\") " pod="openshift-image-registry/image-registry-8c795b88c-jzr6x"
May 25 12:15:36.899995 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:36.899873    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/486a5c2b-34d6-4041-a7aa-f4bce7b4d1db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bmlx5\" (UID: \"486a5c2b-34d6-4041-a7aa-f4bce7b4d1db\") " pod="openshift-insights/insights-runtime-extractor-bmlx5"
May 25 12:15:36.899995 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:36.899938    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/486a5c2b-34d6-4041-a7aa-f4bce7b4d1db-data-volume\") pod \"insights-runtime-extractor-bmlx5\" (UID: \"486a5c2b-34d6-4041-a7aa-f4bce7b4d1db\") " pod="openshift-insights/insights-runtime-extractor-bmlx5"
May 25 12:15:36.899995 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:36.899967    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pdqh\" (UniqueName: \"kubernetes.io/projected/ba09aab0-85f9-424b-b19f-5c930df4fe5c-kube-api-access-2pdqh\") pod \"image-registry-8c795b88c-jzr6x\" (UID: \"ba09aab0-85f9-424b-b19f-5c930df4fe5c\") " pod="openshift-image-registry/image-registry-8c795b88c-jzr6x"
May 25 12:15:36.899995 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:36.899991    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ba09aab0-85f9-424b-b19f-5c930df4fe5c-ca-trust-extracted\") pod \"image-registry-8c795b88c-jzr6x\" (UID: \"ba09aab0-85f9-424b-b19f-5c930df4fe5c\") " pod="openshift-image-registry/image-registry-8c795b88c-jzr6x"
May 25 12:15:36.900227 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:36.900018    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ba09aab0-85f9-424b-b19f-5c930df4fe5c-bound-sa-token\") pod \"image-registry-8c795b88c-jzr6x\" (UID: \"ba09aab0-85f9-424b-b19f-5c930df4fe5c\") " pod="openshift-image-registry/image-registry-8c795b88c-jzr6x"
May 25 12:15:36.900227 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:36.900071    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/486a5c2b-34d6-4041-a7aa-f4bce7b4d1db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bmlx5\" (UID: \"486a5c2b-34d6-4041-a7aa-f4bce7b4d1db\") " pod="openshift-insights/insights-runtime-extractor-bmlx5"
May 25 12:15:36.900719 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:36.900698    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-57bd96b888-jmzg5"]
May 25 12:15:36.930161 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:36.930138    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5bb9b845b5-rznpl"]
May 25 12:15:36.931980 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:36.931966    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bb9b845b5-rznpl"
May 25 12:15:36.943914 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:36.943893    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5bb9b845b5-rznpl"]
May 25 12:15:37.000397 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.000367    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8dqwr\" (UniqueName: \"kubernetes.io/projected/486a5c2b-34d6-4041-a7aa-f4bce7b4d1db-kube-api-access-8dqwr\") pod \"insights-runtime-extractor-bmlx5\" (UID: \"486a5c2b-34d6-4041-a7aa-f4bce7b4d1db\") " pod="openshift-insights/insights-runtime-extractor-bmlx5"
May 25 12:15:37.000574 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.000405    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ba09aab0-85f9-424b-b19f-5c930df4fe5c-trusted-ca\") pod \"image-registry-8c795b88c-jzr6x\" (UID: \"ba09aab0-85f9-424b-b19f-5c930df4fe5c\") " pod="openshift-image-registry/image-registry-8c795b88c-jzr6x"
May 25 12:15:37.000574 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.000432    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/486a5c2b-34d6-4041-a7aa-f4bce7b4d1db-crio-socket\") pod \"insights-runtime-extractor-bmlx5\" (UID: \"486a5c2b-34d6-4041-a7aa-f4bce7b4d1db\") " pod="openshift-insights/insights-runtime-extractor-bmlx5"
May 25 12:15:37.000574 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.000481    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/486a5c2b-34d6-4041-a7aa-f4bce7b4d1db-crio-socket\") pod \"insights-runtime-extractor-bmlx5\" (UID: \"486a5c2b-34d6-4041-a7aa-f4bce7b4d1db\") " pod="openshift-insights/insights-runtime-extractor-bmlx5"
May 25 12:15:37.000574 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.000534    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ba09aab0-85f9-424b-b19f-5c930df4fe5c-registry-tls\") pod \"image-registry-8c795b88c-jzr6x\" (UID: \"ba09aab0-85f9-424b-b19f-5c930df4fe5c\") " pod="openshift-image-registry/image-registry-8c795b88c-jzr6x"
May 25 12:15:37.000574 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.000575    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ba09aab0-85f9-424b-b19f-5c930df4fe5c-registry-certificates\") pod \"image-registry-8c795b88c-jzr6x\" (UID: \"ba09aab0-85f9-424b-b19f-5c930df4fe5c\") " pod="openshift-image-registry/image-registry-8c795b88c-jzr6x"
May 25 12:15:37.000897 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.000602    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ba09aab0-85f9-424b-b19f-5c930df4fe5c-image-registry-private-configuration\") pod \"image-registry-8c795b88c-jzr6x\" (UID: \"ba09aab0-85f9-424b-b19f-5c930df4fe5c\") " pod="openshift-image-registry/image-registry-8c795b88c-jzr6x"
May 25 12:15:37.000897 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.000619    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ba09aab0-85f9-424b-b19f-5c930df4fe5c-installation-pull-secrets\") pod \"image-registry-8c795b88c-jzr6x\" (UID: \"ba09aab0-85f9-424b-b19f-5c930df4fe5c\") " pod="openshift-image-registry/image-registry-8c795b88c-jzr6x"
May 25 12:15:37.000897 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.000660    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/486a5c2b-34d6-4041-a7aa-f4bce7b4d1db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bmlx5\" (UID: \"486a5c2b-34d6-4041-a7aa-f4bce7b4d1db\") " pod="openshift-insights/insights-runtime-extractor-bmlx5"
May 25 12:15:37.000897 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.000713    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbcmz\" (UniqueName: \"kubernetes.io/projected/67702967-b5f5-412e-8d06-ecc66d064257-kube-api-access-tbcmz\") pod \"downloads-57bd96b888-jmzg5\" (UID: \"67702967-b5f5-412e-8d06-ecc66d064257\") " pod="openshift-console/downloads-57bd96b888-jmzg5"
May 25 12:15:37.000897 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.000752    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/486a5c2b-34d6-4041-a7aa-f4bce7b4d1db-data-volume\") pod \"insights-runtime-extractor-bmlx5\" (UID: \"486a5c2b-34d6-4041-a7aa-f4bce7b4d1db\") " pod="openshift-insights/insights-runtime-extractor-bmlx5"
May 25 12:15:37.000897 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.000782    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2pdqh\" (UniqueName: \"kubernetes.io/projected/ba09aab0-85f9-424b-b19f-5c930df4fe5c-kube-api-access-2pdqh\") pod \"image-registry-8c795b88c-jzr6x\" (UID: \"ba09aab0-85f9-424b-b19f-5c930df4fe5c\") " pod="openshift-image-registry/image-registry-8c795b88c-jzr6x"
May 25 12:15:37.000897 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.000812    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ba09aab0-85f9-424b-b19f-5c930df4fe5c-ca-trust-extracted\") pod \"image-registry-8c795b88c-jzr6x\" (UID: \"ba09aab0-85f9-424b-b19f-5c930df4fe5c\") " pod="openshift-image-registry/image-registry-8c795b88c-jzr6x"
May 25 12:15:37.000897 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.000846    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ba09aab0-85f9-424b-b19f-5c930df4fe5c-bound-sa-token\") pod \"image-registry-8c795b88c-jzr6x\" (UID: \"ba09aab0-85f9-424b-b19f-5c930df4fe5c\") " pod="openshift-image-registry/image-registry-8c795b88c-jzr6x"
May 25 12:15:37.001204 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.000903    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/486a5c2b-34d6-4041-a7aa-f4bce7b4d1db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bmlx5\" (UID: \"486a5c2b-34d6-4041-a7aa-f4bce7b4d1db\") " pod="openshift-insights/insights-runtime-extractor-bmlx5"
May 25 12:15:37.001500 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.001447    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ba09aab0-85f9-424b-b19f-5c930df4fe5c-ca-trust-extracted\") pod \"image-registry-8c795b88c-jzr6x\" (UID: \"ba09aab0-85f9-424b-b19f-5c930df4fe5c\") " pod="openshift-image-registry/image-registry-8c795b88c-jzr6x"
May 25 12:15:37.001632 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.001533    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/486a5c2b-34d6-4041-a7aa-f4bce7b4d1db-data-volume\") pod \"insights-runtime-extractor-bmlx5\" (UID: \"486a5c2b-34d6-4041-a7aa-f4bce7b4d1db\") " pod="openshift-insights/insights-runtime-extractor-bmlx5"
May 25 12:15:37.001719 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.001668    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ba09aab0-85f9-424b-b19f-5c930df4fe5c-trusted-ca\") pod \"image-registry-8c795b88c-jzr6x\" (UID: \"ba09aab0-85f9-424b-b19f-5c930df4fe5c\") " pod="openshift-image-registry/image-registry-8c795b88c-jzr6x"
May 25 12:15:37.003596 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.002087    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ba09aab0-85f9-424b-b19f-5c930df4fe5c-registry-certificates\") pod \"image-registry-8c795b88c-jzr6x\" (UID: \"ba09aab0-85f9-424b-b19f-5c930df4fe5c\") " pod="openshift-image-registry/image-registry-8c795b88c-jzr6x"
May 25 12:15:37.003596 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.002356    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/486a5c2b-34d6-4041-a7aa-f4bce7b4d1db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bmlx5\" (UID: \"486a5c2b-34d6-4041-a7aa-f4bce7b4d1db\") " pod="openshift-insights/insights-runtime-extractor-bmlx5"
May 25 12:15:37.003923 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.003884    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/486a5c2b-34d6-4041-a7aa-f4bce7b4d1db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bmlx5\" (UID: \"486a5c2b-34d6-4041-a7aa-f4bce7b4d1db\") " pod="openshift-insights/insights-runtime-extractor-bmlx5"
May 25 12:15:37.004295 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.004273    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ba09aab0-85f9-424b-b19f-5c930df4fe5c-installation-pull-secrets\") pod \"image-registry-8c795b88c-jzr6x\" (UID: \"ba09aab0-85f9-424b-b19f-5c930df4fe5c\") " pod="openshift-image-registry/image-registry-8c795b88c-jzr6x"
May 25 12:15:37.005726 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.005705    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ba09aab0-85f9-424b-b19f-5c930df4fe5c-registry-tls\") pod \"image-registry-8c795b88c-jzr6x\" (UID: \"ba09aab0-85f9-424b-b19f-5c930df4fe5c\") " pod="openshift-image-registry/image-registry-8c795b88c-jzr6x"
May 25 12:15:37.006123 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.006104    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8c795b88c-jzr6x"
May 25 12:15:37.009390 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.009365    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dqwr\" (UniqueName: \"kubernetes.io/projected/486a5c2b-34d6-4041-a7aa-f4bce7b4d1db-kube-api-access-8dqwr\") pod \"insights-runtime-extractor-bmlx5\" (UID: \"486a5c2b-34d6-4041-a7aa-f4bce7b4d1db\") " pod="openshift-insights/insights-runtime-extractor-bmlx5"
May 25 12:15:37.009873 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.009857    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pdqh\" (UniqueName: \"kubernetes.io/projected/ba09aab0-85f9-424b-b19f-5c930df4fe5c-kube-api-access-2pdqh\") pod \"image-registry-8c795b88c-jzr6x\" (UID: \"ba09aab0-85f9-424b-b19f-5c930df4fe5c\") " pod="openshift-image-registry/image-registry-8c795b88c-jzr6x"
May 25 12:15:37.010296 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.010273    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ba09aab0-85f9-424b-b19f-5c930df4fe5c-bound-sa-token\") pod \"image-registry-8c795b88c-jzr6x\" (UID: \"ba09aab0-85f9-424b-b19f-5c930df4fe5c\") " pod="openshift-image-registry/image-registry-8c795b88c-jzr6x"
May 25 12:15:37.010371 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.010317    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ba09aab0-85f9-424b-b19f-5c930df4fe5c-image-registry-private-configuration\") pod \"image-registry-8c795b88c-jzr6x\" (UID: \"ba09aab0-85f9-424b-b19f-5c930df4fe5c\") " pod="openshift-image-registry/image-registry-8c795b88c-jzr6x"
May 25 12:15:37.021352 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.021334    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8c795b88c-jzr6x"
May 25 12:15:37.095599 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.095535    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bmlx5"
May 25 12:15:37.101480 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.101458    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ba09aab0-85f9-424b-b19f-5c930df4fe5c-installation-pull-secrets\") pod \"ba09aab0-85f9-424b-b19f-5c930df4fe5c\" (UID: \"ba09aab0-85f9-424b-b19f-5c930df4fe5c\") "
May 25 12:15:37.101584 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.101502    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ba09aab0-85f9-424b-b19f-5c930df4fe5c-bound-sa-token\") pod \"ba09aab0-85f9-424b-b19f-5c930df4fe5c\" (UID: \"ba09aab0-85f9-424b-b19f-5c930df4fe5c\") "
May 25 12:15:37.101584 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.101532    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2pdqh\" (UniqueName: \"kubernetes.io/projected/ba09aab0-85f9-424b-b19f-5c930df4fe5c-kube-api-access-2pdqh\") pod \"ba09aab0-85f9-424b-b19f-5c930df4fe5c\" (UID: \"ba09aab0-85f9-424b-b19f-5c930df4fe5c\") "
May 25 12:15:37.101584 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.101559    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ba09aab0-85f9-424b-b19f-5c930df4fe5c-registry-certificates\") pod \"ba09aab0-85f9-424b-b19f-5c930df4fe5c\" (UID: \"ba09aab0-85f9-424b-b19f-5c930df4fe5c\") "
May 25 12:15:37.101771 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.101583    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ba09aab0-85f9-424b-b19f-5c930df4fe5c-ca-trust-extracted\") pod \"ba09aab0-85f9-424b-b19f-5c930df4fe5c\" (UID: \"ba09aab0-85f9-424b-b19f-5c930df4fe5c\") "
May 25 12:15:37.101771 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.101612    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ba09aab0-85f9-424b-b19f-5c930df4fe5c-trusted-ca\") pod \"ba09aab0-85f9-424b-b19f-5c930df4fe5c\" (UID: \"ba09aab0-85f9-424b-b19f-5c930df4fe5c\") "
May 25 12:15:37.101771 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.101725    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ba09aab0-85f9-424b-b19f-5c930df4fe5c-registry-tls\") pod \"ba09aab0-85f9-424b-b19f-5c930df4fe5c\" (UID: \"ba09aab0-85f9-424b-b19f-5c930df4fe5c\") "
May 25 12:15:37.101771 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.101751    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ba09aab0-85f9-424b-b19f-5c930df4fe5c-image-registry-private-configuration\") pod \"ba09aab0-85f9-424b-b19f-5c930df4fe5c\" (UID: \"ba09aab0-85f9-424b-b19f-5c930df4fe5c\") "
May 25 12:15:37.101970 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.101867    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tbcmz\" (UniqueName: \"kubernetes.io/projected/67702967-b5f5-412e-8d06-ecc66d064257-kube-api-access-tbcmz\") pod \"downloads-57bd96b888-jmzg5\" (UID: \"67702967-b5f5-412e-8d06-ecc66d064257\") " pod="openshift-console/downloads-57bd96b888-jmzg5"
May 25 12:15:37.101970 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.101904    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c429816e-6e0c-4c9d-931f-793af9bac4c6-bound-sa-token\") pod \"image-registry-5bb9b845b5-rznpl\" (UID: \"c429816e-6e0c-4c9d-931f-793af9bac4c6\") " pod="openshift-image-registry/image-registry-5bb9b845b5-rznpl"
May 25 12:15:37.101970 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.101888    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba09aab0-85f9-424b-b19f-5c930df4fe5c-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "ba09aab0-85f9-424b-b19f-5c930df4fe5c" (UID: "ba09aab0-85f9-424b-b19f-5c930df4fe5c"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:15:37.101970 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.101936    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c429816e-6e0c-4c9d-931f-793af9bac4c6-installation-pull-secrets\") pod \"image-registry-5bb9b845b5-rznpl\" (UID: \"c429816e-6e0c-4c9d-931f-793af9bac4c6\") " pod="openshift-image-registry/image-registry-5bb9b845b5-rznpl"
May 25 12:15:37.102175 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.101972    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c429816e-6e0c-4c9d-931f-793af9bac4c6-image-registry-private-configuration\") pod \"image-registry-5bb9b845b5-rznpl\" (UID: \"c429816e-6e0c-4c9d-931f-793af9bac4c6\") " pod="openshift-image-registry/image-registry-5bb9b845b5-rznpl"
May 25 12:15:37.102175 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.102000    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c429816e-6e0c-4c9d-931f-793af9bac4c6-registry-certificates\") pod \"image-registry-5bb9b845b5-rznpl\" (UID: \"c429816e-6e0c-4c9d-931f-793af9bac4c6\") " pod="openshift-image-registry/image-registry-5bb9b845b5-rznpl"
May 25 12:15:37.102175 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.102063    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c429816e-6e0c-4c9d-931f-793af9bac4c6-registry-tls\") pod \"image-registry-5bb9b845b5-rznpl\" (UID: \"c429816e-6e0c-4c9d-931f-793af9bac4c6\") " pod="openshift-image-registry/image-registry-5bb9b845b5-rznpl"
May 25 12:15:37.102175 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.102087    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba09aab0-85f9-424b-b19f-5c930df4fe5c-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "ba09aab0-85f9-424b-b19f-5c930df4fe5c" (UID: "ba09aab0-85f9-424b-b19f-5c930df4fe5c"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:15:37.102175 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.102108    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghkvd\" (UniqueName: \"kubernetes.io/projected/c429816e-6e0c-4c9d-931f-793af9bac4c6-kube-api-access-ghkvd\") pod \"image-registry-5bb9b845b5-rznpl\" (UID: \"c429816e-6e0c-4c9d-931f-793af9bac4c6\") " pod="openshift-image-registry/image-registry-5bb9b845b5-rznpl"
May 25 12:15:37.102175 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.102140    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c429816e-6e0c-4c9d-931f-793af9bac4c6-trusted-ca\") pod \"image-registry-5bb9b845b5-rznpl\" (UID: \"c429816e-6e0c-4c9d-931f-793af9bac4c6\") " pod="openshift-image-registry/image-registry-5bb9b845b5-rznpl"
May 25 12:15:37.102175 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.102164    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c429816e-6e0c-4c9d-931f-793af9bac4c6-ca-trust-extracted\") pod \"image-registry-5bb9b845b5-rznpl\" (UID: \"c429816e-6e0c-4c9d-931f-793af9bac4c6\") " pod="openshift-image-registry/image-registry-5bb9b845b5-rznpl"
May 25 12:15:37.102511 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.102217    2555 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ba09aab0-85f9-424b-b19f-5c930df4fe5c-ca-trust-extracted\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:15:37.102511 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.102231    2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ba09aab0-85f9-424b-b19f-5c930df4fe5c-trusted-ca\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:15:37.102701 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.102671    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba09aab0-85f9-424b-b19f-5c930df4fe5c-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "ba09aab0-85f9-424b-b19f-5c930df4fe5c" (UID: "ba09aab0-85f9-424b-b19f-5c930df4fe5c"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:15:37.104061 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.104018    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba09aab0-85f9-424b-b19f-5c930df4fe5c-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "ba09aab0-85f9-424b-b19f-5c930df4fe5c" (UID: "ba09aab0-85f9-424b-b19f-5c930df4fe5c"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:15:37.104151 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.104105    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba09aab0-85f9-424b-b19f-5c930df4fe5c-kube-api-access-2pdqh" (OuterVolumeSpecName: "kube-api-access-2pdqh") pod "ba09aab0-85f9-424b-b19f-5c930df4fe5c" (UID: "ba09aab0-85f9-424b-b19f-5c930df4fe5c"). InnerVolumeSpecName "kube-api-access-2pdqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:15:37.104301 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.104278    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba09aab0-85f9-424b-b19f-5c930df4fe5c-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "ba09aab0-85f9-424b-b19f-5c930df4fe5c" (UID: "ba09aab0-85f9-424b-b19f-5c930df4fe5c"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:15:37.104537 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.104511    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba09aab0-85f9-424b-b19f-5c930df4fe5c-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "ba09aab0-85f9-424b-b19f-5c930df4fe5c" (UID: "ba09aab0-85f9-424b-b19f-5c930df4fe5c"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:15:37.104609 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.104571    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba09aab0-85f9-424b-b19f-5c930df4fe5c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "ba09aab0-85f9-424b-b19f-5c930df4fe5c" (UID: "ba09aab0-85f9-424b-b19f-5c930df4fe5c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:15:37.109047 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.109025    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbcmz\" (UniqueName: \"kubernetes.io/projected/67702967-b5f5-412e-8d06-ecc66d064257-kube-api-access-tbcmz\") pod \"downloads-57bd96b888-jmzg5\" (UID: \"67702967-b5f5-412e-8d06-ecc66d064257\") " pod="openshift-console/downloads-57bd96b888-jmzg5"
May 25 12:15:37.197600 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.197570    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-57bd96b888-jmzg5"
May 25 12:15:37.203521 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.203495    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c429816e-6e0c-4c9d-931f-793af9bac4c6-registry-tls\") pod \"image-registry-5bb9b845b5-rznpl\" (UID: \"c429816e-6e0c-4c9d-931f-793af9bac4c6\") " pod="openshift-image-registry/image-registry-5bb9b845b5-rznpl"
May 25 12:15:37.203632 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.203547    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ghkvd\" (UniqueName: \"kubernetes.io/projected/c429816e-6e0c-4c9d-931f-793af9bac4c6-kube-api-access-ghkvd\") pod \"image-registry-5bb9b845b5-rznpl\" (UID: \"c429816e-6e0c-4c9d-931f-793af9bac4c6\") " pod="openshift-image-registry/image-registry-5bb9b845b5-rznpl"
May 25 12:15:37.203632 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.203576    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c429816e-6e0c-4c9d-931f-793af9bac4c6-trusted-ca\") pod \"image-registry-5bb9b845b5-rznpl\" (UID: \"c429816e-6e0c-4c9d-931f-793af9bac4c6\") " pod="openshift-image-registry/image-registry-5bb9b845b5-rznpl"
May 25 12:15:37.203632 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.203596    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c429816e-6e0c-4c9d-931f-793af9bac4c6-ca-trust-extracted\") pod \"image-registry-5bb9b845b5-rznpl\" (UID: \"c429816e-6e0c-4c9d-931f-793af9bac4c6\") " pod="openshift-image-registry/image-registry-5bb9b845b5-rznpl"
May 25 12:15:37.203794 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.203634    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c429816e-6e0c-4c9d-931f-793af9bac4c6-bound-sa-token\") pod \"image-registry-5bb9b845b5-rznpl\" (UID: \"c429816e-6e0c-4c9d-931f-793af9bac4c6\") " pod="openshift-image-registry/image-registry-5bb9b845b5-rznpl"
May 25 12:15:37.203794 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.203690    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c429816e-6e0c-4c9d-931f-793af9bac4c6-installation-pull-secrets\") pod \"image-registry-5bb9b845b5-rznpl\" (UID: \"c429816e-6e0c-4c9d-931f-793af9bac4c6\") " pod="openshift-image-registry/image-registry-5bb9b845b5-rznpl"
May 25 12:15:37.203794 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.203724    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c429816e-6e0c-4c9d-931f-793af9bac4c6-image-registry-private-configuration\") pod \"image-registry-5bb9b845b5-rznpl\" (UID: \"c429816e-6e0c-4c9d-931f-793af9bac4c6\") " pod="openshift-image-registry/image-registry-5bb9b845b5-rznpl"
May 25 12:15:37.203794 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.203755    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c429816e-6e0c-4c9d-931f-793af9bac4c6-registry-certificates\") pod \"image-registry-5bb9b845b5-rznpl\" (UID: \"c429816e-6e0c-4c9d-931f-793af9bac4c6\") " pod="openshift-image-registry/image-registry-5bb9b845b5-rznpl"
May 25 12:15:37.203962 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.203825    2555 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ba09aab0-85f9-424b-b19f-5c930df4fe5c-registry-tls\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:15:37.203962 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.203837    2555 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ba09aab0-85f9-424b-b19f-5c930df4fe5c-image-registry-private-configuration\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:15:37.203962 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.203853    2555 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ba09aab0-85f9-424b-b19f-5c930df4fe5c-installation-pull-secrets\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:15:37.203962 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.203868    2555 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ba09aab0-85f9-424b-b19f-5c930df4fe5c-bound-sa-token\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:15:37.203962 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.203881    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pdqh\" (UniqueName: \"kubernetes.io/projected/ba09aab0-85f9-424b-b19f-5c930df4fe5c-kube-api-access-2pdqh\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:15:37.203962 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.203894    2555 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ba09aab0-85f9-424b-b19f-5c930df4fe5c-registry-certificates\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:15:37.204290 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.204211    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c429816e-6e0c-4c9d-931f-793af9bac4c6-ca-trust-extracted\") pod \"image-registry-5bb9b845b5-rznpl\" (UID: \"c429816e-6e0c-4c9d-931f-793af9bac4c6\") " pod="openshift-image-registry/image-registry-5bb9b845b5-rznpl"
May 25 12:15:37.204855 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.204832    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c429816e-6e0c-4c9d-931f-793af9bac4c6-trusted-ca\") pod \"image-registry-5bb9b845b5-rznpl\" (UID: \"c429816e-6e0c-4c9d-931f-793af9bac4c6\") " pod="openshift-image-registry/image-registry-5bb9b845b5-rznpl"
May 25 12:15:37.205063 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.205041    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c429816e-6e0c-4c9d-931f-793af9bac4c6-registry-certificates\") pod \"image-registry-5bb9b845b5-rznpl\" (UID: \"c429816e-6e0c-4c9d-931f-793af9bac4c6\") " pod="openshift-image-registry/image-registry-5bb9b845b5-rznpl"
May 25 12:15:37.205985 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.205962    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c429816e-6e0c-4c9d-931f-793af9bac4c6-registry-tls\") pod \"image-registry-5bb9b845b5-rznpl\" (UID: \"c429816e-6e0c-4c9d-931f-793af9bac4c6\") " pod="openshift-image-registry/image-registry-5bb9b845b5-rznpl"
May 25 12:15:37.206539 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.206514    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c429816e-6e0c-4c9d-931f-793af9bac4c6-installation-pull-secrets\") pod \"image-registry-5bb9b845b5-rznpl\" (UID: \"c429816e-6e0c-4c9d-931f-793af9bac4c6\") " pod="openshift-image-registry/image-registry-5bb9b845b5-rznpl"
May 25 12:15:37.206652 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.206623    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c429816e-6e0c-4c9d-931f-793af9bac4c6-image-registry-private-configuration\") pod \"image-registry-5bb9b845b5-rznpl\" (UID: \"c429816e-6e0c-4c9d-931f-793af9bac4c6\") " pod="openshift-image-registry/image-registry-5bb9b845b5-rznpl"
May 25 12:15:37.210061 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.210030    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c429816e-6e0c-4c9d-931f-793af9bac4c6-bound-sa-token\") pod \"image-registry-5bb9b845b5-rznpl\" (UID: \"c429816e-6e0c-4c9d-931f-793af9bac4c6\") " pod="openshift-image-registry/image-registry-5bb9b845b5-rznpl"
May 25 12:15:37.210195 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.210178    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghkvd\" (UniqueName: \"kubernetes.io/projected/c429816e-6e0c-4c9d-931f-793af9bac4c6-kube-api-access-ghkvd\") pod \"image-registry-5bb9b845b5-rznpl\" (UID: \"c429816e-6e0c-4c9d-931f-793af9bac4c6\") " pod="openshift-image-registry/image-registry-5bb9b845b5-rznpl"
May 25 12:15:37.213890 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.213871    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bmlx5"]
May 25 12:15:37.218394 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:15:37.218352    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod486a5c2b_34d6_4041_a7aa_f4bce7b4d1db.slice/crio-2ff040e7a23c8481a3924272736073ee1060aaeb5bcf9e588c93bd8f702e86f9 WatchSource:0}: Error finding container 2ff040e7a23c8481a3924272736073ee1060aaeb5bcf9e588c93bd8f702e86f9: Status 404 returned error can't find the container with id 2ff040e7a23c8481a3924272736073ee1060aaeb5bcf9e588c93bd8f702e86f9
May 25 12:15:37.240794 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.240764    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bb9b845b5-rznpl"
May 25 12:15:37.331111 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.331012    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-57bd96b888-jmzg5"]
May 25 12:15:37.334117 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:15:37.334089    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67702967_b5f5_412e_8d06_ecc66d064257.slice/crio-d56a80a88abb77e725e506a643b53b3a7311ad0541ef62dfb459cdad73340b2c WatchSource:0}: Error finding container d56a80a88abb77e725e506a643b53b3a7311ad0541ef62dfb459cdad73340b2c: Status 404 returned error can't find the container with id d56a80a88abb77e725e506a643b53b3a7311ad0541ef62dfb459cdad73340b2c
May 25 12:15:37.376979 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:37.376907    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5bb9b845b5-rznpl"]
May 25 12:15:37.379300 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:15:37.379279    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc429816e_6e0c_4c9d_931f_793af9bac4c6.slice/crio-d2f1b89ee0d18830c68f9970f82594c630a7ab16aeb43d42b40452f66298e058 WatchSource:0}: Error finding container d2f1b89ee0d18830c68f9970f82594c630a7ab16aeb43d42b40452f66298e058: Status 404 returned error can't find the container with id d2f1b89ee0d18830c68f9970f82594c630a7ab16aeb43d42b40452f66298e058
May 25 12:15:38.010227 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:38.010198    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bmlx5" event={"ID":"486a5c2b-34d6-4041-a7aa-f4bce7b4d1db","Type":"ContainerStarted","Data":"2253196189be4ce4c7935188850393d3e7980806558d7d1597e243a21eaca72e"}
May 25 12:15:38.010575 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:38.010239    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bmlx5" event={"ID":"486a5c2b-34d6-4041-a7aa-f4bce7b4d1db","Type":"ContainerStarted","Data":"b61377259bedc915bd6a5d4ecf954a96cd0872409768bf53d46af335398acaf3"}
May 25 12:15:38.010575 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:38.010253    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bmlx5" event={"ID":"486a5c2b-34d6-4041-a7aa-f4bce7b4d1db","Type":"ContainerStarted","Data":"2ff040e7a23c8481a3924272736073ee1060aaeb5bcf9e588c93bd8f702e86f9"}
May 25 12:15:38.011417 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:38.011383    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-57bd96b888-jmzg5" event={"ID":"67702967-b5f5-412e-8d06-ecc66d064257","Type":"ContainerStarted","Data":"d56a80a88abb77e725e506a643b53b3a7311ad0541ef62dfb459cdad73340b2c"}
May 25 12:15:38.012861 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:38.012841    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bb9b845b5-rznpl" event={"ID":"c429816e-6e0c-4c9d-931f-793af9bac4c6","Type":"ContainerStarted","Data":"44d7d96a234f8b2c8abc10d4e5e398757cf643cf4c4c9aa13d9c1cab3ad07b12"}
May 25 12:15:38.012955 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:38.012867    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bb9b845b5-rznpl" event={"ID":"c429816e-6e0c-4c9d-931f-793af9bac4c6","Type":"ContainerStarted","Data":"d2f1b89ee0d18830c68f9970f82594c630a7ab16aeb43d42b40452f66298e058"}
May 25 12:15:38.012955 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:38.012877    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8c795b88c-jzr6x"
May 25 12:15:38.013226 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:38.013204    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5bb9b845b5-rznpl"
May 25 12:15:38.033160 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:38.033119    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5bb9b845b5-rznpl" podStartSLOduration=2.033104832 podStartE2EDuration="2.033104832s" podCreationTimestamp="2026-05-25 12:15:36 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:15:38.031764603 +0000 UTC m=+150.991815013" watchObservedRunningTime="2026-05-25 12:15:38.033104832 +0000 UTC m=+150.993155238"
May 25 12:15:38.054862 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:38.054832    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-8c795b88c-jzr6x"]
May 25 12:15:38.058272 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:38.058248    2555 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-8c795b88c-jzr6x"]
May 25 12:15:39.597892 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:39.597863    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba09aab0-85f9-424b-b19f-5c930df4fe5c" path="/var/lib/kubelet/pods/ba09aab0-85f9-424b-b19f-5c930df4fe5c/volumes"
May 25 12:15:40.019671 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:40.019625    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bmlx5" event={"ID":"486a5c2b-34d6-4041-a7aa-f4bce7b4d1db","Type":"ContainerStarted","Data":"b1f577cfd1e1a324653e9a9aa2acbf4d79d71ddcb7dd90b6a09f3cbb13b77465"}
May 25 12:15:43.979171 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:15:43.979125    2555 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-gn5dh" podUID="a5b139bd-e9a1-45b9-9d31-11a98965a2e0"
May 25 12:15:43.995302 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:15:43.995205    2555 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-8mblp" podUID="26ec0201-e1d2-4816-944f-bbdb03159591"
May 25 12:15:44.029849 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:44.029824    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gn5dh"
May 25 12:15:44.776582 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:44.776503    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-bmlx5" podStartSLOduration=6.589531194 podStartE2EDuration="8.776483353s" podCreationTimestamp="2026-05-25 12:15:36 +0000 UTC" firstStartedPulling="2026-05-25 12:15:37.281580271 +0000 UTC m=+150.241630655" lastFinishedPulling="2026-05-25 12:15:39.468532415 +0000 UTC m=+152.428582814" observedRunningTime="2026-05-25 12:15:40.03404534 +0000 UTC m=+152.994095767" watchObservedRunningTime="2026-05-25 12:15:44.776483353 +0000 UTC m=+157.736533760"
May 25 12:15:44.777244 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:44.777225    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-65657ccdf5-sbcfl"]
May 25 12:15:44.781657 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:44.781624    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65657ccdf5-sbcfl"
May 25 12:15:44.783636 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:44.783611    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
May 25 12:15:44.783768 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:44.783615    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
May 25 12:15:44.784233 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:44.784199    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-wc5r9\""
May 25 12:15:44.784233 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:44.784214    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
May 25 12:15:44.784417 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:44.784244    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
May 25 12:15:44.784500 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:44.784483    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
May 25 12:15:44.790588 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:44.790567    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65657ccdf5-sbcfl"]
May 25 12:15:44.976433 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:44.976379    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/904e6201-eb48-403d-822a-81f5ab186906-console-oauth-config\") pod \"console-65657ccdf5-sbcfl\" (UID: \"904e6201-eb48-403d-822a-81f5ab186906\") " pod="openshift-console/console-65657ccdf5-sbcfl"
May 25 12:15:44.976636 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:44.976443    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/904e6201-eb48-403d-822a-81f5ab186906-console-config\") pod \"console-65657ccdf5-sbcfl\" (UID: \"904e6201-eb48-403d-822a-81f5ab186906\") " pod="openshift-console/console-65657ccdf5-sbcfl"
May 25 12:15:44.976636 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:44.976486    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf9j2\" (UniqueName: \"kubernetes.io/projected/904e6201-eb48-403d-822a-81f5ab186906-kube-api-access-pf9j2\") pod \"console-65657ccdf5-sbcfl\" (UID: \"904e6201-eb48-403d-822a-81f5ab186906\") " pod="openshift-console/console-65657ccdf5-sbcfl"
May 25 12:15:44.976749 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:44.976677    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/904e6201-eb48-403d-822a-81f5ab186906-service-ca\") pod \"console-65657ccdf5-sbcfl\" (UID: \"904e6201-eb48-403d-822a-81f5ab186906\") " pod="openshift-console/console-65657ccdf5-sbcfl"
May 25 12:15:44.976749 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:44.976715    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/904e6201-eb48-403d-822a-81f5ab186906-console-serving-cert\") pod \"console-65657ccdf5-sbcfl\" (UID: \"904e6201-eb48-403d-822a-81f5ab186906\") " pod="openshift-console/console-65657ccdf5-sbcfl"
May 25 12:15:44.976836 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:44.976766    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/904e6201-eb48-403d-822a-81f5ab186906-oauth-serving-cert\") pod \"console-65657ccdf5-sbcfl\" (UID: \"904e6201-eb48-403d-822a-81f5ab186906\") " pod="openshift-console/console-65657ccdf5-sbcfl"
May 25 12:15:45.077783 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:45.077701    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/904e6201-eb48-403d-822a-81f5ab186906-service-ca\") pod \"console-65657ccdf5-sbcfl\" (UID: \"904e6201-eb48-403d-822a-81f5ab186906\") " pod="openshift-console/console-65657ccdf5-sbcfl"
May 25 12:15:45.077783 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:45.077746    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/904e6201-eb48-403d-822a-81f5ab186906-console-serving-cert\") pod \"console-65657ccdf5-sbcfl\" (UID: \"904e6201-eb48-403d-822a-81f5ab186906\") " pod="openshift-console/console-65657ccdf5-sbcfl"
May 25 12:15:45.078277 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:45.077796    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/904e6201-eb48-403d-822a-81f5ab186906-oauth-serving-cert\") pod \"console-65657ccdf5-sbcfl\" (UID: \"904e6201-eb48-403d-822a-81f5ab186906\") " pod="openshift-console/console-65657ccdf5-sbcfl"
May 25 12:15:45.078277 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:45.077827    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/904e6201-eb48-403d-822a-81f5ab186906-console-oauth-config\") pod \"console-65657ccdf5-sbcfl\" (UID: \"904e6201-eb48-403d-822a-81f5ab186906\") " pod="openshift-console/console-65657ccdf5-sbcfl"
May 25 12:15:45.078277 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:45.077988    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/904e6201-eb48-403d-822a-81f5ab186906-console-config\") pod \"console-65657ccdf5-sbcfl\" (UID: \"904e6201-eb48-403d-822a-81f5ab186906\") " pod="openshift-console/console-65657ccdf5-sbcfl"
May 25 12:15:45.078277 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:45.078048    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pf9j2\" (UniqueName: \"kubernetes.io/projected/904e6201-eb48-403d-822a-81f5ab186906-kube-api-access-pf9j2\") pod \"console-65657ccdf5-sbcfl\" (UID: \"904e6201-eb48-403d-822a-81f5ab186906\") " pod="openshift-console/console-65657ccdf5-sbcfl"
May 25 12:15:45.078474 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:45.078460    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/904e6201-eb48-403d-822a-81f5ab186906-service-ca\") pod \"console-65657ccdf5-sbcfl\" (UID: \"904e6201-eb48-403d-822a-81f5ab186906\") " pod="openshift-console/console-65657ccdf5-sbcfl"
May 25 12:15:45.078533 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:45.078504    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/904e6201-eb48-403d-822a-81f5ab186906-oauth-serving-cert\") pod \"console-65657ccdf5-sbcfl\" (UID: \"904e6201-eb48-403d-822a-81f5ab186906\") " pod="openshift-console/console-65657ccdf5-sbcfl"
May 25 12:15:45.078822 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:45.078797    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/904e6201-eb48-403d-822a-81f5ab186906-console-config\") pod \"console-65657ccdf5-sbcfl\" (UID: \"904e6201-eb48-403d-822a-81f5ab186906\") " pod="openshift-console/console-65657ccdf5-sbcfl"
May 25 12:15:45.080621 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:45.080601    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/904e6201-eb48-403d-822a-81f5ab186906-console-serving-cert\") pod \"console-65657ccdf5-sbcfl\" (UID: \"904e6201-eb48-403d-822a-81f5ab186906\") " pod="openshift-console/console-65657ccdf5-sbcfl"
May 25 12:15:45.080858 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:45.080841    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/904e6201-eb48-403d-822a-81f5ab186906-console-oauth-config\") pod \"console-65657ccdf5-sbcfl\" (UID: \"904e6201-eb48-403d-822a-81f5ab186906\") " pod="openshift-console/console-65657ccdf5-sbcfl"
May 25 12:15:45.086041 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:45.086021    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf9j2\" (UniqueName: \"kubernetes.io/projected/904e6201-eb48-403d-822a-81f5ab186906-kube-api-access-pf9j2\") pod \"console-65657ccdf5-sbcfl\" (UID: \"904e6201-eb48-403d-822a-81f5ab186906\") " pod="openshift-console/console-65657ccdf5-sbcfl"
May 25 12:15:45.092771 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:45.092750    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65657ccdf5-sbcfl"
May 25 12:15:45.221357 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:45.221322    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65657ccdf5-sbcfl"]
May 25 12:15:45.224710 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:15:45.224685    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod904e6201_eb48_403d_822a_81f5ab186906.slice/crio-d21771e3f9f4c8556f3bc4684a4424a585ae39cad9bc8bd8ca00f6f5e569ab6c WatchSource:0}: Error finding container d21771e3f9f4c8556f3bc4684a4424a585ae39cad9bc8bd8ca00f6f5e569ab6c: Status 404 returned error can't find the container with id d21771e3f9f4c8556f3bc4684a4424a585ae39cad9bc8bd8ca00f6f5e569ab6c
May 25 12:15:45.608732 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:15:45.608689    2555 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-nwlcx" podUID="30f9c156-1292-4927-9159-eba19cf76012"
May 25 12:15:46.037115 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:46.037073    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65657ccdf5-sbcfl" event={"ID":"904e6201-eb48-403d-822a-81f5ab186906","Type":"ContainerStarted","Data":"d21771e3f9f4c8556f3bc4684a4424a585ae39cad9bc8bd8ca00f6f5e569ab6c"}
May 25 12:15:46.276735 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:46.276630    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-7ltcx"]
May 25 12:15:46.279762 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:46.279738    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7779bbdd78-7ltcx"
May 25 12:15:46.281959 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:46.281841    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
May 25 12:15:46.281959 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:46.281867    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
May 25 12:15:46.282921 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:46.282657    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
May 25 12:15:46.282921 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:46.282701    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
May 25 12:15:46.282921 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:46.282656    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
May 25 12:15:46.282921 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:46.282894    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-r27wg\""
May 25 12:15:46.288554 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:46.288194    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/82327b68-6f0f-4a92-848b-83c34c6625ce-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-7ltcx\" (UID: \"82327b68-6f0f-4a92-848b-83c34c6625ce\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-7ltcx"
May 25 12:15:46.288554 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:46.288250    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76jpr\" (UniqueName: \"kubernetes.io/projected/82327b68-6f0f-4a92-848b-83c34c6625ce-kube-api-access-76jpr\") pod \"prometheus-operator-7779bbdd78-7ltcx\" (UID: \"82327b68-6f0f-4a92-848b-83c34c6625ce\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-7ltcx"
May 25 12:15:46.288554 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:46.288281    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/82327b68-6f0f-4a92-848b-83c34c6625ce-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-7ltcx\" (UID: \"82327b68-6f0f-4a92-848b-83c34c6625ce\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-7ltcx"
May 25 12:15:46.289024 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:46.288332    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/82327b68-6f0f-4a92-848b-83c34c6625ce-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-7ltcx\" (UID: \"82327b68-6f0f-4a92-848b-83c34c6625ce\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-7ltcx"
May 25 12:15:46.290258 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:46.289612    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-7ltcx"]
May 25 12:15:46.389749 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:46.389709    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/82327b68-6f0f-4a92-848b-83c34c6625ce-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-7ltcx\" (UID: \"82327b68-6f0f-4a92-848b-83c34c6625ce\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-7ltcx"
May 25 12:15:46.389923 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:46.389782    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-76jpr\" (UniqueName: \"kubernetes.io/projected/82327b68-6f0f-4a92-848b-83c34c6625ce-kube-api-access-76jpr\") pod \"prometheus-operator-7779bbdd78-7ltcx\" (UID: \"82327b68-6f0f-4a92-848b-83c34c6625ce\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-7ltcx"
May 25 12:15:46.389923 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:46.389814    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/82327b68-6f0f-4a92-848b-83c34c6625ce-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-7ltcx\" (UID: \"82327b68-6f0f-4a92-848b-83c34c6625ce\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-7ltcx"
May 25 12:15:46.389923 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:46.389854    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/82327b68-6f0f-4a92-848b-83c34c6625ce-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-7ltcx\" (UID: \"82327b68-6f0f-4a92-848b-83c34c6625ce\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-7ltcx"
May 25 12:15:46.390180 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:15:46.390032    2555 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
May 25 12:15:46.390180 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:15:46.390086    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82327b68-6f0f-4a92-848b-83c34c6625ce-prometheus-operator-tls podName:82327b68-6f0f-4a92-848b-83c34c6625ce nodeName:}" failed. No retries permitted until 2026-05-25 12:15:46.890067968 +0000 UTC m=+159.850118353 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/82327b68-6f0f-4a92-848b-83c34c6625ce-prometheus-operator-tls") pod "prometheus-operator-7779bbdd78-7ltcx" (UID: "82327b68-6f0f-4a92-848b-83c34c6625ce") : secret "prometheus-operator-tls" not found
May 25 12:15:46.390698 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:46.390633    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/82327b68-6f0f-4a92-848b-83c34c6625ce-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-7ltcx\" (UID: \"82327b68-6f0f-4a92-848b-83c34c6625ce\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-7ltcx"
May 25 12:15:46.392696 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:46.392661    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/82327b68-6f0f-4a92-848b-83c34c6625ce-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-7ltcx\" (UID: \"82327b68-6f0f-4a92-848b-83c34c6625ce\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-7ltcx"
May 25 12:15:46.398594 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:46.398569    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76jpr\" (UniqueName: \"kubernetes.io/projected/82327b68-6f0f-4a92-848b-83c34c6625ce-kube-api-access-76jpr\") pod \"prometheus-operator-7779bbdd78-7ltcx\" (UID: \"82327b68-6f0f-4a92-848b-83c34c6625ce\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-7ltcx"
May 25 12:15:46.560410 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:46.560332    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c674b587d-t6lxh"]
May 25 12:15:46.562946 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:46.562922    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c674b587d-t6lxh"
May 25 12:15:46.570434 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:46.570058    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
May 25 12:15:46.571502 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:46.571466    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c674b587d-t6lxh"]
May 25 12:15:46.591147 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:46.591120    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c458cd01-59c9-4a50-8af7-0e43855047c9-console-oauth-config\") pod \"console-6c674b587d-t6lxh\" (UID: \"c458cd01-59c9-4a50-8af7-0e43855047c9\") " pod="openshift-console/console-6c674b587d-t6lxh"
May 25 12:15:46.591274 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:46.591161    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c458cd01-59c9-4a50-8af7-0e43855047c9-oauth-serving-cert\") pod \"console-6c674b587d-t6lxh\" (UID: \"c458cd01-59c9-4a50-8af7-0e43855047c9\") " pod="openshift-console/console-6c674b587d-t6lxh"
May 25 12:15:46.591274 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:46.591191    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c458cd01-59c9-4a50-8af7-0e43855047c9-console-serving-cert\") pod \"console-6c674b587d-t6lxh\" (UID: \"c458cd01-59c9-4a50-8af7-0e43855047c9\") " pod="openshift-console/console-6c674b587d-t6lxh"
May 25 12:15:46.591274 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:46.591226    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c458cd01-59c9-4a50-8af7-0e43855047c9-trusted-ca-bundle\") pod \"console-6c674b587d-t6lxh\" (UID: \"c458cd01-59c9-4a50-8af7-0e43855047c9\") " pod="openshift-console/console-6c674b587d-t6lxh"
May 25 12:15:46.591274 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:46.591264    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c458cd01-59c9-4a50-8af7-0e43855047c9-service-ca\") pod \"console-6c674b587d-t6lxh\" (UID: \"c458cd01-59c9-4a50-8af7-0e43855047c9\") " pod="openshift-console/console-6c674b587d-t6lxh"
May 25 12:15:46.591485 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:46.591294    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffjlh\" (UniqueName: \"kubernetes.io/projected/c458cd01-59c9-4a50-8af7-0e43855047c9-kube-api-access-ffjlh\") pod \"console-6c674b587d-t6lxh\" (UID: \"c458cd01-59c9-4a50-8af7-0e43855047c9\") " pod="openshift-console/console-6c674b587d-t6lxh"
May 25 12:15:46.591485 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:46.591328    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c458cd01-59c9-4a50-8af7-0e43855047c9-console-config\") pod \"console-6c674b587d-t6lxh\" (UID: \"c458cd01-59c9-4a50-8af7-0e43855047c9\") " pod="openshift-console/console-6c674b587d-t6lxh"
May 25 12:15:46.692587 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:46.692555    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c458cd01-59c9-4a50-8af7-0e43855047c9-service-ca\") pod \"console-6c674b587d-t6lxh\" (UID: \"c458cd01-59c9-4a50-8af7-0e43855047c9\") " pod="openshift-console/console-6c674b587d-t6lxh"
May 25 12:15:46.692760 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:46.692596    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ffjlh\" (UniqueName: \"kubernetes.io/projected/c458cd01-59c9-4a50-8af7-0e43855047c9-kube-api-access-ffjlh\") pod \"console-6c674b587d-t6lxh\" (UID: \"c458cd01-59c9-4a50-8af7-0e43855047c9\") " pod="openshift-console/console-6c674b587d-t6lxh"
May 25 12:15:46.692760 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:46.692629    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c458cd01-59c9-4a50-8af7-0e43855047c9-console-config\") pod \"console-6c674b587d-t6lxh\" (UID: \"c458cd01-59c9-4a50-8af7-0e43855047c9\") " pod="openshift-console/console-6c674b587d-t6lxh"
May 25 12:15:46.692952 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:46.692927    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c458cd01-59c9-4a50-8af7-0e43855047c9-console-oauth-config\") pod \"console-6c674b587d-t6lxh\" (UID: \"c458cd01-59c9-4a50-8af7-0e43855047c9\") " pod="openshift-console/console-6c674b587d-t6lxh"
May 25 12:15:46.693075 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:46.692963    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c458cd01-59c9-4a50-8af7-0e43855047c9-oauth-serving-cert\") pod \"console-6c674b587d-t6lxh\" (UID: \"c458cd01-59c9-4a50-8af7-0e43855047c9\") " pod="openshift-console/console-6c674b587d-t6lxh"
May 25 12:15:46.693075 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:46.692991    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c458cd01-59c9-4a50-8af7-0e43855047c9-console-serving-cert\") pod \"console-6c674b587d-t6lxh\" (UID: \"c458cd01-59c9-4a50-8af7-0e43855047c9\") " pod="openshift-console/console-6c674b587d-t6lxh"
May 25 12:15:46.693075 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:46.693024    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c458cd01-59c9-4a50-8af7-0e43855047c9-trusted-ca-bundle\") pod \"console-6c674b587d-t6lxh\" (UID: \"c458cd01-59c9-4a50-8af7-0e43855047c9\") " pod="openshift-console/console-6c674b587d-t6lxh"
May 25 12:15:46.693468 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:46.693420    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c458cd01-59c9-4a50-8af7-0e43855047c9-service-ca\") pod \"console-6c674b587d-t6lxh\" (UID: \"c458cd01-59c9-4a50-8af7-0e43855047c9\") " pod="openshift-console/console-6c674b587d-t6lxh"
May 25 12:15:46.693468 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:46.693432    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c458cd01-59c9-4a50-8af7-0e43855047c9-console-config\") pod \"console-6c674b587d-t6lxh\" (UID: \"c458cd01-59c9-4a50-8af7-0e43855047c9\") " pod="openshift-console/console-6c674b587d-t6lxh"
May 25 12:15:46.693624 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:46.693581    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c458cd01-59c9-4a50-8af7-0e43855047c9-oauth-serving-cert\") pod \"console-6c674b587d-t6lxh\" (UID: \"c458cd01-59c9-4a50-8af7-0e43855047c9\") " pod="openshift-console/console-6c674b587d-t6lxh"
May 25 12:15:46.693813 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:46.693791    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c458cd01-59c9-4a50-8af7-0e43855047c9-trusted-ca-bundle\") pod \"console-6c674b587d-t6lxh\" (UID: \"c458cd01-59c9-4a50-8af7-0e43855047c9\") " pod="openshift-console/console-6c674b587d-t6lxh"
May 25 12:15:46.696278 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:46.696232    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c458cd01-59c9-4a50-8af7-0e43855047c9-console-serving-cert\") pod \"console-6c674b587d-t6lxh\" (UID: \"c458cd01-59c9-4a50-8af7-0e43855047c9\") " pod="openshift-console/console-6c674b587d-t6lxh"
May 25 12:15:46.696373 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:46.696361    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c458cd01-59c9-4a50-8af7-0e43855047c9-console-oauth-config\") pod \"console-6c674b587d-t6lxh\" (UID: \"c458cd01-59c9-4a50-8af7-0e43855047c9\") " pod="openshift-console/console-6c674b587d-t6lxh"
May 25 12:15:46.700328 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:46.700287    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffjlh\" (UniqueName: \"kubernetes.io/projected/c458cd01-59c9-4a50-8af7-0e43855047c9-kube-api-access-ffjlh\") pod \"console-6c674b587d-t6lxh\" (UID: \"c458cd01-59c9-4a50-8af7-0e43855047c9\") " pod="openshift-console/console-6c674b587d-t6lxh"
May 25 12:15:46.874908 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:46.874820    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c674b587d-t6lxh"
May 25 12:15:46.895056 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:46.895024    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/82327b68-6f0f-4a92-848b-83c34c6625ce-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-7ltcx\" (UID: \"82327b68-6f0f-4a92-848b-83c34c6625ce\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-7ltcx"
May 25 12:15:46.897593 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:46.897572    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/82327b68-6f0f-4a92-848b-83c34c6625ce-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-7ltcx\" (UID: \"82327b68-6f0f-4a92-848b-83c34c6625ce\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-7ltcx"
May 25 12:15:47.191944 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:47.191908    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7779bbdd78-7ltcx"
May 25 12:15:49.117553 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:49.117504    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26ec0201-e1d2-4816-944f-bbdb03159591-cert\") pod \"ingress-canary-8mblp\" (UID: \"26ec0201-e1d2-4816-944f-bbdb03159591\") " pod="openshift-ingress-canary/ingress-canary-8mblp"
May 25 12:15:49.118072 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:49.117611    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5b139bd-e9a1-45b9-9d31-11a98965a2e0-metrics-tls\") pod \"dns-default-gn5dh\" (UID: \"a5b139bd-e9a1-45b9-9d31-11a98965a2e0\") " pod="openshift-dns/dns-default-gn5dh"
May 25 12:15:49.120285 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:49.120259    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5b139bd-e9a1-45b9-9d31-11a98965a2e0-metrics-tls\") pod \"dns-default-gn5dh\" (UID: \"a5b139bd-e9a1-45b9-9d31-11a98965a2e0\") " pod="openshift-dns/dns-default-gn5dh"
May 25 12:15:49.120285 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:49.120272    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26ec0201-e1d2-4816-944f-bbdb03159591-cert\") pod \"ingress-canary-8mblp\" (UID: \"26ec0201-e1d2-4816-944f-bbdb03159591\") " pod="openshift-ingress-canary/ingress-canary-8mblp"
May 25 12:15:49.133004 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:49.132983    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fp9z9\""
May 25 12:15:49.141335 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:49.141308    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gn5dh"
May 25 12:15:53.693157 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:53.693054    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c674b587d-t6lxh"]
May 25 12:15:53.695541 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:15:53.695496    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc458cd01_59c9_4a50_8af7_0e43855047c9.slice/crio-6d4b90479f9eb8fcea07009568545d72bf03f4442b1ef5ff6fbe3a479640b502 WatchSource:0}: Error finding container 6d4b90479f9eb8fcea07009568545d72bf03f4442b1ef5ff6fbe3a479640b502: Status 404 returned error can't find the container with id 6d4b90479f9eb8fcea07009568545d72bf03f4442b1ef5ff6fbe3a479640b502
May 25 12:15:53.917157 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:53.917084    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-7ltcx"]
May 25 12:15:53.920128 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:53.920080    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gn5dh"]
May 25 12:15:53.920457 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:15:53.920432    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82327b68_6f0f_4a92_848b_83c34c6625ce.slice/crio-500db85e5db81ee056a1aa7ff7f747ebd5ca0541a111543b96987325a75a2419 WatchSource:0}: Error finding container 500db85e5db81ee056a1aa7ff7f747ebd5ca0541a111543b96987325a75a2419: Status 404 returned error can't find the container with id 500db85e5db81ee056a1aa7ff7f747ebd5ca0541a111543b96987325a75a2419
May 25 12:15:53.922955 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:15:53.922932    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5b139bd_e9a1_45b9_9d31_11a98965a2e0.slice/crio-eb1968ff8c5aac49077065f0d1e38311a727cc30207356b3d3d9426b5d7736fa WatchSource:0}: Error finding container eb1968ff8c5aac49077065f0d1e38311a727cc30207356b3d3d9426b5d7736fa: Status 404 returned error can't find the container with id eb1968ff8c5aac49077065f0d1e38311a727cc30207356b3d3d9426b5d7736fa
May 25 12:15:54.058239 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:54.058201    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-7ltcx" event={"ID":"82327b68-6f0f-4a92-848b-83c34c6625ce","Type":"ContainerStarted","Data":"500db85e5db81ee056a1aa7ff7f747ebd5ca0541a111543b96987325a75a2419"}
May 25 12:15:54.059975 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:54.059947    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c674b587d-t6lxh" event={"ID":"c458cd01-59c9-4a50-8af7-0e43855047c9","Type":"ContainerStarted","Data":"40b31b5153e1e37510fdd3801b2d3a62ea4a79a84b1abe003f8711b966ee4a30"}
May 25 12:15:54.060155 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:54.060140    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c674b587d-t6lxh" event={"ID":"c458cd01-59c9-4a50-8af7-0e43855047c9","Type":"ContainerStarted","Data":"6d4b90479f9eb8fcea07009568545d72bf03f4442b1ef5ff6fbe3a479640b502"}
May 25 12:15:54.061501 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:54.061456    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-57bd96b888-jmzg5" event={"ID":"67702967-b5f5-412e-8d06-ecc66d064257","Type":"ContainerStarted","Data":"6f7c44983cecf2bed304e72122c6f9d1a670248fbcc050168b9d456259e5ed0c"}
May 25 12:15:54.061726 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:54.061700    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-57bd96b888-jmzg5"
May 25 12:15:54.063245 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:54.063212    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65657ccdf5-sbcfl" event={"ID":"904e6201-eb48-403d-822a-81f5ab186906","Type":"ContainerStarted","Data":"41f3a1e86053dd33086f940fd6337865afe286df0b79ab74bb6d35b222bf934a"}
May 25 12:15:54.064686 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:54.064665    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gn5dh" event={"ID":"a5b139bd-e9a1-45b9-9d31-11a98965a2e0","Type":"ContainerStarted","Data":"eb1968ff8c5aac49077065f0d1e38311a727cc30207356b3d3d9426b5d7736fa"}
May 25 12:15:54.076873 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:54.076852    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-57bd96b888-jmzg5"
May 25 12:15:54.079149 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:54.079101    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-6c674b587d-t6lxh" podStartSLOduration=8.079086868 podStartE2EDuration="8.079086868s" podCreationTimestamp="2026-05-25 12:15:46 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:15:54.078215688 +0000 UTC m=+167.038266104" watchObservedRunningTime="2026-05-25 12:15:54.079086868 +0000 UTC m=+167.039137274"
May 25 12:15:54.092309 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:54.092267    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-65657ccdf5-sbcfl" podStartSLOduration=1.781043782 podStartE2EDuration="10.092255982s" podCreationTimestamp="2026-05-25 12:15:44 +0000 UTC" firstStartedPulling="2026-05-25 12:15:45.227395603 +0000 UTC m=+158.187445990" lastFinishedPulling="2026-05-25 12:15:53.538607796 +0000 UTC m=+166.498658190" observedRunningTime="2026-05-25 12:15:54.091617846 +0000 UTC m=+167.051668322" watchObservedRunningTime="2026-05-25 12:15:54.092255982 +0000 UTC m=+167.052306390"
May 25 12:15:54.105825 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:54.105787    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/downloads-57bd96b888-jmzg5" podStartSLOduration=1.851878944 podStartE2EDuration="18.105774151s" podCreationTimestamp="2026-05-25 12:15:36 +0000 UTC" firstStartedPulling="2026-05-25 12:15:37.335914882 +0000 UTC m=+150.295965265" lastFinishedPulling="2026-05-25 12:15:53.589810077 +0000 UTC m=+166.549860472" observedRunningTime="2026-05-25 12:15:54.104548483 +0000 UTC m=+167.064598890" watchObservedRunningTime="2026-05-25 12:15:54.105774151 +0000 UTC m=+167.065824570"
May 25 12:15:55.094013 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:55.093938    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-65657ccdf5-sbcfl"
May 25 12:15:55.094013 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:55.093984    2555 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-65657ccdf5-sbcfl"
May 25 12:15:55.100898 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:55.100608    2555 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-65657ccdf5-sbcfl"
May 25 12:15:55.594504 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:55.593978    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8mblp"
May 25 12:15:55.597384 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:55.596003    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-zkb72\""
May 25 12:15:55.604719 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:55.604696    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8mblp"
May 25 12:15:56.075525 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:56.075496    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-65657ccdf5-sbcfl"
May 25 12:15:56.398246 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:56.398210    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8mblp"]
May 25 12:15:56.403610 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:15:56.403560    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26ec0201_e1d2_4816_944f_bbdb03159591.slice/crio-c46ac8dc76d08c4e790eafafaba39529ce2236d17cdd1357ae8cf0cd48a9f9e1 WatchSource:0}: Error finding container c46ac8dc76d08c4e790eafafaba39529ce2236d17cdd1357ae8cf0cd48a9f9e1: Status 404 returned error can't find the container with id c46ac8dc76d08c4e790eafafaba39529ce2236d17cdd1357ae8cf0cd48a9f9e1
May 25 12:15:56.875112 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:56.875035    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c674b587d-t6lxh"
May 25 12:15:56.875112 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:56.875089    2555 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c674b587d-t6lxh"
May 25 12:15:56.880092 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:56.880070    2555 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c674b587d-t6lxh"
May 25 12:15:57.076527 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:57.076332    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gn5dh" event={"ID":"a5b139bd-e9a1-45b9-9d31-11a98965a2e0","Type":"ContainerStarted","Data":"aea5752d1e9aa4ba565f78e062f829562ca448ca603e4b49219857c6e53857b7"}
May 25 12:15:57.076527 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:57.076375    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gn5dh" event={"ID":"a5b139bd-e9a1-45b9-9d31-11a98965a2e0","Type":"ContainerStarted","Data":"34440a31bfdd2598cf4988b811c9b72758b4c4441442cd56043bab5737a63731"}
May 25 12:15:57.076527 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:57.076516    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-gn5dh"
May 25 12:15:57.078866 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:57.078792    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-7ltcx" event={"ID":"82327b68-6f0f-4a92-848b-83c34c6625ce","Type":"ContainerStarted","Data":"d29bb8ea6d352f085c2226685dffe461620736e325e6f9ca1dc1ffffb8b7af8a"}
May 25 12:15:57.078990 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:57.078828    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-7ltcx" event={"ID":"82327b68-6f0f-4a92-848b-83c34c6625ce","Type":"ContainerStarted","Data":"9cb42d33e2f37732bc0325c21a624a1717d31680b1dfd680701909d5495d0112"}
May 25 12:15:57.080948 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:57.080923    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8mblp" event={"ID":"26ec0201-e1d2-4816-944f-bbdb03159591","Type":"ContainerStarted","Data":"c46ac8dc76d08c4e790eafafaba39529ce2236d17cdd1357ae8cf0cd48a9f9e1"}
May 25 12:15:57.086253 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:57.086215    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c674b587d-t6lxh"
May 25 12:15:57.092471 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:57.092167    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-gn5dh" podStartSLOduration=134.770190442 podStartE2EDuration="2m17.092154932s" podCreationTimestamp="2026-05-25 12:13:40 +0000 UTC" firstStartedPulling="2026-05-25 12:15:53.924904303 +0000 UTC m=+166.884954691" lastFinishedPulling="2026-05-25 12:15:56.246868777 +0000 UTC m=+169.206919181" observedRunningTime="2026-05-25 12:15:57.091916465 +0000 UTC m=+170.051966873" watchObservedRunningTime="2026-05-25 12:15:57.092154932 +0000 UTC m=+170.052205341"
May 25 12:15:57.126918 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:57.126069    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7779bbdd78-7ltcx" podStartSLOduration=8.805875027999999 podStartE2EDuration="11.126051726s" podCreationTimestamp="2026-05-25 12:15:46 +0000 UTC" firstStartedPulling="2026-05-25 12:15:53.922683245 +0000 UTC m=+166.882733629" lastFinishedPulling="2026-05-25 12:15:56.24285993 +0000 UTC m=+169.202910327" observedRunningTime="2026-05-25 12:15:57.124833067 +0000 UTC m=+170.084883471" watchObservedRunningTime="2026-05-25 12:15:57.126051726 +0000 UTC m=+170.086102132"
May 25 12:15:57.143500 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:57.143446    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65657ccdf5-sbcfl"]
May 25 12:15:57.596409 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:57.596298    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwlcx"
May 25 12:15:58.650241 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:58.650207    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-n2rnq"]
May 25 12:15:58.678546 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:58.678519    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-n2rnq"
May 25 12:15:58.680881 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:58.680820    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
May 25 12:15:58.680881 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:58.680850    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
May 25 12:15:58.681261 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:58.681240    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qb5mz\""
May 25 12:15:58.681344 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:58.681278    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
May 25 12:15:58.804084 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:58.804055    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3a57d23a-40f6-46c3-b15e-4d1dd2608e11-node-exporter-wtmp\") pod \"node-exporter-n2rnq\" (UID: \"3a57d23a-40f6-46c3-b15e-4d1dd2608e11\") " pod="openshift-monitoring/node-exporter-n2rnq"
May 25 12:15:58.804179 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:58.804124    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3a57d23a-40f6-46c3-b15e-4d1dd2608e11-node-exporter-accelerators-collector-config\") pod \"node-exporter-n2rnq\" (UID: \"3a57d23a-40f6-46c3-b15e-4d1dd2608e11\") " pod="openshift-monitoring/node-exporter-n2rnq"
May 25 12:15:58.804179 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:58.804156    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a57d23a-40f6-46c3-b15e-4d1dd2608e11-metrics-client-ca\") pod \"node-exporter-n2rnq\" (UID: \"3a57d23a-40f6-46c3-b15e-4d1dd2608e11\") " pod="openshift-monitoring/node-exporter-n2rnq"
May 25 12:15:58.804299 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:58.804179    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a57d23a-40f6-46c3-b15e-4d1dd2608e11-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n2rnq\" (UID: \"3a57d23a-40f6-46c3-b15e-4d1dd2608e11\") " pod="openshift-monitoring/node-exporter-n2rnq"
May 25 12:15:58.804299 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:58.804224    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a57d23a-40f6-46c3-b15e-4d1dd2608e11-sys\") pod \"node-exporter-n2rnq\" (UID: \"3a57d23a-40f6-46c3-b15e-4d1dd2608e11\") " pod="openshift-monitoring/node-exporter-n2rnq"
May 25 12:15:58.804299 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:58.804275    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3a57d23a-40f6-46c3-b15e-4d1dd2608e11-node-exporter-tls\") pod \"node-exporter-n2rnq\" (UID: \"3a57d23a-40f6-46c3-b15e-4d1dd2608e11\") " pod="openshift-monitoring/node-exporter-n2rnq"
May 25 12:15:58.804455 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:58.804313    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3a57d23a-40f6-46c3-b15e-4d1dd2608e11-root\") pod \"node-exporter-n2rnq\" (UID: \"3a57d23a-40f6-46c3-b15e-4d1dd2608e11\") " pod="openshift-monitoring/node-exporter-n2rnq"
May 25 12:15:58.804455 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:58.804344    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwqgk\" (UniqueName: \"kubernetes.io/projected/3a57d23a-40f6-46c3-b15e-4d1dd2608e11-kube-api-access-nwqgk\") pod \"node-exporter-n2rnq\" (UID: \"3a57d23a-40f6-46c3-b15e-4d1dd2608e11\") " pod="openshift-monitoring/node-exporter-n2rnq"
May 25 12:15:58.804455 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:58.804372    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3a57d23a-40f6-46c3-b15e-4d1dd2608e11-node-exporter-textfile\") pod \"node-exporter-n2rnq\" (UID: \"3a57d23a-40f6-46c3-b15e-4d1dd2608e11\") " pod="openshift-monitoring/node-exporter-n2rnq"
May 25 12:15:58.909704 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:58.905419    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3a57d23a-40f6-46c3-b15e-4d1dd2608e11-node-exporter-accelerators-collector-config\") pod \"node-exporter-n2rnq\" (UID: \"3a57d23a-40f6-46c3-b15e-4d1dd2608e11\") " pod="openshift-monitoring/node-exporter-n2rnq"
May 25 12:15:58.909704 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:58.905493    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a57d23a-40f6-46c3-b15e-4d1dd2608e11-metrics-client-ca\") pod \"node-exporter-n2rnq\" (UID: \"3a57d23a-40f6-46c3-b15e-4d1dd2608e11\") " pod="openshift-monitoring/node-exporter-n2rnq"
May 25 12:15:58.909704 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:58.905522    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a57d23a-40f6-46c3-b15e-4d1dd2608e11-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n2rnq\" (UID: \"3a57d23a-40f6-46c3-b15e-4d1dd2608e11\") " pod="openshift-monitoring/node-exporter-n2rnq"
May 25 12:15:58.909704 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:58.905582    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a57d23a-40f6-46c3-b15e-4d1dd2608e11-sys\") pod \"node-exporter-n2rnq\" (UID: \"3a57d23a-40f6-46c3-b15e-4d1dd2608e11\") " pod="openshift-monitoring/node-exporter-n2rnq"
May 25 12:15:58.909704 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:58.905658    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3a57d23a-40f6-46c3-b15e-4d1dd2608e11-node-exporter-tls\") pod \"node-exporter-n2rnq\" (UID: \"3a57d23a-40f6-46c3-b15e-4d1dd2608e11\") " pod="openshift-monitoring/node-exporter-n2rnq"
May 25 12:15:58.909704 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:58.905699    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3a57d23a-40f6-46c3-b15e-4d1dd2608e11-root\") pod \"node-exporter-n2rnq\" (UID: \"3a57d23a-40f6-46c3-b15e-4d1dd2608e11\") " pod="openshift-monitoring/node-exporter-n2rnq"
May 25 12:15:58.909704 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:58.905730    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nwqgk\" (UniqueName: \"kubernetes.io/projected/3a57d23a-40f6-46c3-b15e-4d1dd2608e11-kube-api-access-nwqgk\") pod \"node-exporter-n2rnq\" (UID: \"3a57d23a-40f6-46c3-b15e-4d1dd2608e11\") " pod="openshift-monitoring/node-exporter-n2rnq"
May 25 12:15:58.909704 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:58.905767    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3a57d23a-40f6-46c3-b15e-4d1dd2608e11-node-exporter-textfile\") pod \"node-exporter-n2rnq\" (UID: \"3a57d23a-40f6-46c3-b15e-4d1dd2608e11\") " pod="openshift-monitoring/node-exporter-n2rnq"
May 25 12:15:58.909704 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:58.905809    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3a57d23a-40f6-46c3-b15e-4d1dd2608e11-node-exporter-wtmp\") pod \"node-exporter-n2rnq\" (UID: \"3a57d23a-40f6-46c3-b15e-4d1dd2608e11\") " pod="openshift-monitoring/node-exporter-n2rnq"
May 25 12:15:58.909704 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:58.906039    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3a57d23a-40f6-46c3-b15e-4d1dd2608e11-node-exporter-wtmp\") pod \"node-exporter-n2rnq\" (UID: \"3a57d23a-40f6-46c3-b15e-4d1dd2608e11\") " pod="openshift-monitoring/node-exporter-n2rnq"
May 25 12:15:58.909704 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:58.906036    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3a57d23a-40f6-46c3-b15e-4d1dd2608e11-node-exporter-accelerators-collector-config\") pod \"node-exporter-n2rnq\" (UID: \"3a57d23a-40f6-46c3-b15e-4d1dd2608e11\") " pod="openshift-monitoring/node-exporter-n2rnq"
May 25 12:15:58.909704 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:15:58.906142    2555 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
May 25 12:15:58.909704 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:15:58.906213    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a57d23a-40f6-46c3-b15e-4d1dd2608e11-node-exporter-tls podName:3a57d23a-40f6-46c3-b15e-4d1dd2608e11 nodeName:}" failed. No retries permitted until 2026-05-25 12:15:59.406195743 +0000 UTC m=+172.366246144 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/3a57d23a-40f6-46c3-b15e-4d1dd2608e11-node-exporter-tls") pod "node-exporter-n2rnq" (UID: "3a57d23a-40f6-46c3-b15e-4d1dd2608e11") : secret "node-exporter-tls" not found
May 25 12:15:58.909704 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:58.906795    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3a57d23a-40f6-46c3-b15e-4d1dd2608e11-node-exporter-textfile\") pod \"node-exporter-n2rnq\" (UID: \"3a57d23a-40f6-46c3-b15e-4d1dd2608e11\") " pod="openshift-monitoring/node-exporter-n2rnq"
May 25 12:15:58.909704 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:58.906882    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3a57d23a-40f6-46c3-b15e-4d1dd2608e11-root\") pod \"node-exporter-n2rnq\" (UID: \"3a57d23a-40f6-46c3-b15e-4d1dd2608e11\") " pod="openshift-monitoring/node-exporter-n2rnq"
May 25 12:15:58.909704 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:58.907107    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a57d23a-40f6-46c3-b15e-4d1dd2608e11-metrics-client-ca\") pod \"node-exporter-n2rnq\" (UID: \"3a57d23a-40f6-46c3-b15e-4d1dd2608e11\") " pod="openshift-monitoring/node-exporter-n2rnq"
May 25 12:15:58.909704 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:58.907204    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a57d23a-40f6-46c3-b15e-4d1dd2608e11-sys\") pod \"node-exporter-n2rnq\" (UID: \"3a57d23a-40f6-46c3-b15e-4d1dd2608e11\") " pod="openshift-monitoring/node-exporter-n2rnq"
May 25 12:15:58.910548 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:58.909626    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a57d23a-40f6-46c3-b15e-4d1dd2608e11-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n2rnq\" (UID: \"3a57d23a-40f6-46c3-b15e-4d1dd2608e11\") " pod="openshift-monitoring/node-exporter-n2rnq"
May 25 12:15:58.915539 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:58.915515    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwqgk\" (UniqueName: \"kubernetes.io/projected/3a57d23a-40f6-46c3-b15e-4d1dd2608e11-kube-api-access-nwqgk\") pod \"node-exporter-n2rnq\" (UID: \"3a57d23a-40f6-46c3-b15e-4d1dd2608e11\") " pod="openshift-monitoring/node-exporter-n2rnq"
May 25 12:15:59.020264 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:59.020235    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5bb9b845b5-rznpl"
May 25 12:15:59.090249 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:59.090197    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8mblp" event={"ID":"26ec0201-e1d2-4816-944f-bbdb03159591","Type":"ContainerStarted","Data":"3f85c341ba2505c09ba20c6f692fbd9392cdb0e5e38b943541cc41747b53d3d9"}
May 25 12:15:59.114572 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:59.114516    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8mblp" podStartSLOduration=136.82555865 podStartE2EDuration="2m19.11449939s" podCreationTimestamp="2026-05-25 12:13:40 +0000 UTC" firstStartedPulling="2026-05-25 12:15:56.405534359 +0000 UTC m=+169.365584755" lastFinishedPulling="2026-05-25 12:15:58.694475097 +0000 UTC m=+171.654525495" observedRunningTime="2026-05-25 12:15:59.11323276 +0000 UTC m=+172.073283169" watchObservedRunningTime="2026-05-25 12:15:59.11449939 +0000 UTC m=+172.074549795"
May 25 12:15:59.410267 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:59.410228    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3a57d23a-40f6-46c3-b15e-4d1dd2608e11-node-exporter-tls\") pod \"node-exporter-n2rnq\" (UID: \"3a57d23a-40f6-46c3-b15e-4d1dd2608e11\") " pod="openshift-monitoring/node-exporter-n2rnq"
May 25 12:15:59.412889 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:59.412860    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3a57d23a-40f6-46c3-b15e-4d1dd2608e11-node-exporter-tls\") pod \"node-exporter-n2rnq\" (UID: \"3a57d23a-40f6-46c3-b15e-4d1dd2608e11\") " pod="openshift-monitoring/node-exporter-n2rnq"
May 25 12:15:59.589610 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:15:59.589581    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-n2rnq"
May 25 12:16:00.094256 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:00.094221    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n2rnq" event={"ID":"3a57d23a-40f6-46c3-b15e-4d1dd2608e11","Type":"ContainerStarted","Data":"d316329d8395d4e3a38dbd2d3e66f3e52bd5b239324b6b636e52c46efccd693f"}
May 25 12:16:02.101898 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:02.101862    2555 generic.go:358] "Generic (PLEG): container finished" podID="3a57d23a-40f6-46c3-b15e-4d1dd2608e11" containerID="13b889e24fcd4d7a35c5fb20f029a7a4ec9b05f08b66b26bff1d5f0318eaa5c7" exitCode=0
May 25 12:16:02.102356 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:02.101949    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n2rnq" event={"ID":"3a57d23a-40f6-46c3-b15e-4d1dd2608e11","Type":"ContainerDied","Data":"13b889e24fcd4d7a35c5fb20f029a7a4ec9b05f08b66b26bff1d5f0318eaa5c7"}
May 25 12:16:03.105936 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:03.105903    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n2rnq" event={"ID":"3a57d23a-40f6-46c3-b15e-4d1dd2608e11","Type":"ContainerStarted","Data":"d263798609d8c14fe3c2533ec4a8085f772c60ad59376911cae58ecccaf612e5"}
May 25 12:16:03.105936 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:03.105936    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n2rnq" event={"ID":"3a57d23a-40f6-46c3-b15e-4d1dd2608e11","Type":"ContainerStarted","Data":"215c5d717bcc825cd03ba9b19ffe1549a057ee8da33e20b2e59ba7fc41b53c48"}
May 25 12:16:03.122976 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:03.122932    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-n2rnq" podStartSLOduration=3.628162056 podStartE2EDuration="5.12291799s" podCreationTimestamp="2026-05-25 12:15:58 +0000 UTC" firstStartedPulling="2026-05-25 12:15:59.602738317 +0000 UTC m=+172.562788700" lastFinishedPulling="2026-05-25 12:16:01.097494245 +0000 UTC m=+174.057544634" observedRunningTime="2026-05-25 12:16:03.121500398 +0000 UTC m=+176.081550803" watchObservedRunningTime="2026-05-25 12:16:03.12291799 +0000 UTC m=+176.082968394"
May 25 12:16:07.087163 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:07.087132    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gn5dh"
May 25 12:16:14.358783 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:14.358751    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c674b587d-t6lxh"]
May 25 12:16:19.148865 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:19.148789    2555 generic.go:358] "Generic (PLEG): container finished" podID="3b30b319-e998-4354-99b7-a46815d537cf" containerID="aba3f4dffe8aec2704832b310eb571be150a50e5364bd4e835f79aa0375303c2" exitCode=0
May 25 12:16:19.148865 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:19.148840    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-sp5bh" event={"ID":"3b30b319-e998-4354-99b7-a46815d537cf","Type":"ContainerDied","Data":"aba3f4dffe8aec2704832b310eb571be150a50e5364bd4e835f79aa0375303c2"}
May 25 12:16:19.149262 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:19.149157    2555 scope.go:117] "RemoveContainer" containerID="aba3f4dffe8aec2704832b310eb571be150a50e5364bd4e835f79aa0375303c2"
May 25 12:16:19.803011 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:19.802985    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8mblp_26ec0201-e1d2-4816-944f-bbdb03159591/serve-healthcheck-canary/0.log"
May 25 12:16:20.152577 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:20.152498    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-sp5bh" event={"ID":"3b30b319-e998-4354-99b7-a46815d537cf","Type":"ContainerStarted","Data":"99c44fa13bd90fb2d7e9b71ba7d9407f9f59a54d095f6a27ec4da3706df75e52"}
May 25 12:16:24.112078 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:24.112040    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-65657ccdf5-sbcfl" podUID="904e6201-eb48-403d-822a-81f5ab186906" containerName="console" containerID="cri-o://41f3a1e86053dd33086f940fd6337865afe286df0b79ab74bb6d35b222bf934a" gracePeriod=15
May 25 12:16:24.380753 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:24.380728    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-65657ccdf5-sbcfl_904e6201-eb48-403d-822a-81f5ab186906/console/0.log"
May 25 12:16:24.380876 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:24.380808    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65657ccdf5-sbcfl"
May 25 12:16:24.509026 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:24.508999    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/904e6201-eb48-403d-822a-81f5ab186906-service-ca\") pod \"904e6201-eb48-403d-822a-81f5ab186906\" (UID: \"904e6201-eb48-403d-822a-81f5ab186906\") "
May 25 12:16:24.509180 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:24.509064    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/904e6201-eb48-403d-822a-81f5ab186906-console-oauth-config\") pod \"904e6201-eb48-403d-822a-81f5ab186906\" (UID: \"904e6201-eb48-403d-822a-81f5ab186906\") "
May 25 12:16:24.509180 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:24.509159    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/904e6201-eb48-403d-822a-81f5ab186906-console-config\") pod \"904e6201-eb48-403d-822a-81f5ab186906\" (UID: \"904e6201-eb48-403d-822a-81f5ab186906\") "
May 25 12:16:24.509294 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:24.509194    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pf9j2\" (UniqueName: \"kubernetes.io/projected/904e6201-eb48-403d-822a-81f5ab186906-kube-api-access-pf9j2\") pod \"904e6201-eb48-403d-822a-81f5ab186906\" (UID: \"904e6201-eb48-403d-822a-81f5ab186906\") "
May 25 12:16:24.509294 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:24.509234    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/904e6201-eb48-403d-822a-81f5ab186906-console-serving-cert\") pod \"904e6201-eb48-403d-822a-81f5ab186906\" (UID: \"904e6201-eb48-403d-822a-81f5ab186906\") "
May 25 12:16:24.509294 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:24.509250    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/904e6201-eb48-403d-822a-81f5ab186906-oauth-serving-cert\") pod \"904e6201-eb48-403d-822a-81f5ab186906\" (UID: \"904e6201-eb48-403d-822a-81f5ab186906\") "
May 25 12:16:24.509479 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:24.509374    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/904e6201-eb48-403d-822a-81f5ab186906-service-ca" (OuterVolumeSpecName: "service-ca") pod "904e6201-eb48-403d-822a-81f5ab186906" (UID: "904e6201-eb48-403d-822a-81f5ab186906"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:16:24.509479 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:24.509449    2555 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/904e6201-eb48-403d-822a-81f5ab186906-service-ca\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:16:24.509479 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:24.509466    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/904e6201-eb48-403d-822a-81f5ab186906-console-config" (OuterVolumeSpecName: "console-config") pod "904e6201-eb48-403d-822a-81f5ab186906" (UID: "904e6201-eb48-403d-822a-81f5ab186906"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:16:24.509757 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:24.509732    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/904e6201-eb48-403d-822a-81f5ab186906-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "904e6201-eb48-403d-822a-81f5ab186906" (UID: "904e6201-eb48-403d-822a-81f5ab186906"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:16:24.511250 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:24.511222    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/904e6201-eb48-403d-822a-81f5ab186906-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "904e6201-eb48-403d-822a-81f5ab186906" (UID: "904e6201-eb48-403d-822a-81f5ab186906"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:16:24.511343 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:24.511280    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/904e6201-eb48-403d-822a-81f5ab186906-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "904e6201-eb48-403d-822a-81f5ab186906" (UID: "904e6201-eb48-403d-822a-81f5ab186906"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:16:24.511343 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:24.511285    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/904e6201-eb48-403d-822a-81f5ab186906-kube-api-access-pf9j2" (OuterVolumeSpecName: "kube-api-access-pf9j2") pod "904e6201-eb48-403d-822a-81f5ab186906" (UID: "904e6201-eb48-403d-822a-81f5ab186906"). InnerVolumeSpecName "kube-api-access-pf9j2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:16:24.610698 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:24.610673    2555 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/904e6201-eb48-403d-822a-81f5ab186906-console-serving-cert\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:16:24.610698 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:24.610696    2555 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/904e6201-eb48-403d-822a-81f5ab186906-oauth-serving-cert\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:16:24.610835 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:24.610705    2555 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/904e6201-eb48-403d-822a-81f5ab186906-console-oauth-config\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:16:24.610835 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:24.610715    2555 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/904e6201-eb48-403d-822a-81f5ab186906-console-config\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:16:24.610835 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:24.610725    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf9j2\" (UniqueName: \"kubernetes.io/projected/904e6201-eb48-403d-822a-81f5ab186906-kube-api-access-pf9j2\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:16:25.166225 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:25.166195    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-65657ccdf5-sbcfl_904e6201-eb48-403d-822a-81f5ab186906/console/0.log"
May 25 12:16:25.166683 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:25.166232    2555 generic.go:358] "Generic (PLEG): container finished" podID="904e6201-eb48-403d-822a-81f5ab186906" containerID="41f3a1e86053dd33086f940fd6337865afe286df0b79ab74bb6d35b222bf934a" exitCode=2
May 25 12:16:25.166683 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:25.166262    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65657ccdf5-sbcfl" event={"ID":"904e6201-eb48-403d-822a-81f5ab186906","Type":"ContainerDied","Data":"41f3a1e86053dd33086f940fd6337865afe286df0b79ab74bb6d35b222bf934a"}
May 25 12:16:25.166683 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:25.166293    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65657ccdf5-sbcfl"
May 25 12:16:25.166683 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:25.166311    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65657ccdf5-sbcfl" event={"ID":"904e6201-eb48-403d-822a-81f5ab186906","Type":"ContainerDied","Data":"d21771e3f9f4c8556f3bc4684a4424a585ae39cad9bc8bd8ca00f6f5e569ab6c"}
May 25 12:16:25.166683 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:25.166334    2555 scope.go:117] "RemoveContainer" containerID="41f3a1e86053dd33086f940fd6337865afe286df0b79ab74bb6d35b222bf934a"
May 25 12:16:25.174234 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:25.174217    2555 scope.go:117] "RemoveContainer" containerID="41f3a1e86053dd33086f940fd6337865afe286df0b79ab74bb6d35b222bf934a"
May 25 12:16:25.174493 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:16:25.174471    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41f3a1e86053dd33086f940fd6337865afe286df0b79ab74bb6d35b222bf934a\": container with ID starting with 41f3a1e86053dd33086f940fd6337865afe286df0b79ab74bb6d35b222bf934a not found: ID does not exist" containerID="41f3a1e86053dd33086f940fd6337865afe286df0b79ab74bb6d35b222bf934a"
May 25 12:16:25.174566 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:25.174504    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41f3a1e86053dd33086f940fd6337865afe286df0b79ab74bb6d35b222bf934a"} err="failed to get container status \"41f3a1e86053dd33086f940fd6337865afe286df0b79ab74bb6d35b222bf934a\": rpc error: code = NotFound desc = could not find container \"41f3a1e86053dd33086f940fd6337865afe286df0b79ab74bb6d35b222bf934a\": container with ID starting with 41f3a1e86053dd33086f940fd6337865afe286df0b79ab74bb6d35b222bf934a not found: ID does not exist"
May 25 12:16:25.185169 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:25.185150    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65657ccdf5-sbcfl"]
May 25 12:16:25.188848 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:25.188826    2555 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-65657ccdf5-sbcfl"]
May 25 12:16:25.597821 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:25.597795    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="904e6201-eb48-403d-822a-81f5ab186906" path="/var/lib/kubelet/pods/904e6201-eb48-403d-822a-81f5ab186906/volumes"
May 25 12:16:29.178743 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:29.178660    2555 generic.go:358] "Generic (PLEG): container finished" podID="01c35034-4da3-4712-b62f-77b9082e49fd" containerID="f42b8f4a91787448e3cdb13a75ba7dcf5a1f173ee0c5a09c6461f1be03c963b6" exitCode=0
May 25 12:16:29.178743 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:29.178713    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-cz2wm" event={"ID":"01c35034-4da3-4712-b62f-77b9082e49fd","Type":"ContainerDied","Data":"f42b8f4a91787448e3cdb13a75ba7dcf5a1f173ee0c5a09c6461f1be03c963b6"}
May 25 12:16:29.179257 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:29.178997    2555 scope.go:117] "RemoveContainer" containerID="f42b8f4a91787448e3cdb13a75ba7dcf5a1f173ee0c5a09c6461f1be03c963b6"
May 25 12:16:30.183192 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:30.183159    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-cz2wm" event={"ID":"01c35034-4da3-4712-b62f-77b9082e49fd","Type":"ContainerStarted","Data":"4727d3ba98efd22203f56feb31350ab72316597bcc3cb9f03209912b25f8bd5a"}
May 25 12:16:39.212560 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:39.212524    2555 generic.go:358] "Generic (PLEG): container finished" podID="94423b0c-4155-43d2-8037-2054e0f36255" containerID="17ede3dccb11988def47ffe015750142b59a03893cb79a90f8d356ed5ceef0bf" exitCode=0
May 25 12:16:39.212973 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:39.212583    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-sch65" event={"ID":"94423b0c-4155-43d2-8037-2054e0f36255","Type":"ContainerDied","Data":"17ede3dccb11988def47ffe015750142b59a03893cb79a90f8d356ed5ceef0bf"}
May 25 12:16:39.212973 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:39.212892    2555 scope.go:117] "RemoveContainer" containerID="17ede3dccb11988def47ffe015750142b59a03893cb79a90f8d356ed5ceef0bf"
May 25 12:16:39.377474 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:39.377437    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-6c674b587d-t6lxh" podUID="c458cd01-59c9-4a50-8af7-0e43855047c9" containerName="console" containerID="cri-o://40b31b5153e1e37510fdd3801b2d3a62ea4a79a84b1abe003f8711b966ee4a30" gracePeriod=15
May 25 12:16:39.636281 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:39.636257    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6c674b587d-t6lxh_c458cd01-59c9-4a50-8af7-0e43855047c9/console/0.log"
May 25 12:16:39.636388 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:39.636327    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c674b587d-t6lxh"
May 25 12:16:39.825881 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:39.825799    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c458cd01-59c9-4a50-8af7-0e43855047c9-oauth-serving-cert\") pod \"c458cd01-59c9-4a50-8af7-0e43855047c9\" (UID: \"c458cd01-59c9-4a50-8af7-0e43855047c9\") "
May 25 12:16:39.825881 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:39.825832    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c458cd01-59c9-4a50-8af7-0e43855047c9-console-serving-cert\") pod \"c458cd01-59c9-4a50-8af7-0e43855047c9\" (UID: \"c458cd01-59c9-4a50-8af7-0e43855047c9\") "
May 25 12:16:39.825881 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:39.825865    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c458cd01-59c9-4a50-8af7-0e43855047c9-trusted-ca-bundle\") pod \"c458cd01-59c9-4a50-8af7-0e43855047c9\" (UID: \"c458cd01-59c9-4a50-8af7-0e43855047c9\") "
May 25 12:16:39.826152 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:39.825888    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c458cd01-59c9-4a50-8af7-0e43855047c9-service-ca\") pod \"c458cd01-59c9-4a50-8af7-0e43855047c9\" (UID: \"c458cd01-59c9-4a50-8af7-0e43855047c9\") "
May 25 12:16:39.826152 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:39.825916    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c458cd01-59c9-4a50-8af7-0e43855047c9-console-oauth-config\") pod \"c458cd01-59c9-4a50-8af7-0e43855047c9\" (UID: \"c458cd01-59c9-4a50-8af7-0e43855047c9\") "
May 25 12:16:39.826152 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:39.825947    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c458cd01-59c9-4a50-8af7-0e43855047c9-console-config\") pod \"c458cd01-59c9-4a50-8af7-0e43855047c9\" (UID: \"c458cd01-59c9-4a50-8af7-0e43855047c9\") "
May 25 12:16:39.826152 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:39.825991    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ffjlh\" (UniqueName: \"kubernetes.io/projected/c458cd01-59c9-4a50-8af7-0e43855047c9-kube-api-access-ffjlh\") pod \"c458cd01-59c9-4a50-8af7-0e43855047c9\" (UID: \"c458cd01-59c9-4a50-8af7-0e43855047c9\") "
May 25 12:16:39.826328 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:39.826272    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c458cd01-59c9-4a50-8af7-0e43855047c9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c458cd01-59c9-4a50-8af7-0e43855047c9" (UID: "c458cd01-59c9-4a50-8af7-0e43855047c9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:16:39.826328 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:39.826280    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c458cd01-59c9-4a50-8af7-0e43855047c9-service-ca" (OuterVolumeSpecName: "service-ca") pod "c458cd01-59c9-4a50-8af7-0e43855047c9" (UID: "c458cd01-59c9-4a50-8af7-0e43855047c9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:16:39.826422 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:39.826387    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c458cd01-59c9-4a50-8af7-0e43855047c9-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c458cd01-59c9-4a50-8af7-0e43855047c9" (UID: "c458cd01-59c9-4a50-8af7-0e43855047c9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:16:39.826484 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:39.826457    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c458cd01-59c9-4a50-8af7-0e43855047c9-console-config" (OuterVolumeSpecName: "console-config") pod "c458cd01-59c9-4a50-8af7-0e43855047c9" (UID: "c458cd01-59c9-4a50-8af7-0e43855047c9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:16:39.827913 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:39.827887    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c458cd01-59c9-4a50-8af7-0e43855047c9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c458cd01-59c9-4a50-8af7-0e43855047c9" (UID: "c458cd01-59c9-4a50-8af7-0e43855047c9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:16:39.828068 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:39.828050    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c458cd01-59c9-4a50-8af7-0e43855047c9-kube-api-access-ffjlh" (OuterVolumeSpecName: "kube-api-access-ffjlh") pod "c458cd01-59c9-4a50-8af7-0e43855047c9" (UID: "c458cd01-59c9-4a50-8af7-0e43855047c9"). InnerVolumeSpecName "kube-api-access-ffjlh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:16:39.828113 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:39.828088    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c458cd01-59c9-4a50-8af7-0e43855047c9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c458cd01-59c9-4a50-8af7-0e43855047c9" (UID: "c458cd01-59c9-4a50-8af7-0e43855047c9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:16:39.926903 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:39.926873    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffjlh\" (UniqueName: \"kubernetes.io/projected/c458cd01-59c9-4a50-8af7-0e43855047c9-kube-api-access-ffjlh\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:16:39.926903 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:39.926897    2555 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c458cd01-59c9-4a50-8af7-0e43855047c9-oauth-serving-cert\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:16:39.926903 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:39.926908    2555 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c458cd01-59c9-4a50-8af7-0e43855047c9-console-serving-cert\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:16:39.927093 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:39.926917    2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c458cd01-59c9-4a50-8af7-0e43855047c9-trusted-ca-bundle\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:16:39.927093 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:39.926927    2555 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c458cd01-59c9-4a50-8af7-0e43855047c9-service-ca\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:16:39.927093 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:39.926938    2555 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c458cd01-59c9-4a50-8af7-0e43855047c9-console-oauth-config\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:16:39.927093 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:39.926946    2555 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c458cd01-59c9-4a50-8af7-0e43855047c9-console-config\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:16:40.217729 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:40.217697    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-sch65" event={"ID":"94423b0c-4155-43d2-8037-2054e0f36255","Type":"ContainerStarted","Data":"06ad3e3dbf4bd2425cf0171d6885f9867daf385f573929a9ab418d96073b8541"}
May 25 12:16:40.218837 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:40.218822    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6c674b587d-t6lxh_c458cd01-59c9-4a50-8af7-0e43855047c9/console/0.log"
May 25 12:16:40.218932 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:40.218851    2555 generic.go:358] "Generic (PLEG): container finished" podID="c458cd01-59c9-4a50-8af7-0e43855047c9" containerID="40b31b5153e1e37510fdd3801b2d3a62ea4a79a84b1abe003f8711b966ee4a30" exitCode=2
May 25 12:16:40.218932 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:40.218874    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c674b587d-t6lxh" event={"ID":"c458cd01-59c9-4a50-8af7-0e43855047c9","Type":"ContainerDied","Data":"40b31b5153e1e37510fdd3801b2d3a62ea4a79a84b1abe003f8711b966ee4a30"}
May 25 12:16:40.218932 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:40.218894    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c674b587d-t6lxh"
May 25 12:16:40.218932 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:40.218904    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c674b587d-t6lxh" event={"ID":"c458cd01-59c9-4a50-8af7-0e43855047c9","Type":"ContainerDied","Data":"6d4b90479f9eb8fcea07009568545d72bf03f4442b1ef5ff6fbe3a479640b502"}
May 25 12:16:40.218932 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:40.218919    2555 scope.go:117] "RemoveContainer" containerID="40b31b5153e1e37510fdd3801b2d3a62ea4a79a84b1abe003f8711b966ee4a30"
May 25 12:16:40.226627 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:40.226612    2555 scope.go:117] "RemoveContainer" containerID="40b31b5153e1e37510fdd3801b2d3a62ea4a79a84b1abe003f8711b966ee4a30"
May 25 12:16:40.226928 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:16:40.226910    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40b31b5153e1e37510fdd3801b2d3a62ea4a79a84b1abe003f8711b966ee4a30\": container with ID starting with 40b31b5153e1e37510fdd3801b2d3a62ea4a79a84b1abe003f8711b966ee4a30 not found: ID does not exist" containerID="40b31b5153e1e37510fdd3801b2d3a62ea4a79a84b1abe003f8711b966ee4a30"
May 25 12:16:40.226975 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:40.226937    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40b31b5153e1e37510fdd3801b2d3a62ea4a79a84b1abe003f8711b966ee4a30"} err="failed to get container status \"40b31b5153e1e37510fdd3801b2d3a62ea4a79a84b1abe003f8711b966ee4a30\": rpc error: code = NotFound desc = could not find container \"40b31b5153e1e37510fdd3801b2d3a62ea4a79a84b1abe003f8711b966ee4a30\": container with ID starting with 40b31b5153e1e37510fdd3801b2d3a62ea4a79a84b1abe003f8711b966ee4a30 not found: ID does not exist"
May 25 12:16:40.245779 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:40.245755    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c674b587d-t6lxh"]
May 25 12:16:40.251185 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:40.251165    2555 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6c674b587d-t6lxh"]
May 25 12:16:41.597991 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:16:41.597959    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c458cd01-59c9-4a50-8af7-0e43855047c9" path="/var/lib/kubelet/pods/c458cd01-59c9-4a50-8af7-0e43855047c9/volumes"
May 25 12:17:19.490018 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:19.489972    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30f9c156-1292-4927-9159-eba19cf76012-metrics-certs\") pod \"network-metrics-daemon-nwlcx\" (UID: \"30f9c156-1292-4927-9159-eba19cf76012\") " pod="openshift-multus/network-metrics-daemon-nwlcx"
May 25 12:17:19.492214 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:19.492191    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30f9c156-1292-4927-9159-eba19cf76012-metrics-certs\") pod \"network-metrics-daemon-nwlcx\" (UID: \"30f9c156-1292-4927-9159-eba19cf76012\") " pod="openshift-multus/network-metrics-daemon-nwlcx"
May 25 12:17:19.499161 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:19.499141    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nlm6z\""
May 25 12:17:19.508014 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:19.507996    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nwlcx"
May 25 12:17:19.621479 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:19.621448    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nwlcx"]
May 25 12:17:19.624376 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:17:19.624349    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f9c156_1292_4927_9159_eba19cf76012.slice/crio-6a7b217b2dae9ed64a0190a7ae109cd8f868c3d711fdbe742268abb28d1463c0 WatchSource:0}: Error finding container 6a7b217b2dae9ed64a0190a7ae109cd8f868c3d711fdbe742268abb28d1463c0: Status 404 returned error can't find the container with id 6a7b217b2dae9ed64a0190a7ae109cd8f868c3d711fdbe742268abb28d1463c0
May 25 12:17:20.327265 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:20.327221    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nwlcx" event={"ID":"30f9c156-1292-4927-9159-eba19cf76012","Type":"ContainerStarted","Data":"6a7b217b2dae9ed64a0190a7ae109cd8f868c3d711fdbe742268abb28d1463c0"}
May 25 12:17:21.335155 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:21.335114    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nwlcx" event={"ID":"30f9c156-1292-4927-9159-eba19cf76012","Type":"ContainerStarted","Data":"ce5c64f9b8cb407de272430afb1514a4f1668ec9fb2c63d275c680a51c33b876"}
May 25 12:17:21.335155 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:21.335154    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nwlcx" event={"ID":"30f9c156-1292-4927-9159-eba19cf76012","Type":"ContainerStarted","Data":"08bb14e1dfee68310662be0c4cfaede758d9406f321871cfd78f563faa374841"}
May 25 12:17:21.349709 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:21.349666    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-nwlcx" podStartSLOduration=253.24037618 podStartE2EDuration="4m14.349652843s" podCreationTimestamp="2026-05-25 12:13:07 +0000 UTC" firstStartedPulling="2026-05-25 12:17:19.626046912 +0000 UTC m=+252.586097295" lastFinishedPulling="2026-05-25 12:17:20.735323571 +0000 UTC m=+253.695373958" observedRunningTime="2026-05-25 12:17:21.347755532 +0000 UTC m=+254.307805935" watchObservedRunningTime="2026-05-25 12:17:21.349652843 +0000 UTC m=+254.309703239"
May 25 12:17:22.894702 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:22.894668    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-57b8d79fd4-cgzsk"]
May 25 12:17:22.895344 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:22.895323    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c458cd01-59c9-4a50-8af7-0e43855047c9" containerName="console"
May 25 12:17:22.895435 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:22.895346    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c458cd01-59c9-4a50-8af7-0e43855047c9" containerName="console"
May 25 12:17:22.895435 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:22.895363    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="904e6201-eb48-403d-822a-81f5ab186906" containerName="console"
May 25 12:17:22.895435 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:22.895371    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="904e6201-eb48-403d-822a-81f5ab186906" containerName="console"
May 25 12:17:22.895435 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:22.895430    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="c458cd01-59c9-4a50-8af7-0e43855047c9" containerName="console"
May 25 12:17:22.895630 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:22.895446    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="904e6201-eb48-403d-822a-81f5ab186906" containerName="console"
May 25 12:17:22.897417 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:22.897397    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-57b8d79fd4-cgzsk"
May 25 12:17:22.900288 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:22.900264    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
May 25 12:17:22.901556 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:22.901477    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
May 25 12:17:22.901692 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:22.900787    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
May 25 12:17:22.901692 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:22.900743    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
May 25 12:17:22.901914 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:22.901081    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-r8bpr\""
May 25 12:17:22.902080 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:22.902059    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
May 25 12:17:22.905782 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:22.905765    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
May 25 12:17:22.908704 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:22.908684    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-57b8d79fd4-cgzsk"]
May 25 12:17:23.015205 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:23.015171    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/77329766-b572-41f7-91cb-a0c23fb1cf81-metrics-client-ca\") pod \"telemeter-client-57b8d79fd4-cgzsk\" (UID: \"77329766-b572-41f7-91cb-a0c23fb1cf81\") " pod="openshift-monitoring/telemeter-client-57b8d79fd4-cgzsk"
May 25 12:17:23.015377 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:23.015211    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/77329766-b572-41f7-91cb-a0c23fb1cf81-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-57b8d79fd4-cgzsk\" (UID: \"77329766-b572-41f7-91cb-a0c23fb1cf81\") " pod="openshift-monitoring/telemeter-client-57b8d79fd4-cgzsk"
May 25 12:17:23.015377 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:23.015255    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/77329766-b572-41f7-91cb-a0c23fb1cf81-federate-client-tls\") pod \"telemeter-client-57b8d79fd4-cgzsk\" (UID: \"77329766-b572-41f7-91cb-a0c23fb1cf81\") " pod="openshift-monitoring/telemeter-client-57b8d79fd4-cgzsk"
May 25 12:17:23.015377 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:23.015274    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/77329766-b572-41f7-91cb-a0c23fb1cf81-secret-telemeter-client\") pod \"telemeter-client-57b8d79fd4-cgzsk\" (UID: \"77329766-b572-41f7-91cb-a0c23fb1cf81\") " pod="openshift-monitoring/telemeter-client-57b8d79fd4-cgzsk"
May 25 12:17:23.015377 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:23.015309    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/77329766-b572-41f7-91cb-a0c23fb1cf81-telemeter-client-tls\") pod \"telemeter-client-57b8d79fd4-cgzsk\" (UID: \"77329766-b572-41f7-91cb-a0c23fb1cf81\") " pod="openshift-monitoring/telemeter-client-57b8d79fd4-cgzsk"
May 25 12:17:23.015377 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:23.015326    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77329766-b572-41f7-91cb-a0c23fb1cf81-serving-certs-ca-bundle\") pod \"telemeter-client-57b8d79fd4-cgzsk\" (UID: \"77329766-b572-41f7-91cb-a0c23fb1cf81\") " pod="openshift-monitoring/telemeter-client-57b8d79fd4-cgzsk"
May 25 12:17:23.015377 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:23.015362    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77329766-b572-41f7-91cb-a0c23fb1cf81-telemeter-trusted-ca-bundle\") pod \"telemeter-client-57b8d79fd4-cgzsk\" (UID: \"77329766-b572-41f7-91cb-a0c23fb1cf81\") " pod="openshift-monitoring/telemeter-client-57b8d79fd4-cgzsk"
May 25 12:17:23.015571 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:23.015394    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knbg9\" (UniqueName: \"kubernetes.io/projected/77329766-b572-41f7-91cb-a0c23fb1cf81-kube-api-access-knbg9\") pod \"telemeter-client-57b8d79fd4-cgzsk\" (UID: \"77329766-b572-41f7-91cb-a0c23fb1cf81\") " pod="openshift-monitoring/telemeter-client-57b8d79fd4-cgzsk"
May 25 12:17:23.116561 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:23.116528    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-knbg9\" (UniqueName: \"kubernetes.io/projected/77329766-b572-41f7-91cb-a0c23fb1cf81-kube-api-access-knbg9\") pod \"telemeter-client-57b8d79fd4-cgzsk\" (UID: \"77329766-b572-41f7-91cb-a0c23fb1cf81\") " pod="openshift-monitoring/telemeter-client-57b8d79fd4-cgzsk"
May 25 12:17:23.116757 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:23.116569    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/77329766-b572-41f7-91cb-a0c23fb1cf81-metrics-client-ca\") pod \"telemeter-client-57b8d79fd4-cgzsk\" (UID: \"77329766-b572-41f7-91cb-a0c23fb1cf81\") " pod="openshift-monitoring/telemeter-client-57b8d79fd4-cgzsk"
May 25 12:17:23.116757 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:23.116606    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/77329766-b572-41f7-91cb-a0c23fb1cf81-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-57b8d79fd4-cgzsk\" (UID: \"77329766-b572-41f7-91cb-a0c23fb1cf81\") " pod="openshift-monitoring/telemeter-client-57b8d79fd4-cgzsk"
May 25 12:17:23.116757 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:23.116637    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/77329766-b572-41f7-91cb-a0c23fb1cf81-federate-client-tls\") pod \"telemeter-client-57b8d79fd4-cgzsk\" (UID: \"77329766-b572-41f7-91cb-a0c23fb1cf81\") " pod="openshift-monitoring/telemeter-client-57b8d79fd4-cgzsk"
May 25 12:17:23.116757 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:23.116686    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/77329766-b572-41f7-91cb-a0c23fb1cf81-secret-telemeter-client\") pod \"telemeter-client-57b8d79fd4-cgzsk\" (UID: \"77329766-b572-41f7-91cb-a0c23fb1cf81\") " pod="openshift-monitoring/telemeter-client-57b8d79fd4-cgzsk"
May 25 12:17:23.116757 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:23.116719    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/77329766-b572-41f7-91cb-a0c23fb1cf81-telemeter-client-tls\") pod \"telemeter-client-57b8d79fd4-cgzsk\" (UID: \"77329766-b572-41f7-91cb-a0c23fb1cf81\") " pod="openshift-monitoring/telemeter-client-57b8d79fd4-cgzsk"
May 25 12:17:23.117195 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:23.117158    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77329766-b572-41f7-91cb-a0c23fb1cf81-serving-certs-ca-bundle\") pod \"telemeter-client-57b8d79fd4-cgzsk\" (UID: \"77329766-b572-41f7-91cb-a0c23fb1cf81\") " pod="openshift-monitoring/telemeter-client-57b8d79fd4-cgzsk"
May 25 12:17:23.117307 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:23.117231    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77329766-b572-41f7-91cb-a0c23fb1cf81-telemeter-trusted-ca-bundle\") pod \"telemeter-client-57b8d79fd4-cgzsk\" (UID: \"77329766-b572-41f7-91cb-a0c23fb1cf81\") " pod="openshift-monitoring/telemeter-client-57b8d79fd4-cgzsk"
May 25 12:17:23.117513 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:23.117481    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/77329766-b572-41f7-91cb-a0c23fb1cf81-metrics-client-ca\") pod \"telemeter-client-57b8d79fd4-cgzsk\" (UID: \"77329766-b572-41f7-91cb-a0c23fb1cf81\") " pod="openshift-monitoring/telemeter-client-57b8d79fd4-cgzsk"
May 25 12:17:23.117854 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:23.117829    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77329766-b572-41f7-91cb-a0c23fb1cf81-serving-certs-ca-bundle\") pod \"telemeter-client-57b8d79fd4-cgzsk\" (UID: \"77329766-b572-41f7-91cb-a0c23fb1cf81\") " pod="openshift-monitoring/telemeter-client-57b8d79fd4-cgzsk"
May 25 12:17:23.117959 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:23.117940    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77329766-b572-41f7-91cb-a0c23fb1cf81-telemeter-trusted-ca-bundle\") pod \"telemeter-client-57b8d79fd4-cgzsk\" (UID: \"77329766-b572-41f7-91cb-a0c23fb1cf81\") " pod="openshift-monitoring/telemeter-client-57b8d79fd4-cgzsk"
May 25 12:17:23.119270 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:23.119241    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/77329766-b572-41f7-91cb-a0c23fb1cf81-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-57b8d79fd4-cgzsk\" (UID: \"77329766-b572-41f7-91cb-a0c23fb1cf81\") " pod="openshift-monitoring/telemeter-client-57b8d79fd4-cgzsk"
May 25 12:17:23.119397 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:23.119373    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/77329766-b572-41f7-91cb-a0c23fb1cf81-telemeter-client-tls\") pod \"telemeter-client-57b8d79fd4-cgzsk\" (UID: \"77329766-b572-41f7-91cb-a0c23fb1cf81\") " pod="openshift-monitoring/telemeter-client-57b8d79fd4-cgzsk"
May 25 12:17:23.119526 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:23.119509    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/77329766-b572-41f7-91cb-a0c23fb1cf81-secret-telemeter-client\") pod \"telemeter-client-57b8d79fd4-cgzsk\" (UID: \"77329766-b572-41f7-91cb-a0c23fb1cf81\") " pod="openshift-monitoring/telemeter-client-57b8d79fd4-cgzsk"
May 25 12:17:23.119561 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:23.119526    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/77329766-b572-41f7-91cb-a0c23fb1cf81-federate-client-tls\") pod \"telemeter-client-57b8d79fd4-cgzsk\" (UID: \"77329766-b572-41f7-91cb-a0c23fb1cf81\") " pod="openshift-monitoring/telemeter-client-57b8d79fd4-cgzsk"
May 25 12:17:23.123499 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:23.123476    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knbg9\" (UniqueName: \"kubernetes.io/projected/77329766-b572-41f7-91cb-a0c23fb1cf81-kube-api-access-knbg9\") pod \"telemeter-client-57b8d79fd4-cgzsk\" (UID: \"77329766-b572-41f7-91cb-a0c23fb1cf81\") " pod="openshift-monitoring/telemeter-client-57b8d79fd4-cgzsk"
May 25 12:17:23.208491 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:23.208460    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-57b8d79fd4-cgzsk"
May 25 12:17:23.343791 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:23.343768    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-57b8d79fd4-cgzsk"]
May 25 12:17:23.349802 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:17:23.348778    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77329766_b572_41f7_91cb_a0c23fb1cf81.slice/crio-8d2749ba4a1a9263d2808f2ff09bd2445c2657f3c22968f930b17dbd79abf1d3 WatchSource:0}: Error finding container 8d2749ba4a1a9263d2808f2ff09bd2445c2657f3c22968f930b17dbd79abf1d3: Status 404 returned error can't find the container with id 8d2749ba4a1a9263d2808f2ff09bd2445c2657f3c22968f930b17dbd79abf1d3
May 25 12:17:24.343237 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:24.343200    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-57b8d79fd4-cgzsk" event={"ID":"77329766-b572-41f7-91cb-a0c23fb1cf81","Type":"ContainerStarted","Data":"8d2749ba4a1a9263d2808f2ff09bd2445c2657f3c22968f930b17dbd79abf1d3"}
May 25 12:17:26.350206 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:26.350171    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-57b8d79fd4-cgzsk" event={"ID":"77329766-b572-41f7-91cb-a0c23fb1cf81","Type":"ContainerStarted","Data":"ab096859d96d4fec46df8aa4f8a36e2bf36e10a334f27eee4c48302b30ea3c20"}
May 25 12:17:27.354176 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:27.354098    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-57b8d79fd4-cgzsk" event={"ID":"77329766-b572-41f7-91cb-a0c23fb1cf81","Type":"ContainerStarted","Data":"99c38d52157ebce836996206dcc65c95043286e25a8d8d2e504dacd2ff513a05"}
May 25 12:17:27.354176 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:27.354139    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-57b8d79fd4-cgzsk" event={"ID":"77329766-b572-41f7-91cb-a0c23fb1cf81","Type":"ContainerStarted","Data":"ce83770cdb687f4548acaf3a9f20fa3fe866fb28707cc12adff3b64eb0fde2b0"}
May 25 12:17:27.374082 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:27.374037    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-57b8d79fd4-cgzsk" podStartSLOduration=2.279289189 podStartE2EDuration="5.374022029s" podCreationTimestamp="2026-05-25 12:17:22 +0000 UTC" firstStartedPulling="2026-05-25 12:17:23.351170289 +0000 UTC m=+256.311220673" lastFinishedPulling="2026-05-25 12:17:26.445903127 +0000 UTC m=+259.405953513" observedRunningTime="2026-05-25 12:17:27.372836857 +0000 UTC m=+260.332887261" watchObservedRunningTime="2026-05-25 12:17:27.374022029 +0000 UTC m=+260.334072434"
May 25 12:17:27.932544 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:27.932501    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-d7bd68cf-r7wdn"]
May 25 12:17:27.934928 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:27.934900    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d7bd68cf-r7wdn"
May 25 12:17:27.938251 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:27.938212    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
May 25 12:17:27.938409 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:27.938274    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
May 25 12:17:27.938409 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:27.938229    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
May 25 12:17:27.938409 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:27.938344    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
May 25 12:17:27.938574 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:27.938436    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-wc5r9\""
May 25 12:17:27.938695 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:27.938675    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
May 25 12:17:27.943559 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:27.943536    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
May 25 12:17:27.946831 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:27.946809    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d7bd68cf-r7wdn"]
May 25 12:17:28.057368 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:28.057330    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9c442196-30d0-4274-936c-baad6f5e7a97-console-config\") pod \"console-d7bd68cf-r7wdn\" (UID: \"9c442196-30d0-4274-936c-baad6f5e7a97\") " pod="openshift-console/console-d7bd68cf-r7wdn"
May 25 12:17:28.057599 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:28.057392    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bglv\" (UniqueName: \"kubernetes.io/projected/9c442196-30d0-4274-936c-baad6f5e7a97-kube-api-access-9bglv\") pod \"console-d7bd68cf-r7wdn\" (UID: \"9c442196-30d0-4274-936c-baad6f5e7a97\") " pod="openshift-console/console-d7bd68cf-r7wdn"
May 25 12:17:28.057599 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:28.057464    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9c442196-30d0-4274-936c-baad6f5e7a97-service-ca\") pod \"console-d7bd68cf-r7wdn\" (UID: \"9c442196-30d0-4274-936c-baad6f5e7a97\") " pod="openshift-console/console-d7bd68cf-r7wdn"
May 25 12:17:28.057599 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:28.057519    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9c442196-30d0-4274-936c-baad6f5e7a97-console-oauth-config\") pod \"console-d7bd68cf-r7wdn\" (UID: \"9c442196-30d0-4274-936c-baad6f5e7a97\") " pod="openshift-console/console-d7bd68cf-r7wdn"
May 25 12:17:28.057599 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:28.057554    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9c442196-30d0-4274-936c-baad6f5e7a97-console-serving-cert\") pod \"console-d7bd68cf-r7wdn\" (UID: \"9c442196-30d0-4274-936c-baad6f5e7a97\") " pod="openshift-console/console-d7bd68cf-r7wdn"
May 25 12:17:28.057599 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:28.057574    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c442196-30d0-4274-936c-baad6f5e7a97-trusted-ca-bundle\") pod \"console-d7bd68cf-r7wdn\" (UID: \"9c442196-30d0-4274-936c-baad6f5e7a97\") " pod="openshift-console/console-d7bd68cf-r7wdn"
May 25 12:17:28.057881 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:28.057623    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9c442196-30d0-4274-936c-baad6f5e7a97-oauth-serving-cert\") pod \"console-d7bd68cf-r7wdn\" (UID: \"9c442196-30d0-4274-936c-baad6f5e7a97\") " pod="openshift-console/console-d7bd68cf-r7wdn"
May 25 12:17:28.158835 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:28.158795    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9c442196-30d0-4274-936c-baad6f5e7a97-oauth-serving-cert\") pod \"console-d7bd68cf-r7wdn\" (UID: \"9c442196-30d0-4274-936c-baad6f5e7a97\") " pod="openshift-console/console-d7bd68cf-r7wdn"
May 25 12:17:28.159010 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:28.158876    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9c442196-30d0-4274-936c-baad6f5e7a97-console-config\") pod \"console-d7bd68cf-r7wdn\" (UID: \"9c442196-30d0-4274-936c-baad6f5e7a97\") " pod="openshift-console/console-d7bd68cf-r7wdn"
May 25 12:17:28.159010 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:28.158916    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9bglv\" (UniqueName: \"kubernetes.io/projected/9c442196-30d0-4274-936c-baad6f5e7a97-kube-api-access-9bglv\") pod \"console-d7bd68cf-r7wdn\" (UID: \"9c442196-30d0-4274-936c-baad6f5e7a97\") " pod="openshift-console/console-d7bd68cf-r7wdn"
May 25 12:17:28.159010 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:28.158940    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9c442196-30d0-4274-936c-baad6f5e7a97-service-ca\") pod \"console-d7bd68cf-r7wdn\" (UID: \"9c442196-30d0-4274-936c-baad6f5e7a97\") " pod="openshift-console/console-d7bd68cf-r7wdn"
May 25 12:17:28.159010 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:28.158983    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9c442196-30d0-4274-936c-baad6f5e7a97-console-oauth-config\") pod \"console-d7bd68cf-r7wdn\" (UID: \"9c442196-30d0-4274-936c-baad6f5e7a97\") " pod="openshift-console/console-d7bd68cf-r7wdn"
May 25 12:17:28.159010 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:28.159004    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9c442196-30d0-4274-936c-baad6f5e7a97-console-serving-cert\") pod \"console-d7bd68cf-r7wdn\" (UID: \"9c442196-30d0-4274-936c-baad6f5e7a97\") " pod="openshift-console/console-d7bd68cf-r7wdn"
May 25 12:17:28.159271 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:28.159023    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c442196-30d0-4274-936c-baad6f5e7a97-trusted-ca-bundle\") pod \"console-d7bd68cf-r7wdn\" (UID: \"9c442196-30d0-4274-936c-baad6f5e7a97\") " pod="openshift-console/console-d7bd68cf-r7wdn"
May 25 12:17:28.159731 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:28.159698    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9c442196-30d0-4274-936c-baad6f5e7a97-oauth-serving-cert\") pod \"console-d7bd68cf-r7wdn\" (UID: \"9c442196-30d0-4274-936c-baad6f5e7a97\") " pod="openshift-console/console-d7bd68cf-r7wdn"
May 25 12:17:28.159861 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:28.159753    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9c442196-30d0-4274-936c-baad6f5e7a97-console-config\") pod \"console-d7bd68cf-r7wdn\" (UID: \"9c442196-30d0-4274-936c-baad6f5e7a97\") " pod="openshift-console/console-d7bd68cf-r7wdn"
May 25 12:17:28.160061 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:28.160041    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c442196-30d0-4274-936c-baad6f5e7a97-trusted-ca-bundle\") pod \"console-d7bd68cf-r7wdn\" (UID: \"9c442196-30d0-4274-936c-baad6f5e7a97\") " pod="openshift-console/console-d7bd68cf-r7wdn"
May 25 12:17:28.160129 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:28.160045    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9c442196-30d0-4274-936c-baad6f5e7a97-service-ca\") pod \"console-d7bd68cf-r7wdn\" (UID: \"9c442196-30d0-4274-936c-baad6f5e7a97\") " pod="openshift-console/console-d7bd68cf-r7wdn"
May 25 12:17:28.161448 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:28.161427    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9c442196-30d0-4274-936c-baad6f5e7a97-console-oauth-config\") pod \"console-d7bd68cf-r7wdn\" (UID: \"9c442196-30d0-4274-936c-baad6f5e7a97\") " pod="openshift-console/console-d7bd68cf-r7wdn"
May 25 12:17:28.161553 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:28.161507    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9c442196-30d0-4274-936c-baad6f5e7a97-console-serving-cert\") pod \"console-d7bd68cf-r7wdn\" (UID: \"9c442196-30d0-4274-936c-baad6f5e7a97\") " pod="openshift-console/console-d7bd68cf-r7wdn"
May 25 12:17:28.166018 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:28.165988    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bglv\" (UniqueName: \"kubernetes.io/projected/9c442196-30d0-4274-936c-baad6f5e7a97-kube-api-access-9bglv\") pod \"console-d7bd68cf-r7wdn\" (UID: \"9c442196-30d0-4274-936c-baad6f5e7a97\") " pod="openshift-console/console-d7bd68cf-r7wdn"
May 25 12:17:28.248029 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:28.247955    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d7bd68cf-r7wdn"
May 25 12:17:28.363342 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:28.363321    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d7bd68cf-r7wdn"]
May 25 12:17:28.365508 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:17:28.365480    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c442196_30d0_4274_936c_baad6f5e7a97.slice/crio-28ff1bbee1e959c2764834e1c27d5a2ce79647ede46fb8b976867e2e4af1413b WatchSource:0}: Error finding container 28ff1bbee1e959c2764834e1c27d5a2ce79647ede46fb8b976867e2e4af1413b: Status 404 returned error can't find the container with id 28ff1bbee1e959c2764834e1c27d5a2ce79647ede46fb8b976867e2e4af1413b
May 25 12:17:29.360383 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:29.360345    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d7bd68cf-r7wdn" event={"ID":"9c442196-30d0-4274-936c-baad6f5e7a97","Type":"ContainerStarted","Data":"33c860c0baf5170edeb04423283fa7fd6bdf047f843b92d08af7aee63a3899be"}
May 25 12:17:29.360383 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:29.360381    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d7bd68cf-r7wdn" event={"ID":"9c442196-30d0-4274-936c-baad6f5e7a97","Type":"ContainerStarted","Data":"28ff1bbee1e959c2764834e1c27d5a2ce79647ede46fb8b976867e2e4af1413b"}
May 25 12:17:29.378244 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:29.378201    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-d7bd68cf-r7wdn" podStartSLOduration=2.378188651 podStartE2EDuration="2.378188651s" podCreationTimestamp="2026-05-25 12:17:27 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:17:29.376995682 +0000 UTC m=+262.337046084" watchObservedRunningTime="2026-05-25 12:17:29.378188651 +0000 UTC m=+262.338239055"
May 25 12:17:38.248681 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:38.248627    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-d7bd68cf-r7wdn"
May 25 12:17:38.249123 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:38.248702    2555 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-d7bd68cf-r7wdn"
May 25 12:17:38.253332 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:38.253313    2555 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-d7bd68cf-r7wdn"
May 25 12:17:38.387631 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:17:38.387603    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-d7bd68cf-r7wdn"
May 25 12:18:07.482521 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:18:07.482490    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-2tqtm_00e6f79c-8f05-4330-aa16-1ada2ae9e9cc/console-operator/1.log"
May 25 12:18:07.483210 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:18:07.483175    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-2tqtm_00e6f79c-8f05-4330-aa16-1ada2ae9e9cc/console-operator/1.log"
May 25 12:18:07.494544 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:18:07.494526    2555 kubelet.go:1628] "Image garbage collection succeeded"
May 25 12:18:34.658707 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:18:34.658666    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-8469d49f4b-ppb42"]
May 25 12:18:34.662135 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:18:34.662111    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8469d49f4b-ppb42"
May 25 12:18:34.676771 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:18:34.673547    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8469d49f4b-ppb42"]
May 25 12:18:34.743685 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:18:34.743628    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/deb97e2d-509e-40f5-bb42-611cb81f5b18-trusted-ca-bundle\") pod \"console-8469d49f4b-ppb42\" (UID: \"deb97e2d-509e-40f5-bb42-611cb81f5b18\") " pod="openshift-console/console-8469d49f4b-ppb42"
May 25 12:18:34.743880 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:18:34.743694    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/deb97e2d-509e-40f5-bb42-611cb81f5b18-console-config\") pod \"console-8469d49f4b-ppb42\" (UID: \"deb97e2d-509e-40f5-bb42-611cb81f5b18\") " pod="openshift-console/console-8469d49f4b-ppb42"
May 25 12:18:34.743880 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:18:34.743722    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/deb97e2d-509e-40f5-bb42-611cb81f5b18-service-ca\") pod \"console-8469d49f4b-ppb42\" (UID: \"deb97e2d-509e-40f5-bb42-611cb81f5b18\") " pod="openshift-console/console-8469d49f4b-ppb42"
May 25 12:18:34.743880 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:18:34.743737    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/deb97e2d-509e-40f5-bb42-611cb81f5b18-oauth-serving-cert\") pod \"console-8469d49f4b-ppb42\" (UID: \"deb97e2d-509e-40f5-bb42-611cb81f5b18\") " pod="openshift-console/console-8469d49f4b-ppb42"
May 25 12:18:34.743880 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:18:34.743755    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/deb97e2d-509e-40f5-bb42-611cb81f5b18-console-serving-cert\") pod \"console-8469d49f4b-ppb42\" (UID: \"deb97e2d-509e-40f5-bb42-611cb81f5b18\") " pod="openshift-console/console-8469d49f4b-ppb42"
May 25 12:18:34.743880 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:18:34.743817    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt4g6\" (UniqueName: \"kubernetes.io/projected/deb97e2d-509e-40f5-bb42-611cb81f5b18-kube-api-access-rt4g6\") pod \"console-8469d49f4b-ppb42\" (UID: \"deb97e2d-509e-40f5-bb42-611cb81f5b18\") " pod="openshift-console/console-8469d49f4b-ppb42"
May 25 12:18:34.744134 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:18:34.743914    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/deb97e2d-509e-40f5-bb42-611cb81f5b18-console-oauth-config\") pod \"console-8469d49f4b-ppb42\" (UID: \"deb97e2d-509e-40f5-bb42-611cb81f5b18\") " pod="openshift-console/console-8469d49f4b-ppb42"
May 25 12:18:34.844850 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:18:34.844811    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/deb97e2d-509e-40f5-bb42-611cb81f5b18-trusted-ca-bundle\") pod \"console-8469d49f4b-ppb42\" (UID: \"deb97e2d-509e-40f5-bb42-611cb81f5b18\") " pod="openshift-console/console-8469d49f4b-ppb42"
May 25 12:18:34.844850 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:18:34.844858    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/deb97e2d-509e-40f5-bb42-611cb81f5b18-console-config\") pod \"console-8469d49f4b-ppb42\" (UID: \"deb97e2d-509e-40f5-bb42-611cb81f5b18\") " pod="openshift-console/console-8469d49f4b-ppb42"
May 25 12:18:34.845106 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:18:34.844885    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/deb97e2d-509e-40f5-bb42-611cb81f5b18-service-ca\") pod \"console-8469d49f4b-ppb42\" (UID: \"deb97e2d-509e-40f5-bb42-611cb81f5b18\") " pod="openshift-console/console-8469d49f4b-ppb42"
May 25 12:18:34.845106 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:18:34.845012    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/deb97e2d-509e-40f5-bb42-611cb81f5b18-oauth-serving-cert\") pod \"console-8469d49f4b-ppb42\" (UID: \"deb97e2d-509e-40f5-bb42-611cb81f5b18\") " pod="openshift-console/console-8469d49f4b-ppb42"
May 25 12:18:34.845106 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:18:34.845071    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/deb97e2d-509e-40f5-bb42-611cb81f5b18-console-serving-cert\") pod \"console-8469d49f4b-ppb42\" (UID: \"deb97e2d-509e-40f5-bb42-611cb81f5b18\") " pod="openshift-console/console-8469d49f4b-ppb42"
May 25 12:18:34.845106 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:18:34.845104    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rt4g6\" (UniqueName: \"kubernetes.io/projected/deb97e2d-509e-40f5-bb42-611cb81f5b18-kube-api-access-rt4g6\") pod \"console-8469d49f4b-ppb42\" (UID: \"deb97e2d-509e-40f5-bb42-611cb81f5b18\") " pod="openshift-console/console-8469d49f4b-ppb42"
May 25 12:18:34.845279 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:18:34.845149    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/deb97e2d-509e-40f5-bb42-611cb81f5b18-console-oauth-config\") pod \"console-8469d49f4b-ppb42\" (UID: \"deb97e2d-509e-40f5-bb42-611cb81f5b18\") " pod="openshift-console/console-8469d49f4b-ppb42"
May 25 12:18:34.845660 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:18:34.845616    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/deb97e2d-509e-40f5-bb42-611cb81f5b18-service-ca\") pod \"console-8469d49f4b-ppb42\" (UID: \"deb97e2d-509e-40f5-bb42-611cb81f5b18\") " pod="openshift-console/console-8469d49f4b-ppb42"
May 25 12:18:34.845782 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:18:34.845734    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/deb97e2d-509e-40f5-bb42-611cb81f5b18-console-config\") pod \"console-8469d49f4b-ppb42\" (UID: \"deb97e2d-509e-40f5-bb42-611cb81f5b18\") " pod="openshift-console/console-8469d49f4b-ppb42"
May 25 12:18:34.845832 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:18:34.845795    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/deb97e2d-509e-40f5-bb42-611cb81f5b18-oauth-serving-cert\") pod \"console-8469d49f4b-ppb42\" (UID: \"deb97e2d-509e-40f5-bb42-611cb81f5b18\") " pod="openshift-console/console-8469d49f4b-ppb42"
May 25 12:18:34.845868 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:18:34.845803    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/deb97e2d-509e-40f5-bb42-611cb81f5b18-trusted-ca-bundle\") pod \"console-8469d49f4b-ppb42\" (UID: \"deb97e2d-509e-40f5-bb42-611cb81f5b18\") " pod="openshift-console/console-8469d49f4b-ppb42"
May 25 12:18:34.847694 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:18:34.847666    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/deb97e2d-509e-40f5-bb42-611cb81f5b18-console-serving-cert\") pod \"console-8469d49f4b-ppb42\" (UID: \"deb97e2d-509e-40f5-bb42-611cb81f5b18\") " pod="openshift-console/console-8469d49f4b-ppb42"
May 25 12:18:34.847779 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:18:34.847694    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/deb97e2d-509e-40f5-bb42-611cb81f5b18-console-oauth-config\") pod \"console-8469d49f4b-ppb42\" (UID: \"deb97e2d-509e-40f5-bb42-611cb81f5b18\") " pod="openshift-console/console-8469d49f4b-ppb42"
May 25 12:18:34.853067 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:18:34.853039    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt4g6\" (UniqueName: \"kubernetes.io/projected/deb97e2d-509e-40f5-bb42-611cb81f5b18-kube-api-access-rt4g6\") pod \"console-8469d49f4b-ppb42\" (UID: \"deb97e2d-509e-40f5-bb42-611cb81f5b18\") " pod="openshift-console/console-8469d49f4b-ppb42"
May 25 12:18:34.979508 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:18:34.979464    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8469d49f4b-ppb42"
May 25 12:18:35.108906 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:18:35.108863    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8469d49f4b-ppb42"]
May 25 12:18:35.112057 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:18:35.112028    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeb97e2d_509e_40f5_bb42_611cb81f5b18.slice/crio-09a859a344da9f8a3617948c0466ba324b1606e17bdfe2dfe9940f042d9b38b3 WatchSource:0}: Error finding container 09a859a344da9f8a3617948c0466ba324b1606e17bdfe2dfe9940f042d9b38b3: Status 404 returned error can't find the container with id 09a859a344da9f8a3617948c0466ba324b1606e17bdfe2dfe9940f042d9b38b3
May 25 12:18:35.113777 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:18:35.113759    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 12:18:35.536526 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:18:35.536490    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8469d49f4b-ppb42" event={"ID":"deb97e2d-509e-40f5-bb42-611cb81f5b18","Type":"ContainerStarted","Data":"8ec2914c30a062149d65d86b159b6494518f37f27805513c68bbf72ca139bc50"}
May 25 12:18:35.536526 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:18:35.536527    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8469d49f4b-ppb42" event={"ID":"deb97e2d-509e-40f5-bb42-611cb81f5b18","Type":"ContainerStarted","Data":"09a859a344da9f8a3617948c0466ba324b1606e17bdfe2dfe9940f042d9b38b3"}
May 25 12:18:35.552324 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:18:35.552265    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-8469d49f4b-ppb42" podStartSLOduration=1.552248656 podStartE2EDuration="1.552248656s" podCreationTimestamp="2026-05-25 12:18:34 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:18:35.550835232 +0000 UTC m=+328.510885655" watchObservedRunningTime="2026-05-25 12:18:35.552248656 +0000 UTC m=+328.512299061"
May 25 12:18:44.980012 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:18:44.979971    2555 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8469d49f4b-ppb42"
May 25 12:18:44.980453 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:18:44.980040    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8469d49f4b-ppb42"
May 25 12:18:44.984680 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:18:44.984660    2555 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8469d49f4b-ppb42"
May 25 12:18:45.569420 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:18:45.569389    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8469d49f4b-ppb42"
May 25 12:18:45.611999 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:18:45.611968    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d7bd68cf-r7wdn"]
May 25 12:19:10.632055 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:19:10.631959    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-d7bd68cf-r7wdn" podUID="9c442196-30d0-4274-936c-baad6f5e7a97" containerName="console" containerID="cri-o://33c860c0baf5170edeb04423283fa7fd6bdf047f843b92d08af7aee63a3899be" gracePeriod=15
May 25 12:19:10.872568 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:19:10.872547    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-d7bd68cf-r7wdn_9c442196-30d0-4274-936c-baad6f5e7a97/console/0.log"
May 25 12:19:10.872678 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:19:10.872612    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d7bd68cf-r7wdn"
May 25 12:19:11.007571 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:19:11.007548    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9c442196-30d0-4274-936c-baad6f5e7a97-service-ca\") pod \"9c442196-30d0-4274-936c-baad6f5e7a97\" (UID: \"9c442196-30d0-4274-936c-baad6f5e7a97\") "
May 25 12:19:11.007761 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:19:11.007579    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9c442196-30d0-4274-936c-baad6f5e7a97-oauth-serving-cert\") pod \"9c442196-30d0-4274-936c-baad6f5e7a97\" (UID: \"9c442196-30d0-4274-936c-baad6f5e7a97\") "
May 25 12:19:11.007761 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:19:11.007603    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c442196-30d0-4274-936c-baad6f5e7a97-trusted-ca-bundle\") pod \"9c442196-30d0-4274-936c-baad6f5e7a97\" (UID: \"9c442196-30d0-4274-936c-baad6f5e7a97\") "
May 25 12:19:11.007761 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:19:11.007620    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9c442196-30d0-4274-936c-baad6f5e7a97-console-oauth-config\") pod \"9c442196-30d0-4274-936c-baad6f5e7a97\" (UID: \"9c442196-30d0-4274-936c-baad6f5e7a97\") "
May 25 12:19:11.007761 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:19:11.007637    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9bglv\" (UniqueName: \"kubernetes.io/projected/9c442196-30d0-4274-936c-baad6f5e7a97-kube-api-access-9bglv\") pod \"9c442196-30d0-4274-936c-baad6f5e7a97\" (UID: \"9c442196-30d0-4274-936c-baad6f5e7a97\") "
May 25 12:19:11.007761 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:19:11.007711    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9c442196-30d0-4274-936c-baad6f5e7a97-console-config\") pod \"9c442196-30d0-4274-936c-baad6f5e7a97\" (UID: \"9c442196-30d0-4274-936c-baad6f5e7a97\") "
May 25 12:19:11.007761 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:19:11.007733    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9c442196-30d0-4274-936c-baad6f5e7a97-console-serving-cert\") pod \"9c442196-30d0-4274-936c-baad6f5e7a97\" (UID: \"9c442196-30d0-4274-936c-baad6f5e7a97\") "
May 25 12:19:11.008070 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:19:11.007958    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c442196-30d0-4274-936c-baad6f5e7a97-service-ca" (OuterVolumeSpecName: "service-ca") pod "9c442196-30d0-4274-936c-baad6f5e7a97" (UID: "9c442196-30d0-4274-936c-baad6f5e7a97"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:19:11.008202 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:19:11.008172    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c442196-30d0-4274-936c-baad6f5e7a97-console-config" (OuterVolumeSpecName: "console-config") pod "9c442196-30d0-4274-936c-baad6f5e7a97" (UID: "9c442196-30d0-4274-936c-baad6f5e7a97"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:19:11.008280 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:19:11.008177    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c442196-30d0-4274-936c-baad6f5e7a97-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9c442196-30d0-4274-936c-baad6f5e7a97" (UID: "9c442196-30d0-4274-936c-baad6f5e7a97"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:19:11.008280 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:19:11.008235    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c442196-30d0-4274-936c-baad6f5e7a97-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "9c442196-30d0-4274-936c-baad6f5e7a97" (UID: "9c442196-30d0-4274-936c-baad6f5e7a97"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:19:11.009912 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:19:11.009883    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c442196-30d0-4274-936c-baad6f5e7a97-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9c442196-30d0-4274-936c-baad6f5e7a97" (UID: "9c442196-30d0-4274-936c-baad6f5e7a97"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:19:11.009912 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:19:11.009899    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c442196-30d0-4274-936c-baad6f5e7a97-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9c442196-30d0-4274-936c-baad6f5e7a97" (UID: "9c442196-30d0-4274-936c-baad6f5e7a97"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:19:11.010039 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:19:11.009935    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c442196-30d0-4274-936c-baad6f5e7a97-kube-api-access-9bglv" (OuterVolumeSpecName: "kube-api-access-9bglv") pod "9c442196-30d0-4274-936c-baad6f5e7a97" (UID: "9c442196-30d0-4274-936c-baad6f5e7a97"). InnerVolumeSpecName "kube-api-access-9bglv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:19:11.108425 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:19:11.108395    2555 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9c442196-30d0-4274-936c-baad6f5e7a97-console-oauth-config\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:19:11.108425 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:19:11.108426    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bglv\" (UniqueName: \"kubernetes.io/projected/9c442196-30d0-4274-936c-baad6f5e7a97-kube-api-access-9bglv\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:19:11.108604 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:19:11.108437    2555 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9c442196-30d0-4274-936c-baad6f5e7a97-console-config\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:19:11.108604 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:19:11.108447    2555 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9c442196-30d0-4274-936c-baad6f5e7a97-console-serving-cert\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:19:11.108604 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:19:11.108456    2555 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9c442196-30d0-4274-936c-baad6f5e7a97-service-ca\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:19:11.108604 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:19:11.108464    2555 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9c442196-30d0-4274-936c-baad6f5e7a97-oauth-serving-cert\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:19:11.108604 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:19:11.108472    2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c442196-30d0-4274-936c-baad6f5e7a97-trusted-ca-bundle\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:19:11.631766 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:19:11.631743    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-d7bd68cf-r7wdn_9c442196-30d0-4274-936c-baad6f5e7a97/console/0.log"
May 25 12:19:11.631881 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:19:11.631783    2555 generic.go:358] "Generic (PLEG): container finished" podID="9c442196-30d0-4274-936c-baad6f5e7a97" containerID="33c860c0baf5170edeb04423283fa7fd6bdf047f843b92d08af7aee63a3899be" exitCode=2
May 25 12:19:11.631881 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:19:11.631868    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d7bd68cf-r7wdn" event={"ID":"9c442196-30d0-4274-936c-baad6f5e7a97","Type":"ContainerDied","Data":"33c860c0baf5170edeb04423283fa7fd6bdf047f843b92d08af7aee63a3899be"}
May 25 12:19:11.631960 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:19:11.631883    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d7bd68cf-r7wdn"
May 25 12:19:11.631960 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:19:11.631902    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d7bd68cf-r7wdn" event={"ID":"9c442196-30d0-4274-936c-baad6f5e7a97","Type":"ContainerDied","Data":"28ff1bbee1e959c2764834e1c27d5a2ce79647ede46fb8b976867e2e4af1413b"}
May 25 12:19:11.631960 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:19:11.631918    2555 scope.go:117] "RemoveContainer" containerID="33c860c0baf5170edeb04423283fa7fd6bdf047f843b92d08af7aee63a3899be"
May 25 12:19:11.639363 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:19:11.639338    2555 scope.go:117] "RemoveContainer" containerID="33c860c0baf5170edeb04423283fa7fd6bdf047f843b92d08af7aee63a3899be"
May 25 12:19:11.639632 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:19:11.639570    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33c860c0baf5170edeb04423283fa7fd6bdf047f843b92d08af7aee63a3899be\": container with ID starting with 33c860c0baf5170edeb04423283fa7fd6bdf047f843b92d08af7aee63a3899be not found: ID does not exist" containerID="33c860c0baf5170edeb04423283fa7fd6bdf047f843b92d08af7aee63a3899be"
May 25 12:19:11.639632 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:19:11.639593    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33c860c0baf5170edeb04423283fa7fd6bdf047f843b92d08af7aee63a3899be"} err="failed to get container status \"33c860c0baf5170edeb04423283fa7fd6bdf047f843b92d08af7aee63a3899be\": rpc error: code = NotFound desc = could not find container \"33c860c0baf5170edeb04423283fa7fd6bdf047f843b92d08af7aee63a3899be\": container with ID starting with 33c860c0baf5170edeb04423283fa7fd6bdf047f843b92d08af7aee63a3899be not found: ID does not exist"
May 25 12:19:11.646597 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:19:11.646577    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d7bd68cf-r7wdn"]
May 25 12:19:11.649717 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:19:11.649698    2555 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-d7bd68cf-r7wdn"]
May 25 12:19:13.598442 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:19:13.598410    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c442196-30d0-4274-936c-baad6f5e7a97" path="/var/lib/kubelet/pods/9c442196-30d0-4274-936c-baad6f5e7a97/volumes"
May 25 12:19:22.953411 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:19:22.953379    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-xtm76"]
May 25 12:19:22.953855 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:19:22.953652    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c442196-30d0-4274-936c-baad6f5e7a97" containerName="console"
May 25 12:19:22.953855 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:19:22.953667    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c442196-30d0-4274-936c-baad6f5e7a97" containerName="console"
May 25 12:19:22.953855 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:19:22.953729    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="9c442196-30d0-4274-936c-baad6f5e7a97" containerName="console"
May 25 12:19:22.958010 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:19:22.957991    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtm76"
May 25 12:19:22.959753 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:19:22.959735    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 25 12:19:22.963580 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:19:22.963556    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xtm76"]
May 25 12:19:23.091112 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:19:23.091065    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/24f7bcc2-ad35-49e6-b69c-175a02068e46-dbus\") pod \"global-pull-secret-syncer-xtm76\" (UID: \"24f7bcc2-ad35-49e6-b69c-175a02068e46\") " pod="kube-system/global-pull-secret-syncer-xtm76"
May 25 12:19:23.091112 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:19:23.091116    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24f7bcc2-ad35-49e6-b69c-175a02068e46-original-pull-secret\") pod \"global-pull-secret-syncer-xtm76\" (UID: \"24f7bcc2-ad35-49e6-b69c-175a02068e46\") " pod="kube-system/global-pull-secret-syncer-xtm76"
May 25 12:19:23.091307 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:19:23.091208    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/24f7bcc2-ad35-49e6-b69c-175a02068e46-kubelet-config\") pod \"global-pull-secret-syncer-xtm76\" (UID: \"24f7bcc2-ad35-49e6-b69c-175a02068e46\") " pod="kube-system/global-pull-secret-syncer-xtm76"
May 25 12:19:23.192488 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:19:23.192458    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/24f7bcc2-ad35-49e6-b69c-175a02068e46-dbus\") pod \"global-pull-secret-syncer-xtm76\" (UID: \"24f7bcc2-ad35-49e6-b69c-175a02068e46\") " pod="kube-system/global-pull-secret-syncer-xtm76"
May 25 12:19:23.192632 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:19:23.192500    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24f7bcc2-ad35-49e6-b69c-175a02068e46-original-pull-secret\") pod \"global-pull-secret-syncer-xtm76\" (UID: \"24f7bcc2-ad35-49e6-b69c-175a02068e46\") " pod="kube-system/global-pull-secret-syncer-xtm76"
May 25 12:19:23.192632 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:19:23.192552    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/24f7bcc2-ad35-49e6-b69c-175a02068e46-kubelet-config\") pod \"global-pull-secret-syncer-xtm76\" (UID: \"24f7bcc2-ad35-49e6-b69c-175a02068e46\") " pod="kube-system/global-pull-secret-syncer-xtm76"
May 25 12:19:23.192632 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:19:23.192610    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/24f7bcc2-ad35-49e6-b69c-175a02068e46-kubelet-config\") pod \"global-pull-secret-syncer-xtm76\" (UID: \"24f7bcc2-ad35-49e6-b69c-175a02068e46\") " pod="kube-system/global-pull-secret-syncer-xtm76"
May 25 12:19:23.192763 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:19:23.192614    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/24f7bcc2-ad35-49e6-b69c-175a02068e46-dbus\") pod \"global-pull-secret-syncer-xtm76\" (UID: \"24f7bcc2-ad35-49e6-b69c-175a02068e46\") " pod="kube-system/global-pull-secret-syncer-xtm76"
May 25 12:19:23.194790 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:19:23.194774    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24f7bcc2-ad35-49e6-b69c-175a02068e46-original-pull-secret\") pod \"global-pull-secret-syncer-xtm76\" (UID: \"24f7bcc2-ad35-49e6-b69c-175a02068e46\") " pod="kube-system/global-pull-secret-syncer-xtm76"
May 25 12:19:23.267275 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:19:23.267222    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtm76"
May 25 12:19:23.387452 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:19:23.387420    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xtm76"]
May 25 12:19:23.390292 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:19:23.390264    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24f7bcc2_ad35_49e6_b69c_175a02068e46.slice/crio-bea869be194753d33cb9b768141390f7808cb989c44654da22645da65d3394d3 WatchSource:0}: Error finding container bea869be194753d33cb9b768141390f7808cb989c44654da22645da65d3394d3: Status 404 returned error can't find the container with id bea869be194753d33cb9b768141390f7808cb989c44654da22645da65d3394d3
May 25 12:19:23.666464 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:19:23.666383    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xtm76" event={"ID":"24f7bcc2-ad35-49e6-b69c-175a02068e46","Type":"ContainerStarted","Data":"bea869be194753d33cb9b768141390f7808cb989c44654da22645da65d3394d3"}
May 25 12:19:28.680271 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:19:28.680231    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xtm76" event={"ID":"24f7bcc2-ad35-49e6-b69c-175a02068e46","Type":"ContainerStarted","Data":"87474d943cc8ed7282d9f4eb63e4e2422f85c982bf5d78909c54723b7adf665c"}
May 25 12:19:28.694312 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:19:28.694268    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-xtm76" podStartSLOduration=1.625326026 podStartE2EDuration="6.694255249s" podCreationTimestamp="2026-05-25 12:19:22 +0000 UTC" firstStartedPulling="2026-05-25 12:19:23.391864649 +0000 UTC m=+376.351915033" lastFinishedPulling="2026-05-25 12:19:28.460793854 +0000 UTC m=+381.420844256" observedRunningTime="2026-05-25 12:19:28.692792108 +0000 UTC m=+381.652842514" watchObservedRunningTime="2026-05-25 12:19:28.694255249 +0000 UTC m=+381.654305655"
May 25 12:20:17.521661 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:17.521605    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7k54c"]
May 25 12:20:17.528265 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:17.528238    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7k54c"
May 25 12:20:17.530604 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:17.530573    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-wwfbp\""
May 25 12:20:17.530744 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:17.530606    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
May 25 12:20:17.530744 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:17.530684    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
May 25 12:20:17.532136 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:17.532115    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7k54c"]
May 25 12:20:17.591507 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:17.591482    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hkjm\" (UniqueName: \"kubernetes.io/projected/ab33eb61-d257-498e-8da0-2d2d64f6582d-kube-api-access-9hkjm\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7k54c\" (UID: \"ab33eb61-d257-498e-8da0-2d2d64f6582d\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7k54c"
May 25 12:20:17.591631 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:17.591535    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab33eb61-d257-498e-8da0-2d2d64f6582d-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7k54c\" (UID: \"ab33eb61-d257-498e-8da0-2d2d64f6582d\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7k54c"
May 25 12:20:17.591631 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:17.591583    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab33eb61-d257-498e-8da0-2d2d64f6582d-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7k54c\" (UID: \"ab33eb61-d257-498e-8da0-2d2d64f6582d\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7k54c"
May 25 12:20:17.692685 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:17.692632    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab33eb61-d257-498e-8da0-2d2d64f6582d-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7k54c\" (UID: \"ab33eb61-d257-498e-8da0-2d2d64f6582d\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7k54c"
May 25 12:20:17.692848 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:17.692698    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9hkjm\" (UniqueName: \"kubernetes.io/projected/ab33eb61-d257-498e-8da0-2d2d64f6582d-kube-api-access-9hkjm\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7k54c\" (UID: \"ab33eb61-d257-498e-8da0-2d2d64f6582d\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7k54c"
May 25 12:20:17.692848 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:17.692750    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab33eb61-d257-498e-8da0-2d2d64f6582d-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7k54c\" (UID: \"ab33eb61-d257-498e-8da0-2d2d64f6582d\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7k54c"
May 25 12:20:17.693079 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:17.693056    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab33eb61-d257-498e-8da0-2d2d64f6582d-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7k54c\" (UID: \"ab33eb61-d257-498e-8da0-2d2d64f6582d\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7k54c"
May 25 12:20:17.693145 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:17.693105    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab33eb61-d257-498e-8da0-2d2d64f6582d-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7k54c\" (UID: \"ab33eb61-d257-498e-8da0-2d2d64f6582d\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7k54c"
May 25 12:20:17.701225 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:17.701200    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hkjm\" (UniqueName: \"kubernetes.io/projected/ab33eb61-d257-498e-8da0-2d2d64f6582d-kube-api-access-9hkjm\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7k54c\" (UID: \"ab33eb61-d257-498e-8da0-2d2d64f6582d\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7k54c"
May 25 12:20:17.837736 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:17.837667    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7k54c"
May 25 12:20:17.951576 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:17.951537    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7k54c"]
May 25 12:20:17.953848 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:20:17.953819    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab33eb61_d257_498e_8da0_2d2d64f6582d.slice/crio-681eddc9cb5369e14659e7153f962205e118e81b601ea1a1085235f343d1fd7d WatchSource:0}: Error finding container 681eddc9cb5369e14659e7153f962205e118e81b601ea1a1085235f343d1fd7d: Status 404 returned error can't find the container with id 681eddc9cb5369e14659e7153f962205e118e81b601ea1a1085235f343d1fd7d
May 25 12:20:18.815009 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:18.814971    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7k54c" event={"ID":"ab33eb61-d257-498e-8da0-2d2d64f6582d","Type":"ContainerStarted","Data":"681eddc9cb5369e14659e7153f962205e118e81b601ea1a1085235f343d1fd7d"}
May 25 12:20:25.835071 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:25.835040    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7k54c" event={"ID":"ab33eb61-d257-498e-8da0-2d2d64f6582d","Type":"ContainerStarted","Data":"378a8c9eb6c4de512d9f1a49b7295a36441674e05d35cd0bf8c9e561830d6cbf"}
May 25 12:20:26.839272 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:26.839238    2555 generic.go:358] "Generic (PLEG): container finished" podID="ab33eb61-d257-498e-8da0-2d2d64f6582d" containerID="378a8c9eb6c4de512d9f1a49b7295a36441674e05d35cd0bf8c9e561830d6cbf" exitCode=0
May 25 12:20:26.839635 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:26.839289    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7k54c" event={"ID":"ab33eb61-d257-498e-8da0-2d2d64f6582d","Type":"ContainerDied","Data":"378a8c9eb6c4de512d9f1a49b7295a36441674e05d35cd0bf8c9e561830d6cbf"}
May 25 12:20:33.859104 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:33.859064    2555 generic.go:358] "Generic (PLEG): container finished" podID="ab33eb61-d257-498e-8da0-2d2d64f6582d" containerID="df98f66919900e11525121e416e2b5488d7e0f7d4677bd160bb9435aa882dc31" exitCode=0
May 25 12:20:33.859465 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:33.859111    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7k54c" event={"ID":"ab33eb61-d257-498e-8da0-2d2d64f6582d","Type":"ContainerDied","Data":"df98f66919900e11525121e416e2b5488d7e0f7d4677bd160bb9435aa882dc31"}
May 25 12:20:41.885284 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:41.885250    2555 generic.go:358] "Generic (PLEG): container finished" podID="ab33eb61-d257-498e-8da0-2d2d64f6582d" containerID="4c9f78c00b40a9378824844e2bd5a09c1085b8d96617aa458de1180af021ecdf" exitCode=0
May 25 12:20:41.885695 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:41.885336    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7k54c" event={"ID":"ab33eb61-d257-498e-8da0-2d2d64f6582d","Type":"ContainerDied","Data":"4c9f78c00b40a9378824844e2bd5a09c1085b8d96617aa458de1180af021ecdf"}
May 25 12:20:43.002928 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:43.002908    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7k54c"
May 25 12:20:43.107538 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:43.107503    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9hkjm\" (UniqueName: \"kubernetes.io/projected/ab33eb61-d257-498e-8da0-2d2d64f6582d-kube-api-access-9hkjm\") pod \"ab33eb61-d257-498e-8da0-2d2d64f6582d\" (UID: \"ab33eb61-d257-498e-8da0-2d2d64f6582d\") "
May 25 12:20:43.107717 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:43.107606    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab33eb61-d257-498e-8da0-2d2d64f6582d-bundle\") pod \"ab33eb61-d257-498e-8da0-2d2d64f6582d\" (UID: \"ab33eb61-d257-498e-8da0-2d2d64f6582d\") "
May 25 12:20:43.107717 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:43.107623    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab33eb61-d257-498e-8da0-2d2d64f6582d-util\") pod \"ab33eb61-d257-498e-8da0-2d2d64f6582d\" (UID: \"ab33eb61-d257-498e-8da0-2d2d64f6582d\") "
May 25 12:20:43.108149 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:43.108117    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab33eb61-d257-498e-8da0-2d2d64f6582d-bundle" (OuterVolumeSpecName: "bundle") pod "ab33eb61-d257-498e-8da0-2d2d64f6582d" (UID: "ab33eb61-d257-498e-8da0-2d2d64f6582d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:20:43.109603 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:43.109581    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab33eb61-d257-498e-8da0-2d2d64f6582d-kube-api-access-9hkjm" (OuterVolumeSpecName: "kube-api-access-9hkjm") pod "ab33eb61-d257-498e-8da0-2d2d64f6582d" (UID: "ab33eb61-d257-498e-8da0-2d2d64f6582d"). InnerVolumeSpecName "kube-api-access-9hkjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:20:43.113027 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:43.113003    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab33eb61-d257-498e-8da0-2d2d64f6582d-util" (OuterVolumeSpecName: "util") pod "ab33eb61-d257-498e-8da0-2d2d64f6582d" (UID: "ab33eb61-d257-498e-8da0-2d2d64f6582d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:20:43.209108 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:43.209085    2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab33eb61-d257-498e-8da0-2d2d64f6582d-bundle\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:20:43.209108 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:43.209108    2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab33eb61-d257-498e-8da0-2d2d64f6582d-util\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:20:43.209240 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:43.209118    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hkjm\" (UniqueName: \"kubernetes.io/projected/ab33eb61-d257-498e-8da0-2d2d64f6582d-kube-api-access-9hkjm\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:20:43.891606 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:43.891520    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7k54c"
May 25 12:20:43.891788 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:43.891519    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7k54c" event={"ID":"ab33eb61-d257-498e-8da0-2d2d64f6582d","Type":"ContainerDied","Data":"681eddc9cb5369e14659e7153f962205e118e81b601ea1a1085235f343d1fd7d"}
May 25 12:20:43.891788 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:43.891628    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="681eddc9cb5369e14659e7153f962205e118e81b601ea1a1085235f343d1fd7d"
May 25 12:20:49.341976 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:49.341934    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-qlk6d"]
May 25 12:20:49.342363 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:49.342220    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab33eb61-d257-498e-8da0-2d2d64f6582d" containerName="pull"
May 25 12:20:49.342363 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:49.342231    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab33eb61-d257-498e-8da0-2d2d64f6582d" containerName="pull"
May 25 12:20:49.342363 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:49.342244    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab33eb61-d257-498e-8da0-2d2d64f6582d" containerName="util"
May 25 12:20:49.342363 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:49.342249    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab33eb61-d257-498e-8da0-2d2d64f6582d" containerName="util"
May 25 12:20:49.342363 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:49.342257    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab33eb61-d257-498e-8da0-2d2d64f6582d" containerName="extract"
May 25 12:20:49.342363 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:49.342265    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab33eb61-d257-498e-8da0-2d2d64f6582d" containerName="extract"
May 25 12:20:49.342363 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:49.342318    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="ab33eb61-d257-498e-8da0-2d2d64f6582d" containerName="extract"
May 25 12:20:49.344519 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:49.344503    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-qlk6d"
May 25 12:20:49.346829 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:49.346807    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
May 25 12:20:49.346940 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:49.346869    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
May 25 12:20:49.346940 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:49.346871    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
May 25 12:20:49.347046 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:49.347032    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-5lldk\""
May 25 12:20:49.352256 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:49.352233    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-qlk6d"]
May 25 12:20:49.460418 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:49.460384    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/2e92d755-1944-49bd-a887-841b5f9a870c-certificates\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-qlk6d\" (UID: \"2e92d755-1944-49bd-a887-841b5f9a870c\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-qlk6d"
May 25 12:20:49.460572 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:49.460423    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd7qr\" (UniqueName: \"kubernetes.io/projected/2e92d755-1944-49bd-a887-841b5f9a870c-kube-api-access-vd7qr\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-qlk6d\" (UID: \"2e92d755-1944-49bd-a887-841b5f9a870c\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-qlk6d"
May 25 12:20:49.560999 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:49.560961    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vd7qr\" (UniqueName: \"kubernetes.io/projected/2e92d755-1944-49bd-a887-841b5f9a870c-kube-api-access-vd7qr\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-qlk6d\" (UID: \"2e92d755-1944-49bd-a887-841b5f9a870c\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-qlk6d"
May 25 12:20:49.561195 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:49.561046    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/2e92d755-1944-49bd-a887-841b5f9a870c-certificates\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-qlk6d\" (UID: \"2e92d755-1944-49bd-a887-841b5f9a870c\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-qlk6d"
May 25 12:20:49.563249 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:49.563226    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/2e92d755-1944-49bd-a887-841b5f9a870c-certificates\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-qlk6d\" (UID: \"2e92d755-1944-49bd-a887-841b5f9a870c\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-qlk6d"
May 25 12:20:49.568205 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:49.568178    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd7qr\" (UniqueName: \"kubernetes.io/projected/2e92d755-1944-49bd-a887-841b5f9a870c-kube-api-access-vd7qr\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-qlk6d\" (UID: \"2e92d755-1944-49bd-a887-841b5f9a870c\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-qlk6d"
May 25 12:20:49.655371 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:49.655291    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-qlk6d"
May 25 12:20:49.771286 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:49.771157    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-qlk6d"]
May 25 12:20:49.774086 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:20:49.774059    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e92d755_1944_49bd_a887_841b5f9a870c.slice/crio-4c8de75462c783dfe7ff5b5007eecc4168d22d2171f17577bb39b53dd46669c3 WatchSource:0}: Error finding container 4c8de75462c783dfe7ff5b5007eecc4168d22d2171f17577bb39b53dd46669c3: Status 404 returned error can't find the container with id 4c8de75462c783dfe7ff5b5007eecc4168d22d2171f17577bb39b53dd46669c3
May 25 12:20:49.910115 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:49.910036    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-qlk6d" event={"ID":"2e92d755-1944-49bd-a887-841b5f9a870c","Type":"ContainerStarted","Data":"4c8de75462c783dfe7ff5b5007eecc4168d22d2171f17577bb39b53dd46669c3"}
May 25 12:20:52.921506 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:52.921416    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-qlk6d" event={"ID":"2e92d755-1944-49bd-a887-841b5f9a870c","Type":"ContainerStarted","Data":"a235674a4dbdd8bbda1e017315713ce22346c283c9d9833d9183ed95dddd4266"}
May 25 12:20:52.921506 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:52.921480    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-qlk6d"
May 25 12:20:52.938332 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:52.938286    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-qlk6d" podStartSLOduration=1.060436548 podStartE2EDuration="3.938273999s" podCreationTimestamp="2026-05-25 12:20:49 +0000 UTC" firstStartedPulling="2026-05-25 12:20:49.775841477 +0000 UTC m=+462.735891872" lastFinishedPulling="2026-05-25 12:20:52.653678928 +0000 UTC m=+465.613729323" observedRunningTime="2026-05-25 12:20:52.936876981 +0000 UTC m=+465.896927387" watchObservedRunningTime="2026-05-25 12:20:52.938273999 +0000 UTC m=+465.898324404"
May 25 12:20:53.163438 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:53.163407    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-ffbb595cb-d6mm5"]
May 25 12:20:53.165495 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:53.165479    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-ffbb595cb-d6mm5"
May 25 12:20:53.167673 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:53.167636    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
May 25 12:20:53.167750 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:53.167674    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-dhbsf\""
May 25 12:20:53.167750 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:53.167726    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
May 25 12:20:53.178551 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:53.178532    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-ffbb595cb-d6mm5"]
May 25 12:20:53.191129 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:53.191105    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/503e0f1a-0e63-4045-b12b-a7dc6d433835-cabundle0\") pod \"keda-operator-ffbb595cb-d6mm5\" (UID: \"503e0f1a-0e63-4045-b12b-a7dc6d433835\") " pod="openshift-keda/keda-operator-ffbb595cb-d6mm5"
May 25 12:20:53.191226 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:53.191140    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkmbq\" (UniqueName: \"kubernetes.io/projected/503e0f1a-0e63-4045-b12b-a7dc6d433835-kube-api-access-lkmbq\") pod \"keda-operator-ffbb595cb-d6mm5\" (UID: \"503e0f1a-0e63-4045-b12b-a7dc6d433835\") " pod="openshift-keda/keda-operator-ffbb595cb-d6mm5"
May 25 12:20:53.191274 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:53.191231    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/503e0f1a-0e63-4045-b12b-a7dc6d433835-certificates\") pod \"keda-operator-ffbb595cb-d6mm5\" (UID: \"503e0f1a-0e63-4045-b12b-a7dc6d433835\") " pod="openshift-keda/keda-operator-ffbb595cb-d6mm5"
May 25 12:20:53.291950 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:53.291925    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/503e0f1a-0e63-4045-b12b-a7dc6d433835-cabundle0\") pod \"keda-operator-ffbb595cb-d6mm5\" (UID: \"503e0f1a-0e63-4045-b12b-a7dc6d433835\") " pod="openshift-keda/keda-operator-ffbb595cb-d6mm5"
May 25 12:20:53.292052 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:53.291967    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lkmbq\" (UniqueName: \"kubernetes.io/projected/503e0f1a-0e63-4045-b12b-a7dc6d433835-kube-api-access-lkmbq\") pod \"keda-operator-ffbb595cb-d6mm5\" (UID: \"503e0f1a-0e63-4045-b12b-a7dc6d433835\") " pod="openshift-keda/keda-operator-ffbb595cb-d6mm5"
May 25 12:20:53.292052 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:53.292010    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/503e0f1a-0e63-4045-b12b-a7dc6d433835-certificates\") pod \"keda-operator-ffbb595cb-d6mm5\" (UID: \"503e0f1a-0e63-4045-b12b-a7dc6d433835\") " pod="openshift-keda/keda-operator-ffbb595cb-d6mm5"
May 25 12:20:53.292177 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:20:53.292161    2555 projected.go:264] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
May 25 12:20:53.292234 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:20:53.292183    2555 secret.go:281] references non-existent secret key: ca.crt
May 25 12:20:53.292234 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:20:53.292193    2555 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
May 25 12:20:53.292234 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:20:53.292209    2555 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-d6mm5: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
May 25 12:20:53.292360 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:20:53.292279    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/503e0f1a-0e63-4045-b12b-a7dc6d433835-certificates podName:503e0f1a-0e63-4045-b12b-a7dc6d433835 nodeName:}" failed. No retries permitted until 2026-05-25 12:20:53.792259928 +0000 UTC m=+466.752310317 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/503e0f1a-0e63-4045-b12b-a7dc6d433835-certificates") pod "keda-operator-ffbb595cb-d6mm5" (UID: "503e0f1a-0e63-4045-b12b-a7dc6d433835") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
May 25 12:20:53.292511 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:53.292488    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/503e0f1a-0e63-4045-b12b-a7dc6d433835-cabundle0\") pod \"keda-operator-ffbb595cb-d6mm5\" (UID: \"503e0f1a-0e63-4045-b12b-a7dc6d433835\") " pod="openshift-keda/keda-operator-ffbb595cb-d6mm5"
May 25 12:20:53.300064 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:53.300039    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkmbq\" (UniqueName: \"kubernetes.io/projected/503e0f1a-0e63-4045-b12b-a7dc6d433835-kube-api-access-lkmbq\") pod \"keda-operator-ffbb595cb-d6mm5\" (UID: \"503e0f1a-0e63-4045-b12b-a7dc6d433835\") " pod="openshift-keda/keda-operator-ffbb595cb-d6mm5"
May 25 12:20:53.421339 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:53.421314    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-jvtt8"]
May 25 12:20:53.423617 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:53.423577    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-jvtt8"
May 25 12:20:53.425447 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:53.425431    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
May 25 12:20:53.430583 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:53.430489    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-jvtt8"]
May 25 12:20:53.493396 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:53.493372    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/a867979f-bcba-47f5-93c5-c77b72f2e540-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-jvtt8\" (UID: \"a867979f-bcba-47f5-93c5-c77b72f2e540\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-jvtt8"
May 25 12:20:53.493531 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:53.493428    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57sjc\" (UniqueName: \"kubernetes.io/projected/a867979f-bcba-47f5-93c5-c77b72f2e540-kube-api-access-57sjc\") pod \"keda-metrics-apiserver-7c9f485588-jvtt8\" (UID: \"a867979f-bcba-47f5-93c5-c77b72f2e540\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-jvtt8"
May 25 12:20:53.493531 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:53.493460    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a867979f-bcba-47f5-93c5-c77b72f2e540-certificates\") pod \"keda-metrics-apiserver-7c9f485588-jvtt8\" (UID: \"a867979f-bcba-47f5-93c5-c77b72f2e540\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-jvtt8"
May 25 12:20:53.594591 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:53.594565    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/a867979f-bcba-47f5-93c5-c77b72f2e540-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-jvtt8\" (UID: \"a867979f-bcba-47f5-93c5-c77b72f2e540\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-jvtt8"
May 25 12:20:53.594733 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:53.594659    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-57sjc\" (UniqueName: \"kubernetes.io/projected/a867979f-bcba-47f5-93c5-c77b72f2e540-kube-api-access-57sjc\") pod \"keda-metrics-apiserver-7c9f485588-jvtt8\" (UID: \"a867979f-bcba-47f5-93c5-c77b72f2e540\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-jvtt8"
May 25 12:20:53.594733 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:53.594712    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a867979f-bcba-47f5-93c5-c77b72f2e540-certificates\") pod \"keda-metrics-apiserver-7c9f485588-jvtt8\" (UID: \"a867979f-bcba-47f5-93c5-c77b72f2e540\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-jvtt8"
May 25 12:20:53.594846 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:20:53.594809    2555 secret.go:281] references non-existent secret key: tls.crt
May 25 12:20:53.594846 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:20:53.594829    2555 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
May 25 12:20:53.594938 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:20:53.594849    2555 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-jvtt8: references non-existent secret key: tls.crt
May 25 12:20:53.594938 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:20:53.594908    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a867979f-bcba-47f5-93c5-c77b72f2e540-certificates podName:a867979f-bcba-47f5-93c5-c77b72f2e540 nodeName:}" failed. No retries permitted until 2026-05-25 12:20:54.094890655 +0000 UTC m=+467.054941047 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/a867979f-bcba-47f5-93c5-c77b72f2e540-certificates") pod "keda-metrics-apiserver-7c9f485588-jvtt8" (UID: "a867979f-bcba-47f5-93c5-c77b72f2e540") : references non-existent secret key: tls.crt
May 25 12:20:53.595051 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:53.594968    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/a867979f-bcba-47f5-93c5-c77b72f2e540-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-jvtt8\" (UID: \"a867979f-bcba-47f5-93c5-c77b72f2e540\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-jvtt8"
May 25 12:20:53.607175 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:53.607148    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57sjc\" (UniqueName: \"kubernetes.io/projected/a867979f-bcba-47f5-93c5-c77b72f2e540-kube-api-access-57sjc\") pod \"keda-metrics-apiserver-7c9f485588-jvtt8\" (UID: \"a867979f-bcba-47f5-93c5-c77b72f2e540\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-jvtt8"
May 25 12:20:53.795948 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:53.795910    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/503e0f1a-0e63-4045-b12b-a7dc6d433835-certificates\") pod \"keda-operator-ffbb595cb-d6mm5\" (UID: \"503e0f1a-0e63-4045-b12b-a7dc6d433835\") " pod="openshift-keda/keda-operator-ffbb595cb-d6mm5"
May 25 12:20:53.796127 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:20:53.796061    2555 projected.go:264] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
May 25 12:20:53.796127 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:20:53.796079    2555 secret.go:281] references non-existent secret key: ca.crt
May 25 12:20:53.796127 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:20:53.796089    2555 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
May 25 12:20:53.796127 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:20:53.796102    2555 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-d6mm5: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
May 25 12:20:53.796328 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:20:53.796151    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/503e0f1a-0e63-4045-b12b-a7dc6d433835-certificates podName:503e0f1a-0e63-4045-b12b-a7dc6d433835 nodeName:}" failed. No retries permitted until 2026-05-25 12:20:54.796136011 +0000 UTC m=+467.756186394 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/503e0f1a-0e63-4045-b12b-a7dc6d433835-certificates") pod "keda-operator-ffbb595cb-d6mm5" (UID: "503e0f1a-0e63-4045-b12b-a7dc6d433835") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
May 25 12:20:54.097554 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:54.097473    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a867979f-bcba-47f5-93c5-c77b72f2e540-certificates\") pod \"keda-metrics-apiserver-7c9f485588-jvtt8\" (UID: \"a867979f-bcba-47f5-93c5-c77b72f2e540\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-jvtt8"
May 25 12:20:54.097918 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:20:54.097599    2555 secret.go:281] references non-existent secret key: tls.crt
May 25 12:20:54.097918 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:20:54.097618    2555 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
May 25 12:20:54.097918 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:20:54.097634    2555 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-jvtt8: references non-existent secret key: tls.crt
May 25 12:20:54.097918 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:20:54.097707    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a867979f-bcba-47f5-93c5-c77b72f2e540-certificates podName:a867979f-bcba-47f5-93c5-c77b72f2e540 nodeName:}" failed. No retries permitted until 2026-05-25 12:20:55.097694015 +0000 UTC m=+468.057744398 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/a867979f-bcba-47f5-93c5-c77b72f2e540-certificates") pod "keda-metrics-apiserver-7c9f485588-jvtt8" (UID: "a867979f-bcba-47f5-93c5-c77b72f2e540") : references non-existent secret key: tls.crt
May 25 12:20:54.801903 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:54.801870    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/503e0f1a-0e63-4045-b12b-a7dc6d433835-certificates\") pod \"keda-operator-ffbb595cb-d6mm5\" (UID: \"503e0f1a-0e63-4045-b12b-a7dc6d433835\") " pod="openshift-keda/keda-operator-ffbb595cb-d6mm5"
May 25 12:20:54.802064 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:20:54.802010    2555 secret.go:281] references non-existent secret key: ca.crt
May 25 12:20:54.802064 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:20:54.802030    2555 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
May 25 12:20:54.802064 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:20:54.802038    2555 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-d6mm5: references non-existent secret key: ca.crt
May 25 12:20:54.802169 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:20:54.802088    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/503e0f1a-0e63-4045-b12b-a7dc6d433835-certificates podName:503e0f1a-0e63-4045-b12b-a7dc6d433835 nodeName:}" failed. No retries permitted until 2026-05-25 12:20:56.802073775 +0000 UTC m=+469.762124164 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/503e0f1a-0e63-4045-b12b-a7dc6d433835-certificates") pod "keda-operator-ffbb595cb-d6mm5" (UID: "503e0f1a-0e63-4045-b12b-a7dc6d433835") : references non-existent secret key: ca.crt
May 25 12:20:55.103939 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:55.103863    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a867979f-bcba-47f5-93c5-c77b72f2e540-certificates\") pod \"keda-metrics-apiserver-7c9f485588-jvtt8\" (UID: \"a867979f-bcba-47f5-93c5-c77b72f2e540\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-jvtt8"
May 25 12:20:55.104357 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:20:55.104020    2555 secret.go:281] references non-existent secret key: tls.crt
May 25 12:20:55.104357 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:20:55.104043    2555 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
May 25 12:20:55.104357 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:20:55.104068    2555 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-jvtt8: references non-existent secret key: tls.crt
May 25 12:20:55.104357 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:20:55.104133    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a867979f-bcba-47f5-93c5-c77b72f2e540-certificates podName:a867979f-bcba-47f5-93c5-c77b72f2e540 nodeName:}" failed. No retries permitted until 2026-05-25 12:20:57.104114849 +0000 UTC m=+470.064165232 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/a867979f-bcba-47f5-93c5-c77b72f2e540-certificates") pod "keda-metrics-apiserver-7c9f485588-jvtt8" (UID: "a867979f-bcba-47f5-93c5-c77b72f2e540") : references non-existent secret key: tls.crt
May 25 12:20:56.815561 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:56.815533    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/503e0f1a-0e63-4045-b12b-a7dc6d433835-certificates\") pod \"keda-operator-ffbb595cb-d6mm5\" (UID: \"503e0f1a-0e63-4045-b12b-a7dc6d433835\") " pod="openshift-keda/keda-operator-ffbb595cb-d6mm5"
May 25 12:20:56.817897 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:56.817879    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/503e0f1a-0e63-4045-b12b-a7dc6d433835-certificates\") pod \"keda-operator-ffbb595cb-d6mm5\" (UID: \"503e0f1a-0e63-4045-b12b-a7dc6d433835\") " pod="openshift-keda/keda-operator-ffbb595cb-d6mm5"
May 25 12:20:57.075379 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:57.075291    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-ffbb595cb-d6mm5"
May 25 12:20:57.118430 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:57.118371    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a867979f-bcba-47f5-93c5-c77b72f2e540-certificates\") pod \"keda-metrics-apiserver-7c9f485588-jvtt8\" (UID: \"a867979f-bcba-47f5-93c5-c77b72f2e540\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-jvtt8"
May 25 12:20:57.124896 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:57.124836    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a867979f-bcba-47f5-93c5-c77b72f2e540-certificates\") pod \"keda-metrics-apiserver-7c9f485588-jvtt8\" (UID: \"a867979f-bcba-47f5-93c5-c77b72f2e540\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-jvtt8"
May 25 12:20:57.197513 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:57.197488    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-ffbb595cb-d6mm5"]
May 25 12:20:57.199216 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:20:57.199186    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod503e0f1a_0e63_4045_b12b_a7dc6d433835.slice/crio-c2626e09ccd358bd6587c85dc6c494a953ab906ea8b2a0d8095b553473adea88 WatchSource:0}: Error finding container c2626e09ccd358bd6587c85dc6c494a953ab906ea8b2a0d8095b553473adea88: Status 404 returned error can't find the container with id c2626e09ccd358bd6587c85dc6c494a953ab906ea8b2a0d8095b553473adea88
May 25 12:20:57.336016 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:57.335957    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-jvtt8"
May 25 12:20:57.451069 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:57.451045    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-jvtt8"]
May 25 12:20:57.453043 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:20:57.453017    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda867979f_bcba_47f5_93c5_c77b72f2e540.slice/crio-3a7cc9a680a02b85e44c6228de717757c957e823ec15e2f536e55c4417fbf7da WatchSource:0}: Error finding container 3a7cc9a680a02b85e44c6228de717757c957e823ec15e2f536e55c4417fbf7da: Status 404 returned error can't find the container with id 3a7cc9a680a02b85e44c6228de717757c957e823ec15e2f536e55c4417fbf7da
May 25 12:20:57.936262 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:57.936228    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-ffbb595cb-d6mm5" event={"ID":"503e0f1a-0e63-4045-b12b-a7dc6d433835","Type":"ContainerStarted","Data":"c2626e09ccd358bd6587c85dc6c494a953ab906ea8b2a0d8095b553473adea88"}
May 25 12:20:57.937193 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:20:57.937170    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-jvtt8" event={"ID":"a867979f-bcba-47f5-93c5-c77b72f2e540","Type":"ContainerStarted","Data":"3a7cc9a680a02b85e44c6228de717757c957e823ec15e2f536e55c4417fbf7da"}
May 25 12:21:01.951405 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:21:01.951322    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-jvtt8" event={"ID":"a867979f-bcba-47f5-93c5-c77b72f2e540","Type":"ContainerStarted","Data":"2a8a0691d072bdaddf3cc21cfb1ffefbd49e0e6698c8fa639550c6055956223d"}
May 25 12:21:01.951760 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:21:01.951443    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-jvtt8"
May 25 12:21:01.966925 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:21:01.966879    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-jvtt8" podStartSLOduration=4.730676583 podStartE2EDuration="8.966866752s" podCreationTimestamp="2026-05-25 12:20:53 +0000 UTC" firstStartedPulling="2026-05-25 12:20:57.45429024 +0000 UTC m=+470.414340624" lastFinishedPulling="2026-05-25 12:21:01.690480407 +0000 UTC m=+474.650530793" observedRunningTime="2026-05-25 12:21:01.965192205 +0000 UTC m=+474.925242610" watchObservedRunningTime="2026-05-25 12:21:01.966866752 +0000 UTC m=+474.926917157"
May 25 12:21:07.971751 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:21:07.971712    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-ffbb595cb-d6mm5" event={"ID":"503e0f1a-0e63-4045-b12b-a7dc6d433835","Type":"ContainerStarted","Data":"b38e5ecdc86fc7f975c169a10f1b40ce7ca14d4dc4dc2668e8ee409f5404baf0"}
May 25 12:21:07.972111 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:21:07.971838    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-ffbb595cb-d6mm5"
May 25 12:21:07.987097 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:21:07.987053    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-operator-ffbb595cb-d6mm5" podStartSLOduration=4.902117185 podStartE2EDuration="14.987041139s" podCreationTimestamp="2026-05-25 12:20:53 +0000 UTC" firstStartedPulling="2026-05-25 12:20:57.200559214 +0000 UTC m=+470.160609597" lastFinishedPulling="2026-05-25 12:21:07.285483155 +0000 UTC m=+480.245533551" observedRunningTime="2026-05-25 12:21:07.986103994 +0000 UTC m=+480.946154399" watchObservedRunningTime="2026-05-25 12:21:07.987041139 +0000 UTC m=+480.947091545"
May 25 12:21:12.959456 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:21:12.959429    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-jvtt8"
May 25 12:21:13.927046 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:21:13.927017    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-qlk6d"
May 25 12:21:28.977483 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:21:28.977454    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-ffbb595cb-d6mm5"
May 25 12:22:00.356745 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:00.356668    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-6768985688-xwcgw"]
May 25 12:22:00.358781 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:00.358764    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6768985688-xwcgw"
May 25 12:22:00.360628 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:00.360604    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
May 25 12:22:00.360935 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:00.360913    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
May 25 12:22:00.361074 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:00.360940    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-qdcbh\""
May 25 12:22:00.361074 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:00.360965    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
May 25 12:22:00.367026 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:00.367005    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6768985688-xwcgw"]
May 25 12:22:00.394978 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:00.394957    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-79f457656b-g7fg4"]
May 25 12:22:00.396874 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:00.396860    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-79f457656b-g7fg4"
May 25 12:22:00.398674 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:00.398638    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
May 25 12:22:00.398779 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:00.398741    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-85jzv\""
May 25 12:22:00.405365 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:00.405347    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-79f457656b-g7fg4"]
May 25 12:22:00.497921 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:00.497894    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e0276877-74a8-4402-938c-87bed561e826-data\") pod \"seaweedfs-79f457656b-g7fg4\" (UID: \"e0276877-74a8-4402-938c-87bed561e826\") " pod="kserve/seaweedfs-79f457656b-g7fg4"
May 25 12:22:00.498050 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:00.497924    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/203e26c3-1be4-43d9-9402-4d1f190909a8-cert\") pod \"kserve-controller-manager-6768985688-xwcgw\" (UID: \"203e26c3-1be4-43d9-9402-4d1f190909a8\") " pod="kserve/kserve-controller-manager-6768985688-xwcgw"
May 25 12:22:00.498050 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:00.497944    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f4gh\" (UniqueName: \"kubernetes.io/projected/203e26c3-1be4-43d9-9402-4d1f190909a8-kube-api-access-7f4gh\") pod \"kserve-controller-manager-6768985688-xwcgw\" (UID: \"203e26c3-1be4-43d9-9402-4d1f190909a8\") " pod="kserve/kserve-controller-manager-6768985688-xwcgw"
May 25 12:22:00.498050 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:00.497972    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt8hf\" (UniqueName: \"kubernetes.io/projected/e0276877-74a8-4402-938c-87bed561e826-kube-api-access-pt8hf\") pod \"seaweedfs-79f457656b-g7fg4\" (UID: \"e0276877-74a8-4402-938c-87bed561e826\") " pod="kserve/seaweedfs-79f457656b-g7fg4"
May 25 12:22:00.598868 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:00.598834    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e0276877-74a8-4402-938c-87bed561e826-data\") pod \"seaweedfs-79f457656b-g7fg4\" (UID: \"e0276877-74a8-4402-938c-87bed561e826\") " pod="kserve/seaweedfs-79f457656b-g7fg4"
May 25 12:22:00.598868 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:00.598867    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/203e26c3-1be4-43d9-9402-4d1f190909a8-cert\") pod \"kserve-controller-manager-6768985688-xwcgw\" (UID: \"203e26c3-1be4-43d9-9402-4d1f190909a8\") " pod="kserve/kserve-controller-manager-6768985688-xwcgw"
May 25 12:22:00.599138 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:00.598888    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7f4gh\" (UniqueName: \"kubernetes.io/projected/203e26c3-1be4-43d9-9402-4d1f190909a8-kube-api-access-7f4gh\") pod \"kserve-controller-manager-6768985688-xwcgw\" (UID: \"203e26c3-1be4-43d9-9402-4d1f190909a8\") " pod="kserve/kserve-controller-manager-6768985688-xwcgw"
May 25 12:22:00.599138 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:00.598916    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pt8hf\" (UniqueName: \"kubernetes.io/projected/e0276877-74a8-4402-938c-87bed561e826-kube-api-access-pt8hf\") pod \"seaweedfs-79f457656b-g7fg4\" (UID: \"e0276877-74a8-4402-938c-87bed561e826\") " pod="kserve/seaweedfs-79f457656b-g7fg4"
May 25 12:22:00.599138 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:22:00.598998    2555 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
May 25 12:22:00.599138 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:22:00.599075    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/203e26c3-1be4-43d9-9402-4d1f190909a8-cert podName:203e26c3-1be4-43d9-9402-4d1f190909a8 nodeName:}" failed. No retries permitted until 2026-05-25 12:22:01.099056583 +0000 UTC m=+534.059106971 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/203e26c3-1be4-43d9-9402-4d1f190909a8-cert") pod "kserve-controller-manager-6768985688-xwcgw" (UID: "203e26c3-1be4-43d9-9402-4d1f190909a8") : secret "kserve-webhook-server-cert" not found
May 25 12:22:00.599322 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:00.599173    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e0276877-74a8-4402-938c-87bed561e826-data\") pod \"seaweedfs-79f457656b-g7fg4\" (UID: \"e0276877-74a8-4402-938c-87bed561e826\") " pod="kserve/seaweedfs-79f457656b-g7fg4"
May 25 12:22:00.607002 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:00.606946    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt8hf\" (UniqueName: \"kubernetes.io/projected/e0276877-74a8-4402-938c-87bed561e826-kube-api-access-pt8hf\") pod \"seaweedfs-79f457656b-g7fg4\" (UID: \"e0276877-74a8-4402-938c-87bed561e826\") " pod="kserve/seaweedfs-79f457656b-g7fg4"
May 25 12:22:00.607091 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:00.607015    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f4gh\" (UniqueName: \"kubernetes.io/projected/203e26c3-1be4-43d9-9402-4d1f190909a8-kube-api-access-7f4gh\") pod \"kserve-controller-manager-6768985688-xwcgw\" (UID: \"203e26c3-1be4-43d9-9402-4d1f190909a8\") " pod="kserve/kserve-controller-manager-6768985688-xwcgw"
May 25 12:22:00.705367 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:00.705324    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-79f457656b-g7fg4"
May 25 12:22:00.826113 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:00.826087    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-79f457656b-g7fg4"]
May 25 12:22:00.828604 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:22:00.828577    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0276877_74a8_4402_938c_87bed561e826.slice/crio-dba85d8dd1bf3b28906c9ce8123e61bcea96c11904168023df3d8b8f09c0df6f WatchSource:0}: Error finding container dba85d8dd1bf3b28906c9ce8123e61bcea96c11904168023df3d8b8f09c0df6f: Status 404 returned error can't find the container with id dba85d8dd1bf3b28906c9ce8123e61bcea96c11904168023df3d8b8f09c0df6f
May 25 12:22:01.104132 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:01.104100    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/203e26c3-1be4-43d9-9402-4d1f190909a8-cert\") pod \"kserve-controller-manager-6768985688-xwcgw\" (UID: \"203e26c3-1be4-43d9-9402-4d1f190909a8\") " pod="kserve/kserve-controller-manager-6768985688-xwcgw"
May 25 12:22:01.106360 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:01.106342    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/203e26c3-1be4-43d9-9402-4d1f190909a8-cert\") pod \"kserve-controller-manager-6768985688-xwcgw\" (UID: \"203e26c3-1be4-43d9-9402-4d1f190909a8\") " pod="kserve/kserve-controller-manager-6768985688-xwcgw"
May 25 12:22:01.143793 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:01.143764    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-79f457656b-g7fg4" event={"ID":"e0276877-74a8-4402-938c-87bed561e826","Type":"ContainerStarted","Data":"dba85d8dd1bf3b28906c9ce8123e61bcea96c11904168023df3d8b8f09c0df6f"}
May 25 12:22:01.269408 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:01.269377    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6768985688-xwcgw"
May 25 12:22:01.395655 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:01.395621    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6768985688-xwcgw"]
May 25 12:22:01.397381 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:22:01.397351    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod203e26c3_1be4_43d9_9402_4d1f190909a8.slice/crio-74d96d9efb4b1427ddf970d4aeffaab17514790710fa3061f80172174f78087c WatchSource:0}: Error finding container 74d96d9efb4b1427ddf970d4aeffaab17514790710fa3061f80172174f78087c: Status 404 returned error can't find the container with id 74d96d9efb4b1427ddf970d4aeffaab17514790710fa3061f80172174f78087c
May 25 12:22:02.159583 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:02.159530    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6768985688-xwcgw" event={"ID":"203e26c3-1be4-43d9-9402-4d1f190909a8","Type":"ContainerStarted","Data":"74d96d9efb4b1427ddf970d4aeffaab17514790710fa3061f80172174f78087c"}
May 25 12:22:05.171126 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:05.171071    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6768985688-xwcgw" event={"ID":"203e26c3-1be4-43d9-9402-4d1f190909a8","Type":"ContainerStarted","Data":"3f7928b6173643e6e5316654f95f70a602d9bdffce32960a21c0e61e866968b5"}
May 25 12:22:05.171544 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:05.171208    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-6768985688-xwcgw"
May 25 12:22:05.172771 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:05.172727    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-79f457656b-g7fg4" event={"ID":"e0276877-74a8-4402-938c-87bed561e826","Type":"ContainerStarted","Data":"a6e4f3528cafe89ca70caa39f735be35ad403545e2d56fd77c57cb90c4ce2f18"}
May 25 12:22:05.172987 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:05.172831    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-79f457656b-g7fg4"
May 25 12:22:05.185978 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:05.185938    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/kserve-controller-manager-6768985688-xwcgw" podStartSLOduration=1.628450884 podStartE2EDuration="5.185927536s" podCreationTimestamp="2026-05-25 12:22:00 +0000 UTC" firstStartedPulling="2026-05-25 12:22:01.398688918 +0000 UTC m=+534.358739302" lastFinishedPulling="2026-05-25 12:22:04.956165569 +0000 UTC m=+537.916215954" observedRunningTime="2026-05-25 12:22:05.185225633 +0000 UTC m=+538.145276040" watchObservedRunningTime="2026-05-25 12:22:05.185927536 +0000 UTC m=+538.145977941"
May 25 12:22:05.198985 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:05.198952    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/seaweedfs-79f457656b-g7fg4" podStartSLOduration=1.000139645 podStartE2EDuration="5.198943374s" podCreationTimestamp="2026-05-25 12:22:00 +0000 UTC" firstStartedPulling="2026-05-25 12:22:00.83003828 +0000 UTC m=+533.790088680" lastFinishedPulling="2026-05-25 12:22:05.028842025 +0000 UTC m=+537.988892409" observedRunningTime="2026-05-25 12:22:05.1977427 +0000 UTC m=+538.157793106" watchObservedRunningTime="2026-05-25 12:22:05.198943374 +0000 UTC m=+538.158993780"
May 25 12:22:11.178150 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:11.178113    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-79f457656b-g7fg4"
May 25 12:22:35.195350 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:35.195321    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-6db9bc8779-v7b2x"]
May 25 12:22:35.197870 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:35.197846    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6db9bc8779-v7b2x"
May 25 12:22:35.206657 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:35.206617    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6db9bc8779-v7b2x"]
May 25 12:22:35.265086 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:35.265055    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4d71f6ce-1aec-465f-885e-637117a22cce-service-ca\") pod \"console-6db9bc8779-v7b2x\" (UID: \"4d71f6ce-1aec-465f-885e-637117a22cce\") " pod="openshift-console/console-6db9bc8779-v7b2x"
May 25 12:22:35.265086 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:35.265088    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4d71f6ce-1aec-465f-885e-637117a22cce-oauth-serving-cert\") pod \"console-6db9bc8779-v7b2x\" (UID: \"4d71f6ce-1aec-465f-885e-637117a22cce\") " pod="openshift-console/console-6db9bc8779-v7b2x"
May 25 12:22:35.265259 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:35.265106    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8qg6\" (UniqueName: \"kubernetes.io/projected/4d71f6ce-1aec-465f-885e-637117a22cce-kube-api-access-p8qg6\") pod \"console-6db9bc8779-v7b2x\" (UID: \"4d71f6ce-1aec-465f-885e-637117a22cce\") " pod="openshift-console/console-6db9bc8779-v7b2x"
May 25 12:22:35.265259 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:35.265194    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4d71f6ce-1aec-465f-885e-637117a22cce-console-config\") pod \"console-6db9bc8779-v7b2x\" (UID: \"4d71f6ce-1aec-465f-885e-637117a22cce\") " pod="openshift-console/console-6db9bc8779-v7b2x"
May 25 12:22:35.265259 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:35.265239    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4d71f6ce-1aec-465f-885e-637117a22cce-console-oauth-config\") pod \"console-6db9bc8779-v7b2x\" (UID: \"4d71f6ce-1aec-465f-885e-637117a22cce\") " pod="openshift-console/console-6db9bc8779-v7b2x"
May 25 12:22:35.265398 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:35.265292    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d71f6ce-1aec-465f-885e-637117a22cce-console-serving-cert\") pod \"console-6db9bc8779-v7b2x\" (UID: \"4d71f6ce-1aec-465f-885e-637117a22cce\") " pod="openshift-console/console-6db9bc8779-v7b2x"
May 25 12:22:35.265398 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:35.265330    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d71f6ce-1aec-465f-885e-637117a22cce-trusted-ca-bundle\") pod \"console-6db9bc8779-v7b2x\" (UID: \"4d71f6ce-1aec-465f-885e-637117a22cce\") " pod="openshift-console/console-6db9bc8779-v7b2x"
May 25 12:22:35.365890 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:35.365867    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d71f6ce-1aec-465f-885e-637117a22cce-console-serving-cert\") pod \"console-6db9bc8779-v7b2x\" (UID: \"4d71f6ce-1aec-465f-885e-637117a22cce\") " pod="openshift-console/console-6db9bc8779-v7b2x"
May 25 12:22:35.366021 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:35.365899    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d71f6ce-1aec-465f-885e-637117a22cce-trusted-ca-bundle\") pod \"console-6db9bc8779-v7b2x\" (UID: \"4d71f6ce-1aec-465f-885e-637117a22cce\") " pod="openshift-console/console-6db9bc8779-v7b2x"
May 25 12:22:35.366021 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:35.365943    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4d71f6ce-1aec-465f-885e-637117a22cce-service-ca\") pod \"console-6db9bc8779-v7b2x\" (UID: \"4d71f6ce-1aec-465f-885e-637117a22cce\") " pod="openshift-console/console-6db9bc8779-v7b2x"
May 25 12:22:35.366021 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:35.365960    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4d71f6ce-1aec-465f-885e-637117a22cce-oauth-serving-cert\") pod \"console-6db9bc8779-v7b2x\" (UID: \"4d71f6ce-1aec-465f-885e-637117a22cce\") " pod="openshift-console/console-6db9bc8779-v7b2x"
May 25 12:22:35.366021 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:35.365978    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-p8qg6\" (UniqueName: \"kubernetes.io/projected/4d71f6ce-1aec-465f-885e-637117a22cce-kube-api-access-p8qg6\") pod \"console-6db9bc8779-v7b2x\" (UID: \"4d71f6ce-1aec-465f-885e-637117a22cce\") " pod="openshift-console/console-6db9bc8779-v7b2x"
May 25 12:22:35.366021 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:35.366013    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4d71f6ce-1aec-465f-885e-637117a22cce-console-config\") pod \"console-6db9bc8779-v7b2x\" (UID: \"4d71f6ce-1aec-465f-885e-637117a22cce\") " pod="openshift-console/console-6db9bc8779-v7b2x"
May 25 12:22:35.366335 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:35.366300    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4d71f6ce-1aec-465f-885e-637117a22cce-console-oauth-config\") pod \"console-6db9bc8779-v7b2x\" (UID: \"4d71f6ce-1aec-465f-885e-637117a22cce\") " pod="openshift-console/console-6db9bc8779-v7b2x"
May 25 12:22:35.366766 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:35.366737    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4d71f6ce-1aec-465f-885e-637117a22cce-oauth-serving-cert\") pod \"console-6db9bc8779-v7b2x\" (UID: \"4d71f6ce-1aec-465f-885e-637117a22cce\") " pod="openshift-console/console-6db9bc8779-v7b2x"
May 25 12:22:35.366908 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:35.366873    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4d71f6ce-1aec-465f-885e-637117a22cce-console-config\") pod \"console-6db9bc8779-v7b2x\" (UID: \"4d71f6ce-1aec-465f-885e-637117a22cce\") " pod="openshift-console/console-6db9bc8779-v7b2x"
May 25 12:22:35.367006 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:35.366930    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d71f6ce-1aec-465f-885e-637117a22cce-trusted-ca-bundle\") pod \"console-6db9bc8779-v7b2x\" (UID: \"4d71f6ce-1aec-465f-885e-637117a22cce\") " pod="openshift-console/console-6db9bc8779-v7b2x"
May 25 12:22:35.367006 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:35.366938    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4d71f6ce-1aec-465f-885e-637117a22cce-service-ca\") pod \"console-6db9bc8779-v7b2x\" (UID: \"4d71f6ce-1aec-465f-885e-637117a22cce\") " pod="openshift-console/console-6db9bc8779-v7b2x"
May 25 12:22:35.368354 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:35.368334    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d71f6ce-1aec-465f-885e-637117a22cce-console-serving-cert\") pod \"console-6db9bc8779-v7b2x\" (UID: \"4d71f6ce-1aec-465f-885e-637117a22cce\") " pod="openshift-console/console-6db9bc8779-v7b2x"
May 25 12:22:35.368579 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:35.368562    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4d71f6ce-1aec-465f-885e-637117a22cce-console-oauth-config\") pod \"console-6db9bc8779-v7b2x\" (UID: \"4d71f6ce-1aec-465f-885e-637117a22cce\") " pod="openshift-console/console-6db9bc8779-v7b2x"
May 25 12:22:35.373089 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:35.373066    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8qg6\" (UniqueName: \"kubernetes.io/projected/4d71f6ce-1aec-465f-885e-637117a22cce-kube-api-access-p8qg6\") pod \"console-6db9bc8779-v7b2x\" (UID: \"4d71f6ce-1aec-465f-885e-637117a22cce\") " pod="openshift-console/console-6db9bc8779-v7b2x"
May 25 12:22:35.509159 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:35.509106    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6db9bc8779-v7b2x"
May 25 12:22:35.634457 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:35.634400    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6db9bc8779-v7b2x"]
May 25 12:22:35.636730 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:22:35.636703    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d71f6ce_1aec_465f_885e_637117a22cce.slice/crio-e5f350fa6040beb341ed7cdf036f9fcf50034a3a0bc666d2fcc189ce265ff4f9 WatchSource:0}: Error finding container e5f350fa6040beb341ed7cdf036f9fcf50034a3a0bc666d2fcc189ce265ff4f9: Status 404 returned error can't find the container with id e5f350fa6040beb341ed7cdf036f9fcf50034a3a0bc666d2fcc189ce265ff4f9
May 25 12:22:36.180728 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:36.180701    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-6768985688-xwcgw"
May 25 12:22:36.270260 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:36.270221    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6db9bc8779-v7b2x" event={"ID":"4d71f6ce-1aec-465f-885e-637117a22cce","Type":"ContainerStarted","Data":"5f278e2733aa047b6425877585c251fb7f32247ac260e306ca2b5d72ffb0f6fd"}
May 25 12:22:36.270615 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:36.270266    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6db9bc8779-v7b2x" event={"ID":"4d71f6ce-1aec-465f-885e-637117a22cce","Type":"ContainerStarted","Data":"e5f350fa6040beb341ed7cdf036f9fcf50034a3a0bc666d2fcc189ce265ff4f9"}
May 25 12:22:36.286313 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:36.286265    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-6db9bc8779-v7b2x" podStartSLOduration=1.286251983 podStartE2EDuration="1.286251983s" podCreationTimestamp="2026-05-25 12:22:35 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:22:36.285162612 +0000 UTC m=+569.245213018" watchObservedRunningTime="2026-05-25 12:22:36.286251983 +0000 UTC m=+569.246302389"
May 25 12:22:37.013901 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:37.013867    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-6768985688-xwcgw"]
May 25 12:22:37.014089 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:37.014063    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve/kserve-controller-manager-6768985688-xwcgw" podUID="203e26c3-1be4-43d9-9402-4d1f190909a8" containerName="manager" containerID="cri-o://3f7928b6173643e6e5316654f95f70a602d9bdffce32960a21c0e61e866968b5" gracePeriod=10
May 25 12:22:37.038074 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:37.038048    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-6768985688-7fsm9"]
May 25 12:22:37.085658 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:37.085613    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6768985688-7fsm9"]
May 25 12:22:37.085781 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:37.085766    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6768985688-7fsm9"
May 25 12:22:37.180456 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:37.180427    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdwk7\" (UniqueName: \"kubernetes.io/projected/dfcbe65c-9d64-4170-99b4-5f05ae8bcbf4-kube-api-access-hdwk7\") pod \"kserve-controller-manager-6768985688-7fsm9\" (UID: \"dfcbe65c-9d64-4170-99b4-5f05ae8bcbf4\") " pod="kserve/kserve-controller-manager-6768985688-7fsm9"
May 25 12:22:37.180570 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:37.180476    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dfcbe65c-9d64-4170-99b4-5f05ae8bcbf4-cert\") pod \"kserve-controller-manager-6768985688-7fsm9\" (UID: \"dfcbe65c-9d64-4170-99b4-5f05ae8bcbf4\") " pod="kserve/kserve-controller-manager-6768985688-7fsm9"
May 25 12:22:37.275069 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:37.275033    2555 generic.go:358] "Generic (PLEG): container finished" podID="203e26c3-1be4-43d9-9402-4d1f190909a8" containerID="3f7928b6173643e6e5316654f95f70a602d9bdffce32960a21c0e61e866968b5" exitCode=0
May 25 12:22:37.275427 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:37.275103    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6768985688-xwcgw" event={"ID":"203e26c3-1be4-43d9-9402-4d1f190909a8","Type":"ContainerDied","Data":"3f7928b6173643e6e5316654f95f70a602d9bdffce32960a21c0e61e866968b5"}
May 25 12:22:37.275427 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:37.275132    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6768985688-xwcgw" event={"ID":"203e26c3-1be4-43d9-9402-4d1f190909a8","Type":"ContainerDied","Data":"74d96d9efb4b1427ddf970d4aeffaab17514790710fa3061f80172174f78087c"}
May 25 12:22:37.275427 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:37.275154    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74d96d9efb4b1427ddf970d4aeffaab17514790710fa3061f80172174f78087c"
May 25 12:22:37.279618 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:37.279598    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6768985688-xwcgw"
May 25 12:22:37.281931 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:37.281910    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dfcbe65c-9d64-4170-99b4-5f05ae8bcbf4-cert\") pod \"kserve-controller-manager-6768985688-7fsm9\" (UID: \"dfcbe65c-9d64-4170-99b4-5f05ae8bcbf4\") " pod="kserve/kserve-controller-manager-6768985688-7fsm9"
May 25 12:22:37.282022 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:37.281984    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hdwk7\" (UniqueName: \"kubernetes.io/projected/dfcbe65c-9d64-4170-99b4-5f05ae8bcbf4-kube-api-access-hdwk7\") pod \"kserve-controller-manager-6768985688-7fsm9\" (UID: \"dfcbe65c-9d64-4170-99b4-5f05ae8bcbf4\") " pod="kserve/kserve-controller-manager-6768985688-7fsm9"
May 25 12:22:37.284176 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:37.284154    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dfcbe65c-9d64-4170-99b4-5f05ae8bcbf4-cert\") pod \"kserve-controller-manager-6768985688-7fsm9\" (UID: \"dfcbe65c-9d64-4170-99b4-5f05ae8bcbf4\") " pod="kserve/kserve-controller-manager-6768985688-7fsm9"
May 25 12:22:37.288269 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:37.288248    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdwk7\" (UniqueName: \"kubernetes.io/projected/dfcbe65c-9d64-4170-99b4-5f05ae8bcbf4-kube-api-access-hdwk7\") pod \"kserve-controller-manager-6768985688-7fsm9\" (UID: \"dfcbe65c-9d64-4170-99b4-5f05ae8bcbf4\") " pod="kserve/kserve-controller-manager-6768985688-7fsm9"
May 25 12:22:37.383216 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:37.383191    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7f4gh\" (UniqueName: \"kubernetes.io/projected/203e26c3-1be4-43d9-9402-4d1f190909a8-kube-api-access-7f4gh\") pod \"203e26c3-1be4-43d9-9402-4d1f190909a8\" (UID: \"203e26c3-1be4-43d9-9402-4d1f190909a8\") "
May 25 12:22:37.383357 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:37.383259    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/203e26c3-1be4-43d9-9402-4d1f190909a8-cert\") pod \"203e26c3-1be4-43d9-9402-4d1f190909a8\" (UID: \"203e26c3-1be4-43d9-9402-4d1f190909a8\") "
May 25 12:22:37.385297 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:37.385271    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/203e26c3-1be4-43d9-9402-4d1f190909a8-cert" (OuterVolumeSpecName: "cert") pod "203e26c3-1be4-43d9-9402-4d1f190909a8" (UID: "203e26c3-1be4-43d9-9402-4d1f190909a8"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:22:37.385432 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:37.385409    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/203e26c3-1be4-43d9-9402-4d1f190909a8-kube-api-access-7f4gh" (OuterVolumeSpecName: "kube-api-access-7f4gh") pod "203e26c3-1be4-43d9-9402-4d1f190909a8" (UID: "203e26c3-1be4-43d9-9402-4d1f190909a8"). InnerVolumeSpecName "kube-api-access-7f4gh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:22:37.450756 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:37.450730    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6768985688-7fsm9"
May 25 12:22:37.484686 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:37.484630    2555 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/203e26c3-1be4-43d9-9402-4d1f190909a8-cert\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:22:37.484810 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:37.484690    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f4gh\" (UniqueName: \"kubernetes.io/projected/203e26c3-1be4-43d9-9402-4d1f190909a8-kube-api-access-7f4gh\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:22:37.568624 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:37.568595    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6768985688-7fsm9"]
May 25 12:22:37.571104 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:22:37.571078    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfcbe65c_9d64_4170_99b4_5f05ae8bcbf4.slice/crio-c055dc6427aecfe12a6a0dafe344fc5a3f5265569862a68ee84abb5653855dda WatchSource:0}: Error finding container c055dc6427aecfe12a6a0dafe344fc5a3f5265569862a68ee84abb5653855dda: Status 404 returned error can't find the container with id c055dc6427aecfe12a6a0dafe344fc5a3f5265569862a68ee84abb5653855dda
May 25 12:22:38.280300 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:38.280268    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6768985688-7fsm9" event={"ID":"dfcbe65c-9d64-4170-99b4-5f05ae8bcbf4","Type":"ContainerStarted","Data":"c055dc6427aecfe12a6a0dafe344fc5a3f5265569862a68ee84abb5653855dda"}
May 25 12:22:38.280588 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:38.280303    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6768985688-xwcgw"
May 25 12:22:38.295843 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:38.295802    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-6768985688-xwcgw"]
May 25 12:22:38.302552 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:38.302522    2555 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-6768985688-xwcgw"]
May 25 12:22:39.285374 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:39.285335    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6768985688-7fsm9" event={"ID":"dfcbe65c-9d64-4170-99b4-5f05ae8bcbf4","Type":"ContainerStarted","Data":"9abf254ce08bb4f3c8c6205310d940a6390a7d94e3634f5150c179d5a18c44b4"}
May 25 12:22:39.285806 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:39.285471    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-6768985688-7fsm9"
May 25 12:22:39.300221 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:39.300174    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/kserve-controller-manager-6768985688-7fsm9" podStartSLOduration=1.684733966 podStartE2EDuration="2.300162189s" podCreationTimestamp="2026-05-25 12:22:37 +0000 UTC" firstStartedPulling="2026-05-25 12:22:37.572429362 +0000 UTC m=+570.532479762" lastFinishedPulling="2026-05-25 12:22:38.187857601 +0000 UTC m=+571.147907985" observedRunningTime="2026-05-25 12:22:39.300098434 +0000 UTC m=+572.260148841" watchObservedRunningTime="2026-05-25 12:22:39.300162189 +0000 UTC m=+572.260212595"
May 25 12:22:39.598620 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:39.598548    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="203e26c3-1be4-43d9-9402-4d1f190909a8" path="/var/lib/kubelet/pods/203e26c3-1be4-43d9-9402-4d1f190909a8/volumes"
May 25 12:22:45.509823 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:45.509791    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6db9bc8779-v7b2x"
May 25 12:22:45.510235 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:45.509834    2555 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6db9bc8779-v7b2x"
May 25 12:22:45.514165 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:45.514140    2555 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6db9bc8779-v7b2x"
May 25 12:22:46.311781 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:46.311750    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6db9bc8779-v7b2x"
May 25 12:22:46.351459 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:22:46.351428    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8469d49f4b-ppb42"]
May 25 12:23:07.506344 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:07.506315    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-2tqtm_00e6f79c-8f05-4330-aa16-1ada2ae9e9cc/console-operator/1.log"
May 25 12:23:07.507782 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:07.507760    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-2tqtm_00e6f79c-8f05-4330-aa16-1ada2ae9e9cc/console-operator/1.log"
May 25 12:23:10.294159 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:10.294132    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-6768985688-7fsm9"
May 25 12:23:11.186534 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:11.186502    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-86f7b4b499-x47mq"]
May 25 12:23:11.186922 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:11.186904    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="203e26c3-1be4-43d9-9402-4d1f190909a8" containerName="manager"
May 25 12:23:11.187011 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:11.186924    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="203e26c3-1be4-43d9-9402-4d1f190909a8" containerName="manager"
May 25 12:23:11.187103 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:11.187021    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="203e26c3-1be4-43d9-9402-4d1f190909a8" containerName="manager"
May 25 12:23:11.189661 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:11.189624    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-86f7b4b499-x47mq"
May 25 12:23:11.191956 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:11.191930    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
May 25 12:23:11.192052 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:11.191959    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-ggrhz\""
May 25 12:23:11.197810 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:11.197781    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-86f7b4b499-x47mq"]
May 25 12:23:11.335500 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:11.335477    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a8f4405-598f-4067-8baa-22ae0ebba087-tls-certs\") pod \"model-serving-api-86f7b4b499-x47mq\" (UID: \"6a8f4405-598f-4067-8baa-22ae0ebba087\") " pod="kserve/model-serving-api-86f7b4b499-x47mq"
May 25 12:23:11.335838 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:11.335517    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llkh7\" (UniqueName: \"kubernetes.io/projected/6a8f4405-598f-4067-8baa-22ae0ebba087-kube-api-access-llkh7\") pod \"model-serving-api-86f7b4b499-x47mq\" (UID: \"6a8f4405-598f-4067-8baa-22ae0ebba087\") " pod="kserve/model-serving-api-86f7b4b499-x47mq"
May 25 12:23:11.370943 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:11.370892    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-8469d49f4b-ppb42" podUID="deb97e2d-509e-40f5-bb42-611cb81f5b18" containerName="console" containerID="cri-o://8ec2914c30a062149d65d86b159b6494518f37f27805513c68bbf72ca139bc50" gracePeriod=15
May 25 12:23:11.436628 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:11.436574    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a8f4405-598f-4067-8baa-22ae0ebba087-tls-certs\") pod \"model-serving-api-86f7b4b499-x47mq\" (UID: \"6a8f4405-598f-4067-8baa-22ae0ebba087\") " pod="kserve/model-serving-api-86f7b4b499-x47mq"
May 25 12:23:11.436628 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:11.436615    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-llkh7\" (UniqueName: \"kubernetes.io/projected/6a8f4405-598f-4067-8baa-22ae0ebba087-kube-api-access-llkh7\") pod \"model-serving-api-86f7b4b499-x47mq\" (UID: \"6a8f4405-598f-4067-8baa-22ae0ebba087\") " pod="kserve/model-serving-api-86f7b4b499-x47mq"
May 25 12:23:11.436762 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:23:11.436735    2555 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
May 25 12:23:11.436824 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:23:11.436812    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a8f4405-598f-4067-8baa-22ae0ebba087-tls-certs podName:6a8f4405-598f-4067-8baa-22ae0ebba087 nodeName:}" failed. No retries permitted until 2026-05-25 12:23:11.936795057 +0000 UTC m=+604.896845445 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6a8f4405-598f-4067-8baa-22ae0ebba087-tls-certs") pod "model-serving-api-86f7b4b499-x47mq" (UID: "6a8f4405-598f-4067-8baa-22ae0ebba087") : secret "model-serving-api-tls" not found
May 25 12:23:11.444354 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:11.444330    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llkh7\" (UniqueName: \"kubernetes.io/projected/6a8f4405-598f-4067-8baa-22ae0ebba087-kube-api-access-llkh7\") pod \"model-serving-api-86f7b4b499-x47mq\" (UID: \"6a8f4405-598f-4067-8baa-22ae0ebba087\") " pod="kserve/model-serving-api-86f7b4b499-x47mq"
May 25 12:23:11.617566 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:11.617543    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-8469d49f4b-ppb42_deb97e2d-509e-40f5-bb42-611cb81f5b18/console/0.log"
May 25 12:23:11.617712 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:11.617601    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8469d49f4b-ppb42"
May 25 12:23:11.738267 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:11.738206    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/deb97e2d-509e-40f5-bb42-611cb81f5b18-service-ca\") pod \"deb97e2d-509e-40f5-bb42-611cb81f5b18\" (UID: \"deb97e2d-509e-40f5-bb42-611cb81f5b18\") "
May 25 12:23:11.738267 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:11.738241    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/deb97e2d-509e-40f5-bb42-611cb81f5b18-console-serving-cert\") pod \"deb97e2d-509e-40f5-bb42-611cb81f5b18\" (UID: \"deb97e2d-509e-40f5-bb42-611cb81f5b18\") "
May 25 12:23:11.738445 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:11.738315    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/deb97e2d-509e-40f5-bb42-611cb81f5b18-trusted-ca-bundle\") pod \"deb97e2d-509e-40f5-bb42-611cb81f5b18\" (UID: \"deb97e2d-509e-40f5-bb42-611cb81f5b18\") "
May 25 12:23:11.738445 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:11.738330    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/deb97e2d-509e-40f5-bb42-611cb81f5b18-oauth-serving-cert\") pod \"deb97e2d-509e-40f5-bb42-611cb81f5b18\" (UID: \"deb97e2d-509e-40f5-bb42-611cb81f5b18\") "
May 25 12:23:11.738445 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:11.738352    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/deb97e2d-509e-40f5-bb42-611cb81f5b18-console-oauth-config\") pod \"deb97e2d-509e-40f5-bb42-611cb81f5b18\" (UID: \"deb97e2d-509e-40f5-bb42-611cb81f5b18\") "
May 25 12:23:11.738445 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:11.738377    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/deb97e2d-509e-40f5-bb42-611cb81f5b18-console-config\") pod \"deb97e2d-509e-40f5-bb42-611cb81f5b18\" (UID: \"deb97e2d-509e-40f5-bb42-611cb81f5b18\") "
May 25 12:23:11.738445 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:11.738413    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rt4g6\" (UniqueName: \"kubernetes.io/projected/deb97e2d-509e-40f5-bb42-611cb81f5b18-kube-api-access-rt4g6\") pod \"deb97e2d-509e-40f5-bb42-611cb81f5b18\" (UID: \"deb97e2d-509e-40f5-bb42-611cb81f5b18\") "
May 25 12:23:11.738800 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:11.738757    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deb97e2d-509e-40f5-bb42-611cb81f5b18-service-ca" (OuterVolumeSpecName: "service-ca") pod "deb97e2d-509e-40f5-bb42-611cb81f5b18" (UID: "deb97e2d-509e-40f5-bb42-611cb81f5b18"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:23:11.738919 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:11.738895    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deb97e2d-509e-40f5-bb42-611cb81f5b18-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "deb97e2d-509e-40f5-bb42-611cb81f5b18" (UID: "deb97e2d-509e-40f5-bb42-611cb81f5b18"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:23:11.738919 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:11.738905    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deb97e2d-509e-40f5-bb42-611cb81f5b18-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "deb97e2d-509e-40f5-bb42-611cb81f5b18" (UID: "deb97e2d-509e-40f5-bb42-611cb81f5b18"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:23:11.739036 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:11.738916    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deb97e2d-509e-40f5-bb42-611cb81f5b18-console-config" (OuterVolumeSpecName: "console-config") pod "deb97e2d-509e-40f5-bb42-611cb81f5b18" (UID: "deb97e2d-509e-40f5-bb42-611cb81f5b18"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:23:11.740501 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:11.740466    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/deb97e2d-509e-40f5-bb42-611cb81f5b18-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "deb97e2d-509e-40f5-bb42-611cb81f5b18" (UID: "deb97e2d-509e-40f5-bb42-611cb81f5b18"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:23:11.740581 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:11.740553    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/deb97e2d-509e-40f5-bb42-611cb81f5b18-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "deb97e2d-509e-40f5-bb42-611cb81f5b18" (UID: "deb97e2d-509e-40f5-bb42-611cb81f5b18"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:23:11.740581 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:11.740553    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deb97e2d-509e-40f5-bb42-611cb81f5b18-kube-api-access-rt4g6" (OuterVolumeSpecName: "kube-api-access-rt4g6") pod "deb97e2d-509e-40f5-bb42-611cb81f5b18" (UID: "deb97e2d-509e-40f5-bb42-611cb81f5b18"). InnerVolumeSpecName "kube-api-access-rt4g6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:23:11.839670 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:11.839620    2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/deb97e2d-509e-40f5-bb42-611cb81f5b18-trusted-ca-bundle\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:23:11.839670 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:11.839670    2555 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/deb97e2d-509e-40f5-bb42-611cb81f5b18-oauth-serving-cert\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:23:11.839670 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:11.839680    2555 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/deb97e2d-509e-40f5-bb42-611cb81f5b18-console-oauth-config\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:23:11.839833 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:11.839689    2555 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/deb97e2d-509e-40f5-bb42-611cb81f5b18-console-config\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:23:11.839833 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:11.839698    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt4g6\" (UniqueName: \"kubernetes.io/projected/deb97e2d-509e-40f5-bb42-611cb81f5b18-kube-api-access-rt4g6\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:23:11.839833 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:11.839707    2555 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/deb97e2d-509e-40f5-bb42-611cb81f5b18-service-ca\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:23:11.839833 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:11.839715    2555 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/deb97e2d-509e-40f5-bb42-611cb81f5b18-console-serving-cert\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:23:11.940917 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:11.940894    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a8f4405-598f-4067-8baa-22ae0ebba087-tls-certs\") pod \"model-serving-api-86f7b4b499-x47mq\" (UID: \"6a8f4405-598f-4067-8baa-22ae0ebba087\") " pod="kserve/model-serving-api-86f7b4b499-x47mq"
May 25 12:23:11.943095 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:11.943069    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a8f4405-598f-4067-8baa-22ae0ebba087-tls-certs\") pod \"model-serving-api-86f7b4b499-x47mq\" (UID: \"6a8f4405-598f-4067-8baa-22ae0ebba087\") " pod="kserve/model-serving-api-86f7b4b499-x47mq"
May 25 12:23:12.099927 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:12.099871    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-86f7b4b499-x47mq"
May 25 12:23:12.215900 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:12.215875    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-86f7b4b499-x47mq"]
May 25 12:23:12.218462 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:23:12.218433    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a8f4405_598f_4067_8baa_22ae0ebba087.slice/crio-e3f8118f45fdfa32b4ccf3e5f521f932b35d3813142fc4036c007f1f367af05e WatchSource:0}: Error finding container e3f8118f45fdfa32b4ccf3e5f521f932b35d3813142fc4036c007f1f367af05e: Status 404 returned error can't find the container with id e3f8118f45fdfa32b4ccf3e5f521f932b35d3813142fc4036c007f1f367af05e
May 25 12:23:12.394574 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:12.394514    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-86f7b4b499-x47mq" event={"ID":"6a8f4405-598f-4067-8baa-22ae0ebba087","Type":"ContainerStarted","Data":"e3f8118f45fdfa32b4ccf3e5f521f932b35d3813142fc4036c007f1f367af05e"}
May 25 12:23:12.395634 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:12.395617    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-8469d49f4b-ppb42_deb97e2d-509e-40f5-bb42-611cb81f5b18/console/0.log"
May 25 12:23:12.395750 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:12.395668    2555 generic.go:358] "Generic (PLEG): container finished" podID="deb97e2d-509e-40f5-bb42-611cb81f5b18" containerID="8ec2914c30a062149d65d86b159b6494518f37f27805513c68bbf72ca139bc50" exitCode=2
May 25 12:23:12.395750 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:12.395699    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8469d49f4b-ppb42" event={"ID":"deb97e2d-509e-40f5-bb42-611cb81f5b18","Type":"ContainerDied","Data":"8ec2914c30a062149d65d86b159b6494518f37f27805513c68bbf72ca139bc50"}
May 25 12:23:12.395750 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:12.395737    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8469d49f4b-ppb42" event={"ID":"deb97e2d-509e-40f5-bb42-611cb81f5b18","Type":"ContainerDied","Data":"09a859a344da9f8a3617948c0466ba324b1606e17bdfe2dfe9940f042d9b38b3"}
May 25 12:23:12.395898 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:12.395746    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8469d49f4b-ppb42"
May 25 12:23:12.395898 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:12.395754    2555 scope.go:117] "RemoveContainer" containerID="8ec2914c30a062149d65d86b159b6494518f37f27805513c68bbf72ca139bc50"
May 25 12:23:12.403552 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:12.403537    2555 scope.go:117] "RemoveContainer" containerID="8ec2914c30a062149d65d86b159b6494518f37f27805513c68bbf72ca139bc50"
May 25 12:23:12.403778 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:23:12.403760    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ec2914c30a062149d65d86b159b6494518f37f27805513c68bbf72ca139bc50\": container with ID starting with 8ec2914c30a062149d65d86b159b6494518f37f27805513c68bbf72ca139bc50 not found: ID does not exist" containerID="8ec2914c30a062149d65d86b159b6494518f37f27805513c68bbf72ca139bc50"
May 25 12:23:12.403842 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:12.403786    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ec2914c30a062149d65d86b159b6494518f37f27805513c68bbf72ca139bc50"} err="failed to get container status \"8ec2914c30a062149d65d86b159b6494518f37f27805513c68bbf72ca139bc50\": rpc error: code = NotFound desc = could not find container \"8ec2914c30a062149d65d86b159b6494518f37f27805513c68bbf72ca139bc50\": container with ID starting with 8ec2914c30a062149d65d86b159b6494518f37f27805513c68bbf72ca139bc50 not found: ID does not exist"
May 25 12:23:12.415404 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:12.415383    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8469d49f4b-ppb42"]
May 25 12:23:12.419138 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:12.419120    2555 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8469d49f4b-ppb42"]
May 25 12:23:13.599395 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:13.599367    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="deb97e2d-509e-40f5-bb42-611cb81f5b18" path="/var/lib/kubelet/pods/deb97e2d-509e-40f5-bb42-611cb81f5b18/volumes"
May 25 12:23:15.408503 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:15.408462    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-86f7b4b499-x47mq" event={"ID":"6a8f4405-598f-4067-8baa-22ae0ebba087","Type":"ContainerStarted","Data":"4b53e94705c92e40e6b53c759758f8be57dcb50a6c19619b5a11eb460ed745d9"}
May 25 12:23:15.408962 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:15.408621    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-86f7b4b499-x47mq"
May 25 12:23:15.429017 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:15.428953    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/model-serving-api-86f7b4b499-x47mq" podStartSLOduration=2.006640498 podStartE2EDuration="4.428928583s" podCreationTimestamp="2026-05-25 12:23:11 +0000 UTC" firstStartedPulling="2026-05-25 12:23:12.220190646 +0000 UTC m=+605.180241030" lastFinishedPulling="2026-05-25 12:23:14.642478731 +0000 UTC m=+607.602529115" observedRunningTime="2026-05-25 12:23:15.426566131 +0000 UTC m=+608.386616539" watchObservedRunningTime="2026-05-25 12:23:15.428928583 +0000 UTC m=+608.388978992"
May 25 12:23:26.416120 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:26.416091    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-86f7b4b499-x47mq"
May 25 12:23:37.347864 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:37.347809    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-ddd4dbfd-qgdh2"]
May 25 12:23:37.350160 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:37.348165    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="deb97e2d-509e-40f5-bb42-611cb81f5b18" containerName="console"
May 25 12:23:37.350160 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:37.348176    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="deb97e2d-509e-40f5-bb42-611cb81f5b18" containerName="console"
May 25 12:23:37.350160 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:37.348226    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="deb97e2d-509e-40f5-bb42-611cb81f5b18" containerName="console"
May 25 12:23:37.350985 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:37.350967    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-ddd4dbfd-qgdh2"
May 25 12:23:37.353085 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:37.353065    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
May 25 12:23:37.358576 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:37.358551    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-ddd4dbfd-qgdh2"]
May 25 12:23:37.444040 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:37.444008    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ef53c80c-ef88-4146-a4fc-95f39e0f20f8-data\") pod \"seaweedfs-tls-custom-ddd4dbfd-qgdh2\" (UID: \"ef53c80c-ef88-4146-a4fc-95f39e0f20f8\") " pod="kserve/seaweedfs-tls-custom-ddd4dbfd-qgdh2"
May 25 12:23:37.444165 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:37.444057    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj7sq\" (UniqueName: \"kubernetes.io/projected/ef53c80c-ef88-4146-a4fc-95f39e0f20f8-kube-api-access-lj7sq\") pod \"seaweedfs-tls-custom-ddd4dbfd-qgdh2\" (UID: \"ef53c80c-ef88-4146-a4fc-95f39e0f20f8\") " pod="kserve/seaweedfs-tls-custom-ddd4dbfd-qgdh2"
May 25 12:23:37.545279 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:37.545247    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lj7sq\" (UniqueName: \"kubernetes.io/projected/ef53c80c-ef88-4146-a4fc-95f39e0f20f8-kube-api-access-lj7sq\") pod \"seaweedfs-tls-custom-ddd4dbfd-qgdh2\" (UID: \"ef53c80c-ef88-4146-a4fc-95f39e0f20f8\") " pod="kserve/seaweedfs-tls-custom-ddd4dbfd-qgdh2"
May 25 12:23:37.545428 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:37.545313    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ef53c80c-ef88-4146-a4fc-95f39e0f20f8-data\") pod \"seaweedfs-tls-custom-ddd4dbfd-qgdh2\" (UID: \"ef53c80c-ef88-4146-a4fc-95f39e0f20f8\") " pod="kserve/seaweedfs-tls-custom-ddd4dbfd-qgdh2"
May 25 12:23:37.545630 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:37.545614    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ef53c80c-ef88-4146-a4fc-95f39e0f20f8-data\") pod \"seaweedfs-tls-custom-ddd4dbfd-qgdh2\" (UID: \"ef53c80c-ef88-4146-a4fc-95f39e0f20f8\") " pod="kserve/seaweedfs-tls-custom-ddd4dbfd-qgdh2"
May 25 12:23:37.553147 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:37.553125    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj7sq\" (UniqueName: \"kubernetes.io/projected/ef53c80c-ef88-4146-a4fc-95f39e0f20f8-kube-api-access-lj7sq\") pod \"seaweedfs-tls-custom-ddd4dbfd-qgdh2\" (UID: \"ef53c80c-ef88-4146-a4fc-95f39e0f20f8\") " pod="kserve/seaweedfs-tls-custom-ddd4dbfd-qgdh2"
May 25 12:23:37.661313 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:37.661241    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-ddd4dbfd-qgdh2"
May 25 12:23:37.779762 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:37.779736    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-ddd4dbfd-qgdh2"]
May 25 12:23:37.782294 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:23:37.782257    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef53c80c_ef88_4146_a4fc_95f39e0f20f8.slice/crio-d5ca9241acd356a9e5d3406c233ee36b96dee1cee4936b0885156b213d679dde WatchSource:0}: Error finding container d5ca9241acd356a9e5d3406c233ee36b96dee1cee4936b0885156b213d679dde: Status 404 returned error can't find the container with id d5ca9241acd356a9e5d3406c233ee36b96dee1cee4936b0885156b213d679dde
May 25 12:23:37.783500 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:37.783477    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 12:23:38.486976 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:38.486940    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-ddd4dbfd-qgdh2" event={"ID":"ef53c80c-ef88-4146-a4fc-95f39e0f20f8","Type":"ContainerStarted","Data":"a9186a084a687ad40582ac2194ead673f38dc9e8fb336586e70e7b9c5c8f2e94"}
May 25 12:23:38.486976 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:38.486979    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-ddd4dbfd-qgdh2" event={"ID":"ef53c80c-ef88-4146-a4fc-95f39e0f20f8","Type":"ContainerStarted","Data":"d5ca9241acd356a9e5d3406c233ee36b96dee1cee4936b0885156b213d679dde"}
May 25 12:23:38.501508 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:38.501465    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-ddd4dbfd-qgdh2" podStartSLOduration=1.193234975 podStartE2EDuration="1.501451595s" podCreationTimestamp="2026-05-25 12:23:37 +0000 UTC" firstStartedPulling="2026-05-25 12:23:37.783595417 +0000 UTC m=+630.743645804" lastFinishedPulling="2026-05-25 12:23:38.091812039 +0000 UTC m=+631.051862424" observedRunningTime="2026-05-25 12:23:38.500468875 +0000 UTC m=+631.460519280" watchObservedRunningTime="2026-05-25 12:23:38.501451595 +0000 UTC m=+631.461502000"
May 25 12:23:39.827719 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:39.827684    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-ddd4dbfd-qgdh2"]
May 25 12:23:40.493201 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:40.493156    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve/seaweedfs-tls-custom-ddd4dbfd-qgdh2" podUID="ef53c80c-ef88-4146-a4fc-95f39e0f20f8" containerName="seaweedfs-tls-custom" containerID="cri-o://a9186a084a687ad40582ac2194ead673f38dc9e8fb336586e70e7b9c5c8f2e94" gracePeriod=30
May 25 12:23:41.735707 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:41.735679    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-ddd4dbfd-qgdh2"
May 25 12:23:41.883156 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:41.883074    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ef53c80c-ef88-4146-a4fc-95f39e0f20f8-data\") pod \"ef53c80c-ef88-4146-a4fc-95f39e0f20f8\" (UID: \"ef53c80c-ef88-4146-a4fc-95f39e0f20f8\") "
May 25 12:23:41.883156 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:41.883108    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lj7sq\" (UniqueName: \"kubernetes.io/projected/ef53c80c-ef88-4146-a4fc-95f39e0f20f8-kube-api-access-lj7sq\") pod \"ef53c80c-ef88-4146-a4fc-95f39e0f20f8\" (UID: \"ef53c80c-ef88-4146-a4fc-95f39e0f20f8\") "
May 25 12:23:41.884395 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:41.884372    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef53c80c-ef88-4146-a4fc-95f39e0f20f8-data" (OuterVolumeSpecName: "data") pod "ef53c80c-ef88-4146-a4fc-95f39e0f20f8" (UID: "ef53c80c-ef88-4146-a4fc-95f39e0f20f8"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:23:41.885067 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:41.885044    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef53c80c-ef88-4146-a4fc-95f39e0f20f8-kube-api-access-lj7sq" (OuterVolumeSpecName: "kube-api-access-lj7sq") pod "ef53c80c-ef88-4146-a4fc-95f39e0f20f8" (UID: "ef53c80c-ef88-4146-a4fc-95f39e0f20f8"). InnerVolumeSpecName "kube-api-access-lj7sq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:23:41.983980 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:41.983950    2555 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ef53c80c-ef88-4146-a4fc-95f39e0f20f8-data\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:23:41.983980 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:41.983977    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lj7sq\" (UniqueName: \"kubernetes.io/projected/ef53c80c-ef88-4146-a4fc-95f39e0f20f8-kube-api-access-lj7sq\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:23:42.500429 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:42.500389    2555 generic.go:358] "Generic (PLEG): container finished" podID="ef53c80c-ef88-4146-a4fc-95f39e0f20f8" containerID="a9186a084a687ad40582ac2194ead673f38dc9e8fb336586e70e7b9c5c8f2e94" exitCode=0
May 25 12:23:42.500589 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:42.500451    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-ddd4dbfd-qgdh2"
May 25 12:23:42.500589 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:42.500450    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-ddd4dbfd-qgdh2" event={"ID":"ef53c80c-ef88-4146-a4fc-95f39e0f20f8","Type":"ContainerDied","Data":"a9186a084a687ad40582ac2194ead673f38dc9e8fb336586e70e7b9c5c8f2e94"}
May 25 12:23:42.500589 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:42.500493    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-ddd4dbfd-qgdh2" event={"ID":"ef53c80c-ef88-4146-a4fc-95f39e0f20f8","Type":"ContainerDied","Data":"d5ca9241acd356a9e5d3406c233ee36b96dee1cee4936b0885156b213d679dde"}
May 25 12:23:42.500589 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:42.500514    2555 scope.go:117] "RemoveContainer" containerID="a9186a084a687ad40582ac2194ead673f38dc9e8fb336586e70e7b9c5c8f2e94"
May 25 12:23:42.510155 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:42.510141    2555 scope.go:117] "RemoveContainer" containerID="a9186a084a687ad40582ac2194ead673f38dc9e8fb336586e70e7b9c5c8f2e94"
May 25 12:23:42.510413 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:23:42.510395    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9186a084a687ad40582ac2194ead673f38dc9e8fb336586e70e7b9c5c8f2e94\": container with ID starting with a9186a084a687ad40582ac2194ead673f38dc9e8fb336586e70e7b9c5c8f2e94 not found: ID does not exist" containerID="a9186a084a687ad40582ac2194ead673f38dc9e8fb336586e70e7b9c5c8f2e94"
May 25 12:23:42.510458 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:42.510424    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9186a084a687ad40582ac2194ead673f38dc9e8fb336586e70e7b9c5c8f2e94"} err="failed to get container status \"a9186a084a687ad40582ac2194ead673f38dc9e8fb336586e70e7b9c5c8f2e94\": rpc error: code = NotFound desc = could not find container \"a9186a084a687ad40582ac2194ead673f38dc9e8fb336586e70e7b9c5c8f2e94\": container with ID starting with a9186a084a687ad40582ac2194ead673f38dc9e8fb336586e70e7b9c5c8f2e94 not found: ID does not exist"
May 25 12:23:42.520367 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:42.520329    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-ddd4dbfd-qgdh2"]
May 25 12:23:42.524836 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:42.524810    2555 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve/seaweedfs-tls-custom-ddd4dbfd-qgdh2"]
May 25 12:23:42.549187 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:42.549161    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-5c88b85bb7-ptpbn"]
May 25 12:23:42.549527 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:42.549511    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef53c80c-ef88-4146-a4fc-95f39e0f20f8" containerName="seaweedfs-tls-custom"
May 25 12:23:42.549600 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:42.549531    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef53c80c-ef88-4146-a4fc-95f39e0f20f8" containerName="seaweedfs-tls-custom"
May 25 12:23:42.549675 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:42.549600    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="ef53c80c-ef88-4146-a4fc-95f39e0f20f8" containerName="seaweedfs-tls-custom"
May 25 12:23:42.553706 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:42.553689    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-5c88b85bb7-ptpbn"
May 25 12:23:42.555482 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:42.555464    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
May 25 12:23:42.555573 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:42.555468    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom\""
May 25 12:23:42.558983 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:42.558966    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-5c88b85bb7-ptpbn"]
May 25 12:23:42.690115 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:42.690082    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/19bb1968-d459-488b-b9a8-b2bea71f370b-data\") pod \"seaweedfs-tls-custom-5c88b85bb7-ptpbn\" (UID: \"19bb1968-d459-488b-b9a8-b2bea71f370b\") " pod="kserve/seaweedfs-tls-custom-5c88b85bb7-ptpbn"
May 25 12:23:42.690260 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:42.690124    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qb4b\" (UniqueName: \"kubernetes.io/projected/19bb1968-d459-488b-b9a8-b2bea71f370b-kube-api-access-7qb4b\") pod \"seaweedfs-tls-custom-5c88b85bb7-ptpbn\" (UID: \"19bb1968-d459-488b-b9a8-b2bea71f370b\") " pod="kserve/seaweedfs-tls-custom-5c88b85bb7-ptpbn"
May 25 12:23:42.690318 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:42.690295    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/19bb1968-d459-488b-b9a8-b2bea71f370b-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-5c88b85bb7-ptpbn\" (UID: \"19bb1968-d459-488b-b9a8-b2bea71f370b\") " pod="kserve/seaweedfs-tls-custom-5c88b85bb7-ptpbn"
May 25 12:23:42.790980 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:42.790908    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/19bb1968-d459-488b-b9a8-b2bea71f370b-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-5c88b85bb7-ptpbn\" (UID: \"19bb1968-d459-488b-b9a8-b2bea71f370b\") " pod="kserve/seaweedfs-tls-custom-5c88b85bb7-ptpbn"
May 25 12:23:42.790980 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:42.790966    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/19bb1968-d459-488b-b9a8-b2bea71f370b-data\") pod \"seaweedfs-tls-custom-5c88b85bb7-ptpbn\" (UID: \"19bb1968-d459-488b-b9a8-b2bea71f370b\") " pod="kserve/seaweedfs-tls-custom-5c88b85bb7-ptpbn"
May 25 12:23:42.791333 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:42.790988    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7qb4b\" (UniqueName: \"kubernetes.io/projected/19bb1968-d459-488b-b9a8-b2bea71f370b-kube-api-access-7qb4b\") pod \"seaweedfs-tls-custom-5c88b85bb7-ptpbn\" (UID: \"19bb1968-d459-488b-b9a8-b2bea71f370b\") " pod="kserve/seaweedfs-tls-custom-5c88b85bb7-ptpbn"
May 25 12:23:42.791373 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:42.791353    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/19bb1968-d459-488b-b9a8-b2bea71f370b-data\") pod \"seaweedfs-tls-custom-5c88b85bb7-ptpbn\" (UID: \"19bb1968-d459-488b-b9a8-b2bea71f370b\") " pod="kserve/seaweedfs-tls-custom-5c88b85bb7-ptpbn"
May 25 12:23:42.793239 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:42.793219    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/19bb1968-d459-488b-b9a8-b2bea71f370b-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-5c88b85bb7-ptpbn\" (UID: \"19bb1968-d459-488b-b9a8-b2bea71f370b\") " pod="kserve/seaweedfs-tls-custom-5c88b85bb7-ptpbn"
May 25 12:23:42.798316 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:42.798294    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qb4b\" (UniqueName: \"kubernetes.io/projected/19bb1968-d459-488b-b9a8-b2bea71f370b-kube-api-access-7qb4b\") pod \"seaweedfs-tls-custom-5c88b85bb7-ptpbn\" (UID: \"19bb1968-d459-488b-b9a8-b2bea71f370b\") " pod="kserve/seaweedfs-tls-custom-5c88b85bb7-ptpbn"
May 25 12:23:42.863010 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:42.862982    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-5c88b85bb7-ptpbn"
May 25 12:23:42.984610 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:42.984586    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-5c88b85bb7-ptpbn"]
May 25 12:23:42.986998 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:23:42.986969    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19bb1968_d459_488b_b9a8_b2bea71f370b.slice/crio-2c6b97cd6fd62ebe6ac8cb30bcceb9c3c8b572be253ce4a048b6602d5e7493f6 WatchSource:0}: Error finding container 2c6b97cd6fd62ebe6ac8cb30bcceb9c3c8b572be253ce4a048b6602d5e7493f6: Status 404 returned error can't find the container with id 2c6b97cd6fd62ebe6ac8cb30bcceb9c3c8b572be253ce4a048b6602d5e7493f6
May 25 12:23:43.505391 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:43.505309    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-5c88b85bb7-ptpbn" event={"ID":"19bb1968-d459-488b-b9a8-b2bea71f370b","Type":"ContainerStarted","Data":"34c9c09b32374dabd548d974359434bc300557d8875e16696c89962c40dacf96"}
May 25 12:23:43.505391 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:43.505346    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-5c88b85bb7-ptpbn" event={"ID":"19bb1968-d459-488b-b9a8-b2bea71f370b","Type":"ContainerStarted","Data":"2c6b97cd6fd62ebe6ac8cb30bcceb9c3c8b572be253ce4a048b6602d5e7493f6"}
May 25 12:23:43.521077 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:43.521035    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-5c88b85bb7-ptpbn" podStartSLOduration=1.254355221 podStartE2EDuration="1.521022092s" podCreationTimestamp="2026-05-25 12:23:42 +0000 UTC" firstStartedPulling="2026-05-25 12:23:42.988235777 +0000 UTC m=+635.948286165" lastFinishedPulling="2026-05-25 12:23:43.254902647 +0000 UTC m=+636.214953036" observedRunningTime="2026-05-25 12:23:43.519371459 +0000 UTC m=+636.479421868" watchObservedRunningTime="2026-05-25 12:23:43.521022092 +0000 UTC m=+636.481072497"
May 25 12:23:43.599069 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:43.599038    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef53c80c-ef88-4146-a4fc-95f39e0f20f8" path="/var/lib/kubelet/pods/ef53c80c-ef88-4146-a4fc-95f39e0f20f8/volumes"
May 25 12:23:51.266978 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:51.266929    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-serving-7fd5766db9-vjht7"]
May 25 12:23:51.271625 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:51.271609    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-7fd5766db9-vjht7"
May 25 12:23:51.273508 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:51.273479    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
May 25 12:23:51.273508 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:51.273497    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving\""
May 25 12:23:51.276614 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:51.276597    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-7fd5766db9-vjht7"]
May 25 12:23:51.359719 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:51.359688    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54vzm\" (UniqueName: \"kubernetes.io/projected/931de070-ae52-45bc-aafe-0f893f3f6df4-kube-api-access-54vzm\") pod \"seaweedfs-tls-serving-7fd5766db9-vjht7\" (UID: \"931de070-ae52-45bc-aafe-0f893f3f6df4\") " pod="kserve/seaweedfs-tls-serving-7fd5766db9-vjht7"
May 25 12:23:51.359849 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:51.359745    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/931de070-ae52-45bc-aafe-0f893f3f6df4-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-7fd5766db9-vjht7\" (UID: \"931de070-ae52-45bc-aafe-0f893f3f6df4\") " pod="kserve/seaweedfs-tls-serving-7fd5766db9-vjht7"
May 25 12:23:51.359849 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:51.359804    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/931de070-ae52-45bc-aafe-0f893f3f6df4-data\") pod \"seaweedfs-tls-serving-7fd5766db9-vjht7\" (UID: \"931de070-ae52-45bc-aafe-0f893f3f6df4\") " pod="kserve/seaweedfs-tls-serving-7fd5766db9-vjht7"
May 25 12:23:51.460537 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:51.460507    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/931de070-ae52-45bc-aafe-0f893f3f6df4-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-7fd5766db9-vjht7\" (UID: \"931de070-ae52-45bc-aafe-0f893f3f6df4\") " pod="kserve/seaweedfs-tls-serving-7fd5766db9-vjht7"
May 25 12:23:51.460714 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:51.460550    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/931de070-ae52-45bc-aafe-0f893f3f6df4-data\") pod \"seaweedfs-tls-serving-7fd5766db9-vjht7\" (UID: \"931de070-ae52-45bc-aafe-0f893f3f6df4\") " pod="kserve/seaweedfs-tls-serving-7fd5766db9-vjht7"
May 25 12:23:51.460714 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:51.460622    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-54vzm\" (UniqueName: \"kubernetes.io/projected/931de070-ae52-45bc-aafe-0f893f3f6df4-kube-api-access-54vzm\") pod \"seaweedfs-tls-serving-7fd5766db9-vjht7\" (UID: \"931de070-ae52-45bc-aafe-0f893f3f6df4\") " pod="kserve/seaweedfs-tls-serving-7fd5766db9-vjht7"
May 25 12:23:51.460714 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:23:51.460667    2555 projected.go:264] Couldn't get secret kserve/seaweedfs-tls-serving: secret "seaweedfs-tls-serving" not found
May 25 12:23:51.460714 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:23:51.460702    2555 projected.go:194] Error preparing data for projected volume seaweedfs-tls-serving for pod kserve/seaweedfs-tls-serving-7fd5766db9-vjht7: secret "seaweedfs-tls-serving" not found
May 25 12:23:51.460955 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:23:51.460780    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/931de070-ae52-45bc-aafe-0f893f3f6df4-seaweedfs-tls-serving podName:931de070-ae52-45bc-aafe-0f893f3f6df4 nodeName:}" failed. No retries permitted until 2026-05-25 12:23:51.960757962 +0000 UTC m=+644.920808348 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "seaweedfs-tls-serving" (UniqueName: "kubernetes.io/projected/931de070-ae52-45bc-aafe-0f893f3f6df4-seaweedfs-tls-serving") pod "seaweedfs-tls-serving-7fd5766db9-vjht7" (UID: "931de070-ae52-45bc-aafe-0f893f3f6df4") : secret "seaweedfs-tls-serving" not found
May 25 12:23:51.461043 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:51.461025    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/931de070-ae52-45bc-aafe-0f893f3f6df4-data\") pod \"seaweedfs-tls-serving-7fd5766db9-vjht7\" (UID: \"931de070-ae52-45bc-aafe-0f893f3f6df4\") " pod="kserve/seaweedfs-tls-serving-7fd5766db9-vjht7"
May 25 12:23:51.468616 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:51.468595    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54vzm\" (UniqueName: \"kubernetes.io/projected/931de070-ae52-45bc-aafe-0f893f3f6df4-kube-api-access-54vzm\") pod \"seaweedfs-tls-serving-7fd5766db9-vjht7\" (UID: \"931de070-ae52-45bc-aafe-0f893f3f6df4\") " pod="kserve/seaweedfs-tls-serving-7fd5766db9-vjht7"
May 25 12:23:51.964966 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:51.964917    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/931de070-ae52-45bc-aafe-0f893f3f6df4-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-7fd5766db9-vjht7\" (UID: \"931de070-ae52-45bc-aafe-0f893f3f6df4\") " pod="kserve/seaweedfs-tls-serving-7fd5766db9-vjht7"
May 25 12:23:51.967305 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:51.967269    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/931de070-ae52-45bc-aafe-0f893f3f6df4-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-7fd5766db9-vjht7\" (UID: \"931de070-ae52-45bc-aafe-0f893f3f6df4\") " pod="kserve/seaweedfs-tls-serving-7fd5766db9-vjht7"
May 25 12:23:52.182322 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:52.182287    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-7fd5766db9-vjht7"
May 25 12:23:52.305188 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:52.305153    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-7fd5766db9-vjht7"]
May 25 12:23:52.308045 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:23:52.308018    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod931de070_ae52_45bc_aafe_0f893f3f6df4.slice/crio-ef732ac2fac6befad5a8fa83de8c18b2216e0de95b216834f0ad62d33d6b393c WatchSource:0}: Error finding container ef732ac2fac6befad5a8fa83de8c18b2216e0de95b216834f0ad62d33d6b393c: Status 404 returned error can't find the container with id ef732ac2fac6befad5a8fa83de8c18b2216e0de95b216834f0ad62d33d6b393c
May 25 12:23:52.537959 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:52.537879    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-7fd5766db9-vjht7" event={"ID":"931de070-ae52-45bc-aafe-0f893f3f6df4","Type":"ContainerStarted","Data":"ef732ac2fac6befad5a8fa83de8c18b2216e0de95b216834f0ad62d33d6b393c"}
May 25 12:23:53.543356 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:53.543316    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-7fd5766db9-vjht7" event={"ID":"931de070-ae52-45bc-aafe-0f893f3f6df4","Type":"ContainerStarted","Data":"4440319e733e70647234c2bb6849f67743f5b1b60d25ea1d8ba792c44c34d693"}
May 25 12:23:53.557857 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:23:53.557811    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/seaweedfs-tls-serving-7fd5766db9-vjht7" podStartSLOduration=2.23316488 podStartE2EDuration="2.557798555s" podCreationTimestamp="2026-05-25 12:23:51 +0000 UTC" firstStartedPulling="2026-05-25 12:23:52.309424684 +0000 UTC m=+645.269475083" lastFinishedPulling="2026-05-25 12:23:52.634058374 +0000 UTC m=+645.594108758" observedRunningTime="2026-05-25 12:23:53.555924428 +0000 UTC m=+646.515974833" watchObservedRunningTime="2026-05-25 12:23:53.557798555 +0000 UTC m=+646.517848962"
May 25 12:24:10.958199 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:24:10.958157    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g"]
May 25 12:24:10.963059 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:24:10.963029    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g"
May 25 12:24:10.965163 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:24:10.965132    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
May 25 12:24:10.965312 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:24:10.965157    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
May 25 12:24:10.965312 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:24:10.965198    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 25 12:24:10.965312 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:24:10.965158    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 25 12:24:10.965312 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:24:10.965280    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6qbkc\""
May 25 12:24:10.972277 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:24:10.972249    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g"]
May 25 12:24:11.032993 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:24:11.032958    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18213b05-9a2d-44a7-bc08-6c18eac25e7c-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-54b87fc785-4228g\" (UID: \"18213b05-9a2d-44a7-bc08-6c18eac25e7c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g"
May 25 12:24:11.033173 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:24:11.033026    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18213b05-9a2d-44a7-bc08-6c18eac25e7c-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-54b87fc785-4228g\" (UID: \"18213b05-9a2d-44a7-bc08-6c18eac25e7c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g"
May 25 12:24:11.033173 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:24:11.033065    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18213b05-9a2d-44a7-bc08-6c18eac25e7c-isvc-sklearn-batcher-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-54b87fc785-4228g\" (UID: \"18213b05-9a2d-44a7-bc08-6c18eac25e7c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g"
May 25 12:24:11.033173 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:24:11.033099    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzgdc\" (UniqueName: \"kubernetes.io/projected/18213b05-9a2d-44a7-bc08-6c18eac25e7c-kube-api-access-dzgdc\") pod \"isvc-sklearn-batcher-predictor-54b87fc785-4228g\" (UID: \"18213b05-9a2d-44a7-bc08-6c18eac25e7c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g"
May 25 12:24:11.134201 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:24:11.134163    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18213b05-9a2d-44a7-bc08-6c18eac25e7c-isvc-sklearn-batcher-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-54b87fc785-4228g\" (UID: \"18213b05-9a2d-44a7-bc08-6c18eac25e7c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g"
May 25 12:24:11.134376 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:24:11.134225    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dzgdc\" (UniqueName: \"kubernetes.io/projected/18213b05-9a2d-44a7-bc08-6c18eac25e7c-kube-api-access-dzgdc\") pod \"isvc-sklearn-batcher-predictor-54b87fc785-4228g\" (UID: \"18213b05-9a2d-44a7-bc08-6c18eac25e7c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g"
May 25 12:24:11.134376 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:24:11.134280    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18213b05-9a2d-44a7-bc08-6c18eac25e7c-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-54b87fc785-4228g\" (UID: \"18213b05-9a2d-44a7-bc08-6c18eac25e7c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g"
May 25 12:24:11.134376 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:24:11.134329    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18213b05-9a2d-44a7-bc08-6c18eac25e7c-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-54b87fc785-4228g\" (UID: \"18213b05-9a2d-44a7-bc08-6c18eac25e7c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g"
May 25 12:24:11.134776 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:24:11.134753    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18213b05-9a2d-44a7-bc08-6c18eac25e7c-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-54b87fc785-4228g\" (UID: \"18213b05-9a2d-44a7-bc08-6c18eac25e7c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g"
May 25 12:24:11.134923 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:24:11.134900    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18213b05-9a2d-44a7-bc08-6c18eac25e7c-isvc-sklearn-batcher-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-54b87fc785-4228g\" (UID: \"18213b05-9a2d-44a7-bc08-6c18eac25e7c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g"
May 25 12:24:11.136787 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:24:11.136764    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18213b05-9a2d-44a7-bc08-6c18eac25e7c-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-54b87fc785-4228g\" (UID: \"18213b05-9a2d-44a7-bc08-6c18eac25e7c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g"
May 25 12:24:11.141054 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:24:11.141022    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzgdc\" (UniqueName: \"kubernetes.io/projected/18213b05-9a2d-44a7-bc08-6c18eac25e7c-kube-api-access-dzgdc\") pod \"isvc-sklearn-batcher-predictor-54b87fc785-4228g\" (UID: \"18213b05-9a2d-44a7-bc08-6c18eac25e7c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g"
May 25 12:24:11.276114 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:24:11.276009    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g"
May 25 12:24:11.425198 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:24:11.425139    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g"]
May 25 12:24:11.428014 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:24:11.427980    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18213b05_9a2d_44a7_bc08_6c18eac25e7c.slice/crio-5023578fd72c98fc668799761bb8b64544bcc89f59a7dbbc5820e47f3d674f40 WatchSource:0}: Error finding container 5023578fd72c98fc668799761bb8b64544bcc89f59a7dbbc5820e47f3d674f40: Status 404 returned error can't find the container with id 5023578fd72c98fc668799761bb8b64544bcc89f59a7dbbc5820e47f3d674f40
May 25 12:24:11.605738 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:24:11.605626    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g" event={"ID":"18213b05-9a2d-44a7-bc08-6c18eac25e7c","Type":"ContainerStarted","Data":"5023578fd72c98fc668799761bb8b64544bcc89f59a7dbbc5820e47f3d674f40"}
May 25 12:24:15.628393 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:24:15.628358    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g" event={"ID":"18213b05-9a2d-44a7-bc08-6c18eac25e7c","Type":"ContainerStarted","Data":"1f54d035cedab0f6aeea266779b192186d3397aa444bdfd6d0ace93d6a382fe9"}
May 25 12:24:18.639729 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:24:18.639691    2555 generic.go:358] "Generic (PLEG): container finished" podID="18213b05-9a2d-44a7-bc08-6c18eac25e7c" containerID="1f54d035cedab0f6aeea266779b192186d3397aa444bdfd6d0ace93d6a382fe9" exitCode=0
May 25 12:24:18.640094 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:24:18.639770    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g" event={"ID":"18213b05-9a2d-44a7-bc08-6c18eac25e7c","Type":"ContainerDied","Data":"1f54d035cedab0f6aeea266779b192186d3397aa444bdfd6d0ace93d6a382fe9"}
May 25 12:24:31.694969 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:24:31.694919    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g" event={"ID":"18213b05-9a2d-44a7-bc08-6c18eac25e7c","Type":"ContainerStarted","Data":"64a74676d6ec6bd63db3372205bf9444deec2a874625e0f5433b898f9be34c86"}
May 25 12:24:33.705053 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:24:33.705015    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g" event={"ID":"18213b05-9a2d-44a7-bc08-6c18eac25e7c","Type":"ContainerStarted","Data":"b7a9fa06eb5062eb78a8c10526821a8a775cd00a5daed815f81ade56eb4b7c09"}
May 25 12:24:36.718226 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:24:36.718191    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g" event={"ID":"18213b05-9a2d-44a7-bc08-6c18eac25e7c","Type":"ContainerStarted","Data":"76249da8ab155e95756091b94d35fa23038bb10eec9d58a8111e20d6563fd25d"}
May 25 12:24:36.718686 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:24:36.718471    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g"
May 25 12:24:36.718686 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:24:36.718497    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g"
May 25 12:24:36.718686 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:24:36.718509    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g"
May 25 12:24:36.720051 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:24:36.720011    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g" podUID="18213b05-9a2d-44a7-bc08-6c18eac25e7c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
May 25 12:24:36.720634 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:24:36.720613    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g" podUID="18213b05-9a2d-44a7-bc08-6c18eac25e7c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:24:36.735237 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:24:36.735191    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g" podStartSLOduration=2.165108562 podStartE2EDuration="26.735177673s" podCreationTimestamp="2026-05-25 12:24:10 +0000 UTC" firstStartedPulling="2026-05-25 12:24:11.430056514 +0000 UTC m=+664.390106897" lastFinishedPulling="2026-05-25 12:24:36.000125605 +0000 UTC m=+688.960176008" observedRunningTime="2026-05-25 12:24:36.734197057 +0000 UTC m=+689.694247476" watchObservedRunningTime="2026-05-25 12:24:36.735177673 +0000 UTC m=+689.695228146"
May 25 12:24:37.722127 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:24:37.722092    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g" podUID="18213b05-9a2d-44a7-bc08-6c18eac25e7c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
May 25 12:24:37.722532 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:24:37.722498    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g" podUID="18213b05-9a2d-44a7-bc08-6c18eac25e7c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:24:42.727632 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:24:42.727602    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g"
May 25 12:24:42.728235 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:24:42.728205    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g" podUID="18213b05-9a2d-44a7-bc08-6c18eac25e7c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
May 25 12:24:42.728556 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:24:42.728534    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g" podUID="18213b05-9a2d-44a7-bc08-6c18eac25e7c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:24:52.728932 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:24:52.728894    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g" podUID="18213b05-9a2d-44a7-bc08-6c18eac25e7c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
May 25 12:24:52.729375 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:24:52.729266    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g" podUID="18213b05-9a2d-44a7-bc08-6c18eac25e7c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:25:02.728621 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:25:02.728520    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g" podUID="18213b05-9a2d-44a7-bc08-6c18eac25e7c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
May 25 12:25:02.729133 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:25:02.729037    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g" podUID="18213b05-9a2d-44a7-bc08-6c18eac25e7c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:25:12.729012 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:25:12.728969    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g" podUID="18213b05-9a2d-44a7-bc08-6c18eac25e7c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
May 25 12:25:12.729455 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:25:12.729426    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g" podUID="18213b05-9a2d-44a7-bc08-6c18eac25e7c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:25:22.728284 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:25:22.728241    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g" podUID="18213b05-9a2d-44a7-bc08-6c18eac25e7c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
May 25 12:25:22.728780 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:25:22.728715    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g" podUID="18213b05-9a2d-44a7-bc08-6c18eac25e7c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:25:32.728136 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:25:32.728094    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g" podUID="18213b05-9a2d-44a7-bc08-6c18eac25e7c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
May 25 12:25:32.728587 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:25:32.728494    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g" podUID="18213b05-9a2d-44a7-bc08-6c18eac25e7c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:25:42.729842 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:25:42.729805    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g"
May 25 12:25:42.730427 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:25:42.729925    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g"
May 25 12:25:55.963738 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:25:55.963634    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g"]
May 25 12:25:55.966243 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:25:55.966006    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g" podUID="18213b05-9a2d-44a7-bc08-6c18eac25e7c" containerName="kserve-container" containerID="cri-o://64a74676d6ec6bd63db3372205bf9444deec2a874625e0f5433b898f9be34c86" gracePeriod=30
May 25 12:25:55.967724 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:25:55.966712    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g" podUID="18213b05-9a2d-44a7-bc08-6c18eac25e7c" containerName="agent" containerID="cri-o://76249da8ab155e95756091b94d35fa23038bb10eec9d58a8111e20d6563fd25d" gracePeriod=30
May 25 12:25:55.967724 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:25:55.966849    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g" podUID="18213b05-9a2d-44a7-bc08-6c18eac25e7c" containerName="kube-rbac-proxy" containerID="cri-o://b7a9fa06eb5062eb78a8c10526821a8a775cd00a5daed815f81ade56eb4b7c09" gracePeriod=30
May 25 12:25:56.092440 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:25:56.092409    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q"]
May 25 12:25:56.095214 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:25:56.095196    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q"
May 25 12:25:56.097221 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:25:56.097193    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
May 25 12:25:56.097221 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:25:56.097215    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
May 25 12:25:56.104543 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:25:56.104517    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q"]
May 25 12:25:56.207050 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:25:56.207006    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a17746c5-1974-4937-9719-fe219bbb2c2f-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q\" (UID: \"a17746c5-1974-4937-9719-fe219bbb2c2f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q"
May 25 12:25:56.207247 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:25:56.207062    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a17746c5-1974-4937-9719-fe219bbb2c2f-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q\" (UID: \"a17746c5-1974-4937-9719-fe219bbb2c2f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q"
May 25 12:25:56.207247 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:25:56.207140    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a17746c5-1974-4937-9719-fe219bbb2c2f-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q\" (UID: \"a17746c5-1974-4937-9719-fe219bbb2c2f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q"
May 25 12:25:56.207247 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:25:56.207181    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q28p\" (UniqueName: \"kubernetes.io/projected/a17746c5-1974-4937-9719-fe219bbb2c2f-kube-api-access-7q28p\") pod \"isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q\" (UID: \"a17746c5-1974-4937-9719-fe219bbb2c2f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q"
May 25 12:25:56.308531 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:25:56.308434    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a17746c5-1974-4937-9719-fe219bbb2c2f-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q\" (UID: \"a17746c5-1974-4937-9719-fe219bbb2c2f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q"
May 25 12:25:56.308531 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:25:56.308485    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a17746c5-1974-4937-9719-fe219bbb2c2f-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q\" (UID: \"a17746c5-1974-4937-9719-fe219bbb2c2f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q"
May 25 12:25:56.308531 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:25:56.308510    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a17746c5-1974-4937-9719-fe219bbb2c2f-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q\" (UID: \"a17746c5-1974-4937-9719-fe219bbb2c2f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q"
May 25 12:25:56.308839 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:25:56.308545    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7q28p\" (UniqueName: \"kubernetes.io/projected/a17746c5-1974-4937-9719-fe219bbb2c2f-kube-api-access-7q28p\") pod \"isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q\" (UID: \"a17746c5-1974-4937-9719-fe219bbb2c2f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q"
May 25 12:25:56.309201 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:25:56.309168    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a17746c5-1974-4937-9719-fe219bbb2c2f-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q\" (UID: \"a17746c5-1974-4937-9719-fe219bbb2c2f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q"
May 25 12:25:56.309467 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:25:56.309449    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a17746c5-1974-4937-9719-fe219bbb2c2f-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q\" (UID: \"a17746c5-1974-4937-9719-fe219bbb2c2f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q"
May 25 12:25:56.311293 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:25:56.311276    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a17746c5-1974-4937-9719-fe219bbb2c2f-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q\" (UID: \"a17746c5-1974-4937-9719-fe219bbb2c2f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q"
May 25 12:25:56.316029 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:25:56.315996    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q28p\" (UniqueName: \"kubernetes.io/projected/a17746c5-1974-4937-9719-fe219bbb2c2f-kube-api-access-7q28p\") pod \"isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q\" (UID: \"a17746c5-1974-4937-9719-fe219bbb2c2f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q"
May 25 12:25:56.407270 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:25:56.407226    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q"
May 25 12:25:56.547182 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:25:56.547136    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q"]
May 25 12:25:56.548826 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:25:56.548792    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda17746c5_1974_4937_9719_fe219bbb2c2f.slice/crio-df9ff8c0fbe8f5b5c564d83f70367e3f2a4ee3b96225a61b46d98609f0b34eba WatchSource:0}: Error finding container df9ff8c0fbe8f5b5c564d83f70367e3f2a4ee3b96225a61b46d98609f0b34eba: Status 404 returned error can't find the container with id df9ff8c0fbe8f5b5c564d83f70367e3f2a4ee3b96225a61b46d98609f0b34eba
May 25 12:25:57.008763 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:25:57.008726    2555 generic.go:358] "Generic (PLEG): container finished" podID="18213b05-9a2d-44a7-bc08-6c18eac25e7c" containerID="b7a9fa06eb5062eb78a8c10526821a8a775cd00a5daed815f81ade56eb4b7c09" exitCode=2
May 25 12:25:57.009229 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:25:57.008809    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g" event={"ID":"18213b05-9a2d-44a7-bc08-6c18eac25e7c","Type":"ContainerDied","Data":"b7a9fa06eb5062eb78a8c10526821a8a775cd00a5daed815f81ade56eb4b7c09"}
May 25 12:25:57.010361 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:25:57.010334    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q" event={"ID":"a17746c5-1974-4937-9719-fe219bbb2c2f","Type":"ContainerStarted","Data":"6ca031ec42d74bf2315106ec855f94f171f7b0c2b43c890efff26228c580b3d3"}
May 25 12:25:57.010361 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:25:57.010364    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q" event={"ID":"a17746c5-1974-4937-9719-fe219bbb2c2f","Type":"ContainerStarted","Data":"df9ff8c0fbe8f5b5c564d83f70367e3f2a4ee3b96225a61b46d98609f0b34eba"}
May 25 12:25:57.722815 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:25:57.722776    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g" podUID="18213b05-9a2d-44a7-bc08-6c18eac25e7c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
May 25 12:26:00.023697 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:00.023664    2555 generic.go:358] "Generic (PLEG): container finished" podID="18213b05-9a2d-44a7-bc08-6c18eac25e7c" containerID="64a74676d6ec6bd63db3372205bf9444deec2a874625e0f5433b898f9be34c86" exitCode=0
May 25 12:26:00.024050 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:00.023724    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g" event={"ID":"18213b05-9a2d-44a7-bc08-6c18eac25e7c","Type":"ContainerDied","Data":"64a74676d6ec6bd63db3372205bf9444deec2a874625e0f5433b898f9be34c86"}
May 25 12:26:01.028101 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:01.028069    2555 generic.go:358] "Generic (PLEG): container finished" podID="a17746c5-1974-4937-9719-fe219bbb2c2f" containerID="6ca031ec42d74bf2315106ec855f94f171f7b0c2b43c890efff26228c580b3d3" exitCode=0
May 25 12:26:01.028466 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:01.028122    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q" event={"ID":"a17746c5-1974-4937-9719-fe219bbb2c2f","Type":"ContainerDied","Data":"6ca031ec42d74bf2315106ec855f94f171f7b0c2b43c890efff26228c580b3d3"}
May 25 12:26:02.034083 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:02.034044    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q" event={"ID":"a17746c5-1974-4937-9719-fe219bbb2c2f","Type":"ContainerStarted","Data":"f3d5d1889a0021175e46749151a1521b976642e962ecaa5f0c326154702640f7"}
May 25 12:26:02.034453 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:02.034092    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q" event={"ID":"a17746c5-1974-4937-9719-fe219bbb2c2f","Type":"ContainerStarted","Data":"c9ea10451ff0369dfca226a0ce9ff4a11ba73940e929717b43c518130dbcc156"}
May 25 12:26:02.034453 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:02.034103    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q" event={"ID":"a17746c5-1974-4937-9719-fe219bbb2c2f","Type":"ContainerStarted","Data":"0ac2ce3f8019560064dd4a81bf8a0ad6a20a6423dabe33694f12106447221688"}
May 25 12:26:02.034453 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:02.034353    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q"
May 25 12:26:02.034453 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:02.034382    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q"
May 25 12:26:02.035548 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:02.035519    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q" podUID="a17746c5-1974-4937-9719-fe219bbb2c2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:5000: connect: connection refused"
May 25 12:26:02.052407 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:02.052353    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q" podStartSLOduration=6.052338858 podStartE2EDuration="6.052338858s" podCreationTimestamp="2026-05-25 12:25:56 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:26:02.05162347 +0000 UTC m=+775.011673877" watchObservedRunningTime="2026-05-25 12:26:02.052338858 +0000 UTC m=+775.012389265"
May 25 12:26:02.722366 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:02.722328    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g" podUID="18213b05-9a2d-44a7-bc08-6c18eac25e7c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
May 25 12:26:02.728798 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:02.728763    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g" podUID="18213b05-9a2d-44a7-bc08-6c18eac25e7c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
May 25 12:26:02.729105 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:02.729086    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g" podUID="18213b05-9a2d-44a7-bc08-6c18eac25e7c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:26:03.038087 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:03.037999    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q"
May 25 12:26:03.038505 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:03.038116    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q" podUID="a17746c5-1974-4937-9719-fe219bbb2c2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:5000: connect: connection refused"
May 25 12:26:03.038936 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:03.038911    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q" podUID="a17746c5-1974-4937-9719-fe219bbb2c2f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:26:04.041505 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:04.041467    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q" podUID="a17746c5-1974-4937-9719-fe219bbb2c2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:5000: connect: connection refused"
May 25 12:26:04.041931 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:04.041830    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q" podUID="a17746c5-1974-4937-9719-fe219bbb2c2f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:26:07.722581 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:07.722544    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g" podUID="18213b05-9a2d-44a7-bc08-6c18eac25e7c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
May 25 12:26:07.722949 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:07.722711    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g"
May 25 12:26:09.045565 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:09.045537    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q"
May 25 12:26:09.046190 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:09.046159    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q" podUID="a17746c5-1974-4937-9719-fe219bbb2c2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:5000: connect: connection refused"
May 25 12:26:09.046323 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:09.046298    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q" podUID="a17746c5-1974-4937-9719-fe219bbb2c2f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:26:12.722241 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:12.722199    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g" podUID="18213b05-9a2d-44a7-bc08-6c18eac25e7c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
May 25 12:26:12.728602 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:12.728570    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g" podUID="18213b05-9a2d-44a7-bc08-6c18eac25e7c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
May 25 12:26:12.729017 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:12.728986    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g" podUID="18213b05-9a2d-44a7-bc08-6c18eac25e7c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:26:17.722964 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:17.722923    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g" podUID="18213b05-9a2d-44a7-bc08-6c18eac25e7c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
May 25 12:26:19.046319 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:19.046275    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q" podUID="a17746c5-1974-4937-9719-fe219bbb2c2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:5000: connect: connection refused"
May 25 12:26:19.046735 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:19.046678    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q" podUID="a17746c5-1974-4937-9719-fe219bbb2c2f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:26:22.722898 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:22.722861    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g" podUID="18213b05-9a2d-44a7-bc08-6c18eac25e7c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
May 25 12:26:22.728212 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:22.728188    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g" podUID="18213b05-9a2d-44a7-bc08-6c18eac25e7c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
May 25 12:26:22.728299 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:22.728292    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g"
May 25 12:26:22.728673 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:22.728626    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g" podUID="18213b05-9a2d-44a7-bc08-6c18eac25e7c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:26:22.728791 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:22.728777    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g"
May 25 12:26:26.117266 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:26.117238    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g"
May 25 12:26:26.117974 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:26.117953    2555 generic.go:358] "Generic (PLEG): container finished" podID="18213b05-9a2d-44a7-bc08-6c18eac25e7c" containerID="76249da8ab155e95756091b94d35fa23038bb10eec9d58a8111e20d6563fd25d" exitCode=0
May 25 12:26:26.118047 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:26.118027    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g" event={"ID":"18213b05-9a2d-44a7-bc08-6c18eac25e7c","Type":"ContainerDied","Data":"76249da8ab155e95756091b94d35fa23038bb10eec9d58a8111e20d6563fd25d"}
May 25 12:26:26.118086 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:26.118066    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g" event={"ID":"18213b05-9a2d-44a7-bc08-6c18eac25e7c","Type":"ContainerDied","Data":"5023578fd72c98fc668799761bb8b64544bcc89f59a7dbbc5820e47f3d674f40"}
May 25 12:26:26.118086 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:26.118082    2555 scope.go:117] "RemoveContainer" containerID="76249da8ab155e95756091b94d35fa23038bb10eec9d58a8111e20d6563fd25d"
May 25 12:26:26.125539 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:26.125521    2555 scope.go:117] "RemoveContainer" containerID="b7a9fa06eb5062eb78a8c10526821a8a775cd00a5daed815f81ade56eb4b7c09"
May 25 12:26:26.133556 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:26.133539    2555 scope.go:117] "RemoveContainer" containerID="64a74676d6ec6bd63db3372205bf9444deec2a874625e0f5433b898f9be34c86"
May 25 12:26:26.133628 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:26.133611    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dzgdc\" (UniqueName: \"kubernetes.io/projected/18213b05-9a2d-44a7-bc08-6c18eac25e7c-kube-api-access-dzgdc\") pod \"18213b05-9a2d-44a7-bc08-6c18eac25e7c\" (UID: \"18213b05-9a2d-44a7-bc08-6c18eac25e7c\") "
May 25 12:26:26.133737 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:26.133719    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18213b05-9a2d-44a7-bc08-6c18eac25e7c-proxy-tls\") pod \"18213b05-9a2d-44a7-bc08-6c18eac25e7c\" (UID: \"18213b05-9a2d-44a7-bc08-6c18eac25e7c\") "
May 25 12:26:26.133793 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:26.133779    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18213b05-9a2d-44a7-bc08-6c18eac25e7c-kserve-provision-location\") pod \"18213b05-9a2d-44a7-bc08-6c18eac25e7c\" (UID: \"18213b05-9a2d-44a7-bc08-6c18eac25e7c\") "
May 25 12:26:26.133841 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:26.133828    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18213b05-9a2d-44a7-bc08-6c18eac25e7c-isvc-sklearn-batcher-kube-rbac-proxy-sar-config\") pod \"18213b05-9a2d-44a7-bc08-6c18eac25e7c\" (UID: \"18213b05-9a2d-44a7-bc08-6c18eac25e7c\") "
May 25 12:26:26.134206 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:26.134128    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18213b05-9a2d-44a7-bc08-6c18eac25e7c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "18213b05-9a2d-44a7-bc08-6c18eac25e7c" (UID: "18213b05-9a2d-44a7-bc08-6c18eac25e7c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:26:26.134394 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:26.134347    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18213b05-9a2d-44a7-bc08-6c18eac25e7c-isvc-sklearn-batcher-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-batcher-kube-rbac-proxy-sar-config") pod "18213b05-9a2d-44a7-bc08-6c18eac25e7c" (UID: "18213b05-9a2d-44a7-bc08-6c18eac25e7c"). InnerVolumeSpecName "isvc-sklearn-batcher-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:26:26.136267 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:26.136241    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18213b05-9a2d-44a7-bc08-6c18eac25e7c-kube-api-access-dzgdc" (OuterVolumeSpecName: "kube-api-access-dzgdc") pod "18213b05-9a2d-44a7-bc08-6c18eac25e7c" (UID: "18213b05-9a2d-44a7-bc08-6c18eac25e7c"). InnerVolumeSpecName "kube-api-access-dzgdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:26:26.136394 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:26.136375    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18213b05-9a2d-44a7-bc08-6c18eac25e7c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "18213b05-9a2d-44a7-bc08-6c18eac25e7c" (UID: "18213b05-9a2d-44a7-bc08-6c18eac25e7c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:26:26.159267 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:26.159245    2555 scope.go:117] "RemoveContainer" containerID="1f54d035cedab0f6aeea266779b192186d3397aa444bdfd6d0ace93d6a382fe9"
May 25 12:26:26.166035 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:26.166007    2555 scope.go:117] "RemoveContainer" containerID="76249da8ab155e95756091b94d35fa23038bb10eec9d58a8111e20d6563fd25d"
May 25 12:26:26.166247 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:26:26.166229    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76249da8ab155e95756091b94d35fa23038bb10eec9d58a8111e20d6563fd25d\": container with ID starting with 76249da8ab155e95756091b94d35fa23038bb10eec9d58a8111e20d6563fd25d not found: ID does not exist" containerID="76249da8ab155e95756091b94d35fa23038bb10eec9d58a8111e20d6563fd25d"
May 25 12:26:26.166292 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:26.166257    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76249da8ab155e95756091b94d35fa23038bb10eec9d58a8111e20d6563fd25d"} err="failed to get container status \"76249da8ab155e95756091b94d35fa23038bb10eec9d58a8111e20d6563fd25d\": rpc error: code = NotFound desc = could not find container \"76249da8ab155e95756091b94d35fa23038bb10eec9d58a8111e20d6563fd25d\": container with ID starting with 76249da8ab155e95756091b94d35fa23038bb10eec9d58a8111e20d6563fd25d not found: ID does not exist"
May 25 12:26:26.166292 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:26.166276    2555 scope.go:117] "RemoveContainer" containerID="b7a9fa06eb5062eb78a8c10526821a8a775cd00a5daed815f81ade56eb4b7c09"
May 25 12:26:26.166494 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:26:26.166477    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7a9fa06eb5062eb78a8c10526821a8a775cd00a5daed815f81ade56eb4b7c09\": container with ID starting with b7a9fa06eb5062eb78a8c10526821a8a775cd00a5daed815f81ade56eb4b7c09 not found: ID does not exist" containerID="b7a9fa06eb5062eb78a8c10526821a8a775cd00a5daed815f81ade56eb4b7c09"
May 25 12:26:26.166555 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:26.166504    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7a9fa06eb5062eb78a8c10526821a8a775cd00a5daed815f81ade56eb4b7c09"} err="failed to get container status \"b7a9fa06eb5062eb78a8c10526821a8a775cd00a5daed815f81ade56eb4b7c09\": rpc error: code = NotFound desc = could not find container \"b7a9fa06eb5062eb78a8c10526821a8a775cd00a5daed815f81ade56eb4b7c09\": container with ID starting with b7a9fa06eb5062eb78a8c10526821a8a775cd00a5daed815f81ade56eb4b7c09 not found: ID does not exist"
May 25 12:26:26.166555 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:26.166528    2555 scope.go:117] "RemoveContainer" containerID="64a74676d6ec6bd63db3372205bf9444deec2a874625e0f5433b898f9be34c86"
May 25 12:26:26.166856 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:26:26.166839    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64a74676d6ec6bd63db3372205bf9444deec2a874625e0f5433b898f9be34c86\": container with ID starting with 64a74676d6ec6bd63db3372205bf9444deec2a874625e0f5433b898f9be34c86 not found: ID does not exist" containerID="64a74676d6ec6bd63db3372205bf9444deec2a874625e0f5433b898f9be34c86"
May 25 12:26:26.166899 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:26.166862    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64a74676d6ec6bd63db3372205bf9444deec2a874625e0f5433b898f9be34c86"} err="failed to get container status \"64a74676d6ec6bd63db3372205bf9444deec2a874625e0f5433b898f9be34c86\": rpc error: code = NotFound desc = could not find container \"64a74676d6ec6bd63db3372205bf9444deec2a874625e0f5433b898f9be34c86\": container with ID starting with 64a74676d6ec6bd63db3372205bf9444deec2a874625e0f5433b898f9be34c86 not found: ID does not exist"
May 25 12:26:26.166899 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:26.166876    2555 scope.go:117] "RemoveContainer" containerID="1f54d035cedab0f6aeea266779b192186d3397aa444bdfd6d0ace93d6a382fe9"
May 25 12:26:26.167100 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:26:26.167083    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f54d035cedab0f6aeea266779b192186d3397aa444bdfd6d0ace93d6a382fe9\": container with ID starting with 1f54d035cedab0f6aeea266779b192186d3397aa444bdfd6d0ace93d6a382fe9 not found: ID does not exist" containerID="1f54d035cedab0f6aeea266779b192186d3397aa444bdfd6d0ace93d6a382fe9"
May 25 12:26:26.167141 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:26.167104    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f54d035cedab0f6aeea266779b192186d3397aa444bdfd6d0ace93d6a382fe9"} err="failed to get container status \"1f54d035cedab0f6aeea266779b192186d3397aa444bdfd6d0ace93d6a382fe9\": rpc error: code = NotFound desc = could not find container \"1f54d035cedab0f6aeea266779b192186d3397aa444bdfd6d0ace93d6a382fe9\": container with ID starting with 1f54d035cedab0f6aeea266779b192186d3397aa444bdfd6d0ace93d6a382fe9 not found: ID does not exist"
May 25 12:26:26.234965 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:26.234910    2555 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18213b05-9a2d-44a7-bc08-6c18eac25e7c-isvc-sklearn-batcher-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:26:26.234965 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:26.234938    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzgdc\" (UniqueName: \"kubernetes.io/projected/18213b05-9a2d-44a7-bc08-6c18eac25e7c-kube-api-access-dzgdc\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:26:26.234965 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:26.234952    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18213b05-9a2d-44a7-bc08-6c18eac25e7c-proxy-tls\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:26:26.235106 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:26.234968    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18213b05-9a2d-44a7-bc08-6c18eac25e7c-kserve-provision-location\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:26:27.123248 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:27.123179    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g"
May 25 12:26:27.144898 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:27.144871    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g"]
May 25 12:26:27.147990 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:27.147967    2555 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-4228g"]
May 25 12:26:27.603595 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:27.603562    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18213b05-9a2d-44a7-bc08-6c18eac25e7c" path="/var/lib/kubelet/pods/18213b05-9a2d-44a7-bc08-6c18eac25e7c/volumes"
May 25 12:26:29.046109 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:29.046066    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q" podUID="a17746c5-1974-4937-9719-fe219bbb2c2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:5000: connect: connection refused"
May 25 12:26:29.046513 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:29.046489    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q" podUID="a17746c5-1974-4937-9719-fe219bbb2c2f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:26:39.046801 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:39.046750    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q" podUID="a17746c5-1974-4937-9719-fe219bbb2c2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:5000: connect: connection refused"
May 25 12:26:39.047279 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:39.047254    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q" podUID="a17746c5-1974-4937-9719-fe219bbb2c2f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:26:49.046888 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:49.046835    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q" podUID="a17746c5-1974-4937-9719-fe219bbb2c2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:5000: connect: connection refused"
May 25 12:26:49.047427 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:49.047227    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q" podUID="a17746c5-1974-4937-9719-fe219bbb2c2f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:26:59.046239 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:59.046195    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q" podUID="a17746c5-1974-4937-9719-fe219bbb2c2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:5000: connect: connection refused"
May 25 12:26:59.046766 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:26:59.046676    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q" podUID="a17746c5-1974-4937-9719-fe219bbb2c2f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:27:09.046809 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:09.046775    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q"
May 25 12:27:09.047274 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:09.046837    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q"
May 25 12:27:21.145881 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:21.145846    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q"]
May 25 12:27:21.146448 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:21.146232    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q" podUID="a17746c5-1974-4937-9719-fe219bbb2c2f" containerName="kserve-container" containerID="cri-o://0ac2ce3f8019560064dd4a81bf8a0ad6a20a6423dabe33694f12106447221688" gracePeriod=30
May 25 12:27:21.146448 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:21.146260    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q" podUID="a17746c5-1974-4937-9719-fe219bbb2c2f" containerName="agent" containerID="cri-o://f3d5d1889a0021175e46749151a1521b976642e962ecaa5f0c326154702640f7" gracePeriod=30
May 25 12:27:21.146559 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:21.146311    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q" podUID="a17746c5-1974-4937-9719-fe219bbb2c2f" containerName="kube-rbac-proxy" containerID="cri-o://c9ea10451ff0369dfca226a0ce9ff4a11ba73940e929717b43c518130dbcc156" gracePeriod=30
May 25 12:27:21.216111 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:21.216082    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-msxcd"]
May 25 12:27:21.216430 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:21.216419    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18213b05-9a2d-44a7-bc08-6c18eac25e7c" containerName="kserve-container"
May 25 12:27:21.216472 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:21.216432    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="18213b05-9a2d-44a7-bc08-6c18eac25e7c" containerName="kserve-container"
May 25 12:27:21.216472 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:21.216450    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18213b05-9a2d-44a7-bc08-6c18eac25e7c" containerName="storage-initializer"
May 25 12:27:21.216472 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:21.216456    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="18213b05-9a2d-44a7-bc08-6c18eac25e7c" containerName="storage-initializer"
May 25 12:27:21.216472 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:21.216466    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18213b05-9a2d-44a7-bc08-6c18eac25e7c" containerName="agent"
May 25 12:27:21.216472 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:21.216471    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="18213b05-9a2d-44a7-bc08-6c18eac25e7c" containerName="agent"
May 25 12:27:21.216619 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:21.216482    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18213b05-9a2d-44a7-bc08-6c18eac25e7c" containerName="kube-rbac-proxy"
May 25 12:27:21.216619 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:21.216488    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="18213b05-9a2d-44a7-bc08-6c18eac25e7c" containerName="kube-rbac-proxy"
May 25 12:27:21.216619 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:21.216538    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="18213b05-9a2d-44a7-bc08-6c18eac25e7c" containerName="agent"
May 25 12:27:21.216619 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:21.216547    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="18213b05-9a2d-44a7-bc08-6c18eac25e7c" containerName="kserve-container"
May 25 12:27:21.216619 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:21.216554    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="18213b05-9a2d-44a7-bc08-6c18eac25e7c" containerName="kube-rbac-proxy"
May 25 12:27:21.220707 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:21.220689    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-msxcd"
May 25 12:27:21.222499 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:21.222461    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
May 25 12:27:21.222590 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:21.222555    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
May 25 12:27:21.229558 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:21.229533    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-msxcd"]
May 25 12:27:21.307255 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:21.307226    2555 generic.go:358] "Generic (PLEG): container finished" podID="a17746c5-1974-4937-9719-fe219bbb2c2f" containerID="c9ea10451ff0369dfca226a0ce9ff4a11ba73940e929717b43c518130dbcc156" exitCode=2
May 25 12:27:21.307365 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:21.307303    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q" event={"ID":"a17746c5-1974-4937-9719-fe219bbb2c2f","Type":"ContainerDied","Data":"c9ea10451ff0369dfca226a0ce9ff4a11ba73940e929717b43c518130dbcc156"}
May 25 12:27:21.353839 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:21.353813    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4drh\" (UniqueName: \"kubernetes.io/projected/c80fbc59-fd33-487b-94f2-24ec039041b6-kube-api-access-l4drh\") pod \"message-dumper-predictor-c7d86bcbd-msxcd\" (UID: \"c80fbc59-fd33-487b-94f2-24ec039041b6\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-msxcd"
May 25 12:27:21.353957 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:21.353872    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c80fbc59-fd33-487b-94f2-24ec039041b6-proxy-tls\") pod \"message-dumper-predictor-c7d86bcbd-msxcd\" (UID: \"c80fbc59-fd33-487b-94f2-24ec039041b6\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-msxcd"
May 25 12:27:21.354021 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:21.353949    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"message-dumper-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c80fbc59-fd33-487b-94f2-24ec039041b6-message-dumper-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-c7d86bcbd-msxcd\" (UID: \"c80fbc59-fd33-487b-94f2-24ec039041b6\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-msxcd"
May 25 12:27:21.454998 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:21.454975    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-l4drh\" (UniqueName: \"kubernetes.io/projected/c80fbc59-fd33-487b-94f2-24ec039041b6-kube-api-access-l4drh\") pod \"message-dumper-predictor-c7d86bcbd-msxcd\" (UID: \"c80fbc59-fd33-487b-94f2-24ec039041b6\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-msxcd"
May 25 12:27:21.455127 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:21.455046    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c80fbc59-fd33-487b-94f2-24ec039041b6-proxy-tls\") pod \"message-dumper-predictor-c7d86bcbd-msxcd\" (UID: \"c80fbc59-fd33-487b-94f2-24ec039041b6\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-msxcd"
May 25 12:27:21.455127 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:21.455090    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"message-dumper-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c80fbc59-fd33-487b-94f2-24ec039041b6-message-dumper-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-c7d86bcbd-msxcd\" (UID: \"c80fbc59-fd33-487b-94f2-24ec039041b6\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-msxcd"
May 25 12:27:21.455233 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:27:21.455171    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-predictor-serving-cert: secret "message-dumper-predictor-serving-cert" not found
May 25 12:27:21.455285 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:27:21.455235    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c80fbc59-fd33-487b-94f2-24ec039041b6-proxy-tls podName:c80fbc59-fd33-487b-94f2-24ec039041b6 nodeName:}" failed. No retries permitted until 2026-05-25 12:27:21.955213726 +0000 UTC m=+854.915264121 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c80fbc59-fd33-487b-94f2-24ec039041b6-proxy-tls") pod "message-dumper-predictor-c7d86bcbd-msxcd" (UID: "c80fbc59-fd33-487b-94f2-24ec039041b6") : secret "message-dumper-predictor-serving-cert" not found
May 25 12:27:21.455734 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:21.455715    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"message-dumper-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c80fbc59-fd33-487b-94f2-24ec039041b6-message-dumper-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-c7d86bcbd-msxcd\" (UID: \"c80fbc59-fd33-487b-94f2-24ec039041b6\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-msxcd"
May 25 12:27:21.462736 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:21.462713    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4drh\" (UniqueName: \"kubernetes.io/projected/c80fbc59-fd33-487b-94f2-24ec039041b6-kube-api-access-l4drh\") pod \"message-dumper-predictor-c7d86bcbd-msxcd\" (UID: \"c80fbc59-fd33-487b-94f2-24ec039041b6\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-msxcd"
May 25 12:27:21.959177 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:21.959137    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c80fbc59-fd33-487b-94f2-24ec039041b6-proxy-tls\") pod \"message-dumper-predictor-c7d86bcbd-msxcd\" (UID: \"c80fbc59-fd33-487b-94f2-24ec039041b6\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-msxcd"
May 25 12:27:21.961440 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:21.961421    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c80fbc59-fd33-487b-94f2-24ec039041b6-proxy-tls\") pod \"message-dumper-predictor-c7d86bcbd-msxcd\" (UID: \"c80fbc59-fd33-487b-94f2-24ec039041b6\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-msxcd"
May 25 12:27:22.131164 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:22.131134    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-msxcd"
May 25 12:27:22.254214 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:22.254133    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-msxcd"]
May 25 12:27:22.256475 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:27:22.256448    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc80fbc59_fd33_487b_94f2_24ec039041b6.slice/crio-23ca8f75daf209adc909753d2fb142d00254b6ef33dfc3014f8499c095a4e975 WatchSource:0}: Error finding container 23ca8f75daf209adc909753d2fb142d00254b6ef33dfc3014f8499c095a4e975: Status 404 returned error can't find the container with id 23ca8f75daf209adc909753d2fb142d00254b6ef33dfc3014f8499c095a4e975
May 25 12:27:22.311134 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:22.311105    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-msxcd" event={"ID":"c80fbc59-fd33-487b-94f2-24ec039041b6","Type":"ContainerStarted","Data":"23ca8f75daf209adc909753d2fb142d00254b6ef33dfc3014f8499c095a4e975"}
May 25 12:27:24.042334 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:24.042296    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q" podUID="a17746c5-1974-4937-9719-fe219bbb2c2f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
May 25 12:27:24.321603 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:24.321522    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-msxcd" event={"ID":"c80fbc59-fd33-487b-94f2-24ec039041b6","Type":"ContainerStarted","Data":"b1901ccc00bb67525c96bc49bcc7a879d9ab5d377c179610b2e25bfa882c44b9"}
May 25 12:27:24.321603 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:24.321561    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-msxcd" event={"ID":"c80fbc59-fd33-487b-94f2-24ec039041b6","Type":"ContainerStarted","Data":"66cf69eb511c53eba10f7defe3a3593505762826a3dddab92eef317a490b98ed"}
May 25 12:27:24.321794 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:24.321623    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-msxcd"
May 25 12:27:24.337784 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:24.337742    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-msxcd" podStartSLOduration=2.223643015 podStartE2EDuration="3.337728263s" podCreationTimestamp="2026-05-25 12:27:21 +0000 UTC" firstStartedPulling="2026-05-25 12:27:22.258277602 +0000 UTC m=+855.218327986" lastFinishedPulling="2026-05-25 12:27:23.372362712 +0000 UTC m=+856.332413234" observedRunningTime="2026-05-25 12:27:24.335936143 +0000 UTC m=+857.295986549" watchObservedRunningTime="2026-05-25 12:27:24.337728263 +0000 UTC m=+857.297778668"
May 25 12:27:25.327308 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:25.327277    2555 generic.go:358] "Generic (PLEG): container finished" podID="a17746c5-1974-4937-9719-fe219bbb2c2f" containerID="0ac2ce3f8019560064dd4a81bf8a0ad6a20a6423dabe33694f12106447221688" exitCode=0
May 25 12:27:25.327773 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:25.327347    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q" event={"ID":"a17746c5-1974-4937-9719-fe219bbb2c2f","Type":"ContainerDied","Data":"0ac2ce3f8019560064dd4a81bf8a0ad6a20a6423dabe33694f12106447221688"}
May 25 12:27:25.327773 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:25.327730    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-msxcd"
May 25 12:27:25.329348 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:25.329329    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-msxcd"
May 25 12:27:29.042054 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:29.042015    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q" podUID="a17746c5-1974-4937-9719-fe219bbb2c2f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
May 25 12:27:29.046373 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:29.046345    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q" podUID="a17746c5-1974-4937-9719-fe219bbb2c2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:5000: connect: connection refused"
May 25 12:27:29.046669 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:29.046624    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q" podUID="a17746c5-1974-4937-9719-fe219bbb2c2f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:27:32.339058 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:32.339031    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-msxcd"
May 25 12:27:34.042522 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:34.042480    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q" podUID="a17746c5-1974-4937-9719-fe219bbb2c2f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
May 25 12:27:34.042944 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:34.042599    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q"
May 25 12:27:39.042031 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:39.041988    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q" podUID="a17746c5-1974-4937-9719-fe219bbb2c2f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
May 25 12:27:39.046315 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:39.046274    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q" podUID="a17746c5-1974-4937-9719-fe219bbb2c2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:5000: connect: connection refused"
May 25 12:27:39.046570 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:39.046552    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q" podUID="a17746c5-1974-4937-9719-fe219bbb2c2f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:27:41.277152 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:41.277116    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8"]
May 25 12:27:41.283832 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:41.283812    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8"
May 25 12:27:41.285657 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:41.285623    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
May 25 12:27:41.285766 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:41.285675    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
May 25 12:27:41.290872 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:41.290848    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8"]
May 25 12:27:41.308829 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:41.308807    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83193704-9e17-4504-87f4-d7c8a668f2a2-kserve-provision-location\") pod \"isvc-logger-predictor-6d94fc7bdd-gbxb8\" (UID: \"83193704-9e17-4504-87f4-d7c8a668f2a2\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8"
May 25 12:27:41.308924 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:41.308848    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb9z8\" (UniqueName: \"kubernetes.io/projected/83193704-9e17-4504-87f4-d7c8a668f2a2-kube-api-access-sb9z8\") pod \"isvc-logger-predictor-6d94fc7bdd-gbxb8\" (UID: \"83193704-9e17-4504-87f4-d7c8a668f2a2\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8"
May 25 12:27:41.308924 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:41.308889    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83193704-9e17-4504-87f4-d7c8a668f2a2-proxy-tls\") pod \"isvc-logger-predictor-6d94fc7bdd-gbxb8\" (UID: \"83193704-9e17-4504-87f4-d7c8a668f2a2\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8"
May 25 12:27:41.308924 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:41.308914    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-logger-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83193704-9e17-4504-87f4-d7c8a668f2a2-isvc-logger-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-6d94fc7bdd-gbxb8\" (UID: \"83193704-9e17-4504-87f4-d7c8a668f2a2\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8"
May 25 12:27:41.409756 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:41.409727    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83193704-9e17-4504-87f4-d7c8a668f2a2-proxy-tls\") pod \"isvc-logger-predictor-6d94fc7bdd-gbxb8\" (UID: \"83193704-9e17-4504-87f4-d7c8a668f2a2\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8"
May 25 12:27:41.409894 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:41.409770    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-logger-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83193704-9e17-4504-87f4-d7c8a668f2a2-isvc-logger-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-6d94fc7bdd-gbxb8\" (UID: \"83193704-9e17-4504-87f4-d7c8a668f2a2\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8"
May 25 12:27:41.409894 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:41.409816    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83193704-9e17-4504-87f4-d7c8a668f2a2-kserve-provision-location\") pod \"isvc-logger-predictor-6d94fc7bdd-gbxb8\" (UID: \"83193704-9e17-4504-87f4-d7c8a668f2a2\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8"
May 25 12:27:41.409894 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:41.409854    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sb9z8\" (UniqueName: \"kubernetes.io/projected/83193704-9e17-4504-87f4-d7c8a668f2a2-kube-api-access-sb9z8\") pod \"isvc-logger-predictor-6d94fc7bdd-gbxb8\" (UID: \"83193704-9e17-4504-87f4-d7c8a668f2a2\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8"
May 25 12:27:41.409894 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:27:41.409869    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-predictor-serving-cert: secret "isvc-logger-predictor-serving-cert" not found
May 25 12:27:41.410056 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:27:41.409934    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83193704-9e17-4504-87f4-d7c8a668f2a2-proxy-tls podName:83193704-9e17-4504-87f4-d7c8a668f2a2 nodeName:}" failed. No retries permitted until 2026-05-25 12:27:41.909914219 +0000 UTC m=+874.869964606 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/83193704-9e17-4504-87f4-d7c8a668f2a2-proxy-tls") pod "isvc-logger-predictor-6d94fc7bdd-gbxb8" (UID: "83193704-9e17-4504-87f4-d7c8a668f2a2") : secret "isvc-logger-predictor-serving-cert" not found
May 25 12:27:41.410264 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:41.410247    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83193704-9e17-4504-87f4-d7c8a668f2a2-kserve-provision-location\") pod \"isvc-logger-predictor-6d94fc7bdd-gbxb8\" (UID: \"83193704-9e17-4504-87f4-d7c8a668f2a2\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8"
May 25 12:27:41.410412 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:41.410395    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-logger-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83193704-9e17-4504-87f4-d7c8a668f2a2-isvc-logger-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-6d94fc7bdd-gbxb8\" (UID: \"83193704-9e17-4504-87f4-d7c8a668f2a2\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8"
May 25 12:27:41.418222 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:41.418202    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb9z8\" (UniqueName: \"kubernetes.io/projected/83193704-9e17-4504-87f4-d7c8a668f2a2-kube-api-access-sb9z8\") pod \"isvc-logger-predictor-6d94fc7bdd-gbxb8\" (UID: \"83193704-9e17-4504-87f4-d7c8a668f2a2\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8"
May 25 12:27:41.914506 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:41.914471    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83193704-9e17-4504-87f4-d7c8a668f2a2-proxy-tls\") pod \"isvc-logger-predictor-6d94fc7bdd-gbxb8\" (UID: \"83193704-9e17-4504-87f4-d7c8a668f2a2\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8"
May 25 12:27:41.916930 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:41.916898    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83193704-9e17-4504-87f4-d7c8a668f2a2-proxy-tls\") pod \"isvc-logger-predictor-6d94fc7bdd-gbxb8\" (UID: \"83193704-9e17-4504-87f4-d7c8a668f2a2\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8"
May 25 12:27:42.194708 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:42.194680    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8"
May 25 12:27:42.318295 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:42.318262    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8"]
May 25 12:27:42.321313 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:27:42.321264    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83193704_9e17_4504_87f4_d7c8a668f2a2.slice/crio-01d28f64afeb8491e6c5c8d86ca2dbc9eba891bf855fe946772d19efdf05803a WatchSource:0}: Error finding container 01d28f64afeb8491e6c5c8d86ca2dbc9eba891bf855fe946772d19efdf05803a: Status 404 returned error can't find the container with id 01d28f64afeb8491e6c5c8d86ca2dbc9eba891bf855fe946772d19efdf05803a
May 25 12:27:42.388146 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:42.388120    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8" event={"ID":"83193704-9e17-4504-87f4-d7c8a668f2a2","Type":"ContainerStarted","Data":"5f63ec91ab6a49b4a9793eac84bf4bfba7f452cea87be402a09fc797a3123160"}
May 25 12:27:42.388291 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:42.388154    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8" event={"ID":"83193704-9e17-4504-87f4-d7c8a668f2a2","Type":"ContainerStarted","Data":"01d28f64afeb8491e6c5c8d86ca2dbc9eba891bf855fe946772d19efdf05803a"}
May 25 12:27:44.042229 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:44.042188    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q" podUID="a17746c5-1974-4937-9719-fe219bbb2c2f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
May 25 12:27:46.404229 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:46.404142    2555 generic.go:358] "Generic (PLEG): container finished" podID="83193704-9e17-4504-87f4-d7c8a668f2a2" containerID="5f63ec91ab6a49b4a9793eac84bf4bfba7f452cea87be402a09fc797a3123160" exitCode=0
May 25 12:27:46.404229 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:46.404180    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8" event={"ID":"83193704-9e17-4504-87f4-d7c8a668f2a2","Type":"ContainerDied","Data":"5f63ec91ab6a49b4a9793eac84bf4bfba7f452cea87be402a09fc797a3123160"}
May 25 12:27:47.409085 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:47.409049    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8" event={"ID":"83193704-9e17-4504-87f4-d7c8a668f2a2","Type":"ContainerStarted","Data":"13b20bf771b6ba0b8205a998f3247489f01daf02cf0a4b4d189bcc616e22e692"}
May 25 12:27:47.409085 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:47.409090    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8" event={"ID":"83193704-9e17-4504-87f4-d7c8a668f2a2","Type":"ContainerStarted","Data":"58e75f46ae921fb83d5f44436a7c7d8bec7bfbd3d77d792ec93f06fd830599c0"}
May 25 12:27:47.409522 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:47.409100    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8" event={"ID":"83193704-9e17-4504-87f4-d7c8a668f2a2","Type":"ContainerStarted","Data":"cfc87e1f1b1ffb6318abe200c134efdecf2eab856dc211d9c3e23053e4eb02a7"}
May 25 12:27:47.409522 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:47.409376    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8"
May 25 12:27:47.409522 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:47.409508    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8"
May 25 12:27:47.410741 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:47.410716    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8" podUID="83193704-9e17-4504-87f4-d7c8a668f2a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
May 25 12:27:47.426783 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:47.426747    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8" podStartSLOduration=6.426736182 podStartE2EDuration="6.426736182s" podCreationTimestamp="2026-05-25 12:27:41 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:27:47.425850201 +0000 UTC m=+880.385900609" watchObservedRunningTime="2026-05-25 12:27:47.426736182 +0000 UTC m=+880.386786586"
May 25 12:27:48.413013 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:48.412971    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8"
May 25 12:27:48.413455 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:48.413062    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8" podUID="83193704-9e17-4504-87f4-d7c8a668f2a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
May 25 12:27:48.414026 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:48.414001    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8" podUID="83193704-9e17-4504-87f4-d7c8a668f2a2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:27:49.042415 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:49.042377    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q" podUID="a17746c5-1974-4937-9719-fe219bbb2c2f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
May 25 12:27:49.046769 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:49.046730    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q" podUID="a17746c5-1974-4937-9719-fe219bbb2c2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:5000: connect: connection refused"
May 25 12:27:49.046911 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:49.046868    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q"
May 25 12:27:49.047015 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:49.046980    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q" podUID="a17746c5-1974-4937-9719-fe219bbb2c2f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:27:49.047113 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:49.047092    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q"
May 25 12:27:49.416735 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:49.416634    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8" podUID="83193704-9e17-4504-87f4-d7c8a668f2a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
May 25 12:27:49.417111 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:49.417007    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8" podUID="83193704-9e17-4504-87f4-d7c8a668f2a2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:27:51.298348 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:51.298326    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q"
May 25 12:27:51.392963 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:51.392938    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a17746c5-1974-4937-9719-fe219bbb2c2f-proxy-tls\") pod \"a17746c5-1974-4937-9719-fe219bbb2c2f\" (UID: \"a17746c5-1974-4937-9719-fe219bbb2c2f\") "
May 25 12:27:51.393175 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:51.392991    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a17746c5-1974-4937-9719-fe219bbb2c2f-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\") pod \"a17746c5-1974-4937-9719-fe219bbb2c2f\" (UID: \"a17746c5-1974-4937-9719-fe219bbb2c2f\") "
May 25 12:27:51.393175 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:51.393049    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a17746c5-1974-4937-9719-fe219bbb2c2f-kserve-provision-location\") pod \"a17746c5-1974-4937-9719-fe219bbb2c2f\" (UID: \"a17746c5-1974-4937-9719-fe219bbb2c2f\") "
May 25 12:27:51.393175 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:51.393092    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7q28p\" (UniqueName: \"kubernetes.io/projected/a17746c5-1974-4937-9719-fe219bbb2c2f-kube-api-access-7q28p\") pod \"a17746c5-1974-4937-9719-fe219bbb2c2f\" (UID: \"a17746c5-1974-4937-9719-fe219bbb2c2f\") "
May 25 12:27:51.393417 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:51.393385    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a17746c5-1974-4937-9719-fe219bbb2c2f-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config") pod "a17746c5-1974-4937-9719-fe219bbb2c2f" (UID: "a17746c5-1974-4937-9719-fe219bbb2c2f"). InnerVolumeSpecName "isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:27:51.393417 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:51.393386    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a17746c5-1974-4937-9719-fe219bbb2c2f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a17746c5-1974-4937-9719-fe219bbb2c2f" (UID: "a17746c5-1974-4937-9719-fe219bbb2c2f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:27:51.394962 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:51.394939    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a17746c5-1974-4937-9719-fe219bbb2c2f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a17746c5-1974-4937-9719-fe219bbb2c2f" (UID: "a17746c5-1974-4937-9719-fe219bbb2c2f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:27:51.395155 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:51.395136    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a17746c5-1974-4937-9719-fe219bbb2c2f-kube-api-access-7q28p" (OuterVolumeSpecName: "kube-api-access-7q28p") pod "a17746c5-1974-4937-9719-fe219bbb2c2f" (UID: "a17746c5-1974-4937-9719-fe219bbb2c2f"). InnerVolumeSpecName "kube-api-access-7q28p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:27:51.427216 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:51.427152    2555 generic.go:358] "Generic (PLEG): container finished" podID="a17746c5-1974-4937-9719-fe219bbb2c2f" containerID="f3d5d1889a0021175e46749151a1521b976642e962ecaa5f0c326154702640f7" exitCode=0
May 25 12:27:51.427309 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:51.427236    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q"
May 25 12:27:51.427309 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:51.427237    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q" event={"ID":"a17746c5-1974-4937-9719-fe219bbb2c2f","Type":"ContainerDied","Data":"f3d5d1889a0021175e46749151a1521b976642e962ecaa5f0c326154702640f7"}
May 25 12:27:51.427309 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:51.427279    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q" event={"ID":"a17746c5-1974-4937-9719-fe219bbb2c2f","Type":"ContainerDied","Data":"df9ff8c0fbe8f5b5c564d83f70367e3f2a4ee3b96225a61b46d98609f0b34eba"}
May 25 12:27:51.427309 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:51.427294    2555 scope.go:117] "RemoveContainer" containerID="f3d5d1889a0021175e46749151a1521b976642e962ecaa5f0c326154702640f7"
May 25 12:27:51.435201 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:51.435184    2555 scope.go:117] "RemoveContainer" containerID="c9ea10451ff0369dfca226a0ce9ff4a11ba73940e929717b43c518130dbcc156"
May 25 12:27:51.442143 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:51.442127    2555 scope.go:117] "RemoveContainer" containerID="0ac2ce3f8019560064dd4a81bf8a0ad6a20a6423dabe33694f12106447221688"
May 25 12:27:51.447720 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:51.447700    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q"]
May 25 12:27:51.449443 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:51.449424    2555 scope.go:117] "RemoveContainer" containerID="6ca031ec42d74bf2315106ec855f94f171f7b0c2b43c890efff26228c580b3d3"
May 25 12:27:51.451432 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:51.451415    2555 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-hfb6q"]
May 25 12:27:51.456365 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:51.456347    2555 scope.go:117] "RemoveContainer" containerID="f3d5d1889a0021175e46749151a1521b976642e962ecaa5f0c326154702640f7"
May 25 12:27:51.456592 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:27:51.456576    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3d5d1889a0021175e46749151a1521b976642e962ecaa5f0c326154702640f7\": container with ID starting with f3d5d1889a0021175e46749151a1521b976642e962ecaa5f0c326154702640f7 not found: ID does not exist" containerID="f3d5d1889a0021175e46749151a1521b976642e962ecaa5f0c326154702640f7"
May 25 12:27:51.456652 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:51.456598    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3d5d1889a0021175e46749151a1521b976642e962ecaa5f0c326154702640f7"} err="failed to get container status \"f3d5d1889a0021175e46749151a1521b976642e962ecaa5f0c326154702640f7\": rpc error: code = NotFound desc = could not find container \"f3d5d1889a0021175e46749151a1521b976642e962ecaa5f0c326154702640f7\": container with ID starting with f3d5d1889a0021175e46749151a1521b976642e962ecaa5f0c326154702640f7 not found: ID does not exist"
May 25 12:27:51.456652 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:51.456614    2555 scope.go:117] "RemoveContainer" containerID="c9ea10451ff0369dfca226a0ce9ff4a11ba73940e929717b43c518130dbcc156"
May 25 12:27:51.456865 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:27:51.456850    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9ea10451ff0369dfca226a0ce9ff4a11ba73940e929717b43c518130dbcc156\": container with ID starting with c9ea10451ff0369dfca226a0ce9ff4a11ba73940e929717b43c518130dbcc156 not found: ID does not exist" containerID="c9ea10451ff0369dfca226a0ce9ff4a11ba73940e929717b43c518130dbcc156"
May 25 12:27:51.456910 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:51.456868    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9ea10451ff0369dfca226a0ce9ff4a11ba73940e929717b43c518130dbcc156"} err="failed to get container status \"c9ea10451ff0369dfca226a0ce9ff4a11ba73940e929717b43c518130dbcc156\": rpc error: code = NotFound desc = could not find container \"c9ea10451ff0369dfca226a0ce9ff4a11ba73940e929717b43c518130dbcc156\": container with ID starting with c9ea10451ff0369dfca226a0ce9ff4a11ba73940e929717b43c518130dbcc156 not found: ID does not exist"
May 25 12:27:51.456910 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:51.456880    2555 scope.go:117] "RemoveContainer" containerID="0ac2ce3f8019560064dd4a81bf8a0ad6a20a6423dabe33694f12106447221688"
May 25 12:27:51.457075 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:27:51.457059    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ac2ce3f8019560064dd4a81bf8a0ad6a20a6423dabe33694f12106447221688\": container with ID starting with 0ac2ce3f8019560064dd4a81bf8a0ad6a20a6423dabe33694f12106447221688 not found: ID does not exist" containerID="0ac2ce3f8019560064dd4a81bf8a0ad6a20a6423dabe33694f12106447221688"
May 25 12:27:51.457115 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:51.457080    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ac2ce3f8019560064dd4a81bf8a0ad6a20a6423dabe33694f12106447221688"} err="failed to get container status \"0ac2ce3f8019560064dd4a81bf8a0ad6a20a6423dabe33694f12106447221688\": rpc error: code = NotFound desc = could not find container \"0ac2ce3f8019560064dd4a81bf8a0ad6a20a6423dabe33694f12106447221688\": container with ID starting with 0ac2ce3f8019560064dd4a81bf8a0ad6a20a6423dabe33694f12106447221688 not found: ID does not exist"
May 25 12:27:51.457115 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:51.457093    2555 scope.go:117] "RemoveContainer" containerID="6ca031ec42d74bf2315106ec855f94f171f7b0c2b43c890efff26228c580b3d3"
May 25 12:27:51.457304 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:27:51.457288    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ca031ec42d74bf2315106ec855f94f171f7b0c2b43c890efff26228c580b3d3\": container with ID starting with 6ca031ec42d74bf2315106ec855f94f171f7b0c2b43c890efff26228c580b3d3 not found: ID does not exist" containerID="6ca031ec42d74bf2315106ec855f94f171f7b0c2b43c890efff26228c580b3d3"
May 25 12:27:51.457343 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:51.457308    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ca031ec42d74bf2315106ec855f94f171f7b0c2b43c890efff26228c580b3d3"} err="failed to get container status \"6ca031ec42d74bf2315106ec855f94f171f7b0c2b43c890efff26228c580b3d3\": rpc error: code = NotFound desc = could not find container \"6ca031ec42d74bf2315106ec855f94f171f7b0c2b43c890efff26228c580b3d3\": container with ID starting with 6ca031ec42d74bf2315106ec855f94f171f7b0c2b43c890efff26228c580b3d3 not found: ID does not exist"
May 25 12:27:51.493900 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:51.493878    2555 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a17746c5-1974-4937-9719-fe219bbb2c2f-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:27:51.493900 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:51.493899    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a17746c5-1974-4937-9719-fe219bbb2c2f-kserve-provision-location\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:27:51.494011 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:51.493910    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7q28p\" (UniqueName: \"kubernetes.io/projected/a17746c5-1974-4937-9719-fe219bbb2c2f-kube-api-access-7q28p\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:27:51.494011 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:51.493920    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a17746c5-1974-4937-9719-fe219bbb2c2f-proxy-tls\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:27:51.598254 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:51.598230    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a17746c5-1974-4937-9719-fe219bbb2c2f" path="/var/lib/kubelet/pods/a17746c5-1974-4937-9719-fe219bbb2c2f/volumes"
May 25 12:27:54.420332 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:54.420307    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8"
May 25 12:27:54.420938 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:54.420907    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8" podUID="83193704-9e17-4504-87f4-d7c8a668f2a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
May 25 12:27:54.421255 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:27:54.421235    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8" podUID="83193704-9e17-4504-87f4-d7c8a668f2a2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:28:04.420929 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:28:04.420838    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8" podUID="83193704-9e17-4504-87f4-d7c8a668f2a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
May 25 12:28:04.421326 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:28:04.421175    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8" podUID="83193704-9e17-4504-87f4-d7c8a668f2a2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:28:07.529580 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:28:07.529550    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-2tqtm_00e6f79c-8f05-4330-aa16-1ada2ae9e9cc/console-operator/1.log"
May 25 12:28:07.532048 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:28:07.532025    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-2tqtm_00e6f79c-8f05-4330-aa16-1ada2ae9e9cc/console-operator/1.log"
May 25 12:28:07.554056 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:28:07.554034    2555 scope.go:117] "RemoveContainer" containerID="3f7928b6173643e6e5316654f95f70a602d9bdffce32960a21c0e61e866968b5"
May 25 12:28:14.421575 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:28:14.421529    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8" podUID="83193704-9e17-4504-87f4-d7c8a668f2a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
May 25 12:28:14.422060 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:28:14.421984    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8" podUID="83193704-9e17-4504-87f4-d7c8a668f2a2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:28:24.421521 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:28:24.421466    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8" podUID="83193704-9e17-4504-87f4-d7c8a668f2a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
May 25 12:28:24.422087 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:28:24.421964    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8" podUID="83193704-9e17-4504-87f4-d7c8a668f2a2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:28:34.421583 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:28:34.421540    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8" podUID="83193704-9e17-4504-87f4-d7c8a668f2a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
May 25 12:28:34.422045 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:28:34.422019    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8" podUID="83193704-9e17-4504-87f4-d7c8a668f2a2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:28:44.421654 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:28:44.421604    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8" podUID="83193704-9e17-4504-87f4-d7c8a668f2a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
May 25 12:28:44.424152 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:28:44.422022    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8" podUID="83193704-9e17-4504-87f4-d7c8a668f2a2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:28:54.421421 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:28:54.421392    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8"
May 25 12:28:54.421864 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:28:54.421698    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8"
May 25 12:29:06.325819 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:06.325795    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-c7d86bcbd-msxcd_c80fbc59-fd33-487b-94f2-24ec039041b6/kserve-container/0.log"
May 25 12:29:06.470918 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:06.470884    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8"]
May 25 12:29:06.471260 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:06.471228    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8" podUID="83193704-9e17-4504-87f4-d7c8a668f2a2" containerName="kserve-container" containerID="cri-o://cfc87e1f1b1ffb6318abe200c134efdecf2eab856dc211d9c3e23053e4eb02a7" gracePeriod=30
May 25 12:29:06.471260 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:06.471241    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8" podUID="83193704-9e17-4504-87f4-d7c8a668f2a2" containerName="agent" containerID="cri-o://13b20bf771b6ba0b8205a998f3247489f01daf02cf0a4b4d189bcc616e22e692" gracePeriod=30
May 25 12:29:06.471413 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:06.471304    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8" podUID="83193704-9e17-4504-87f4-d7c8a668f2a2" containerName="kube-rbac-proxy" containerID="cri-o://58e75f46ae921fb83d5f44436a7c7d8bec7bfbd3d77d792ec93f06fd830599c0" gracePeriod=30
May 25 12:29:06.558157 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:06.558126    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-h5dr4"]
May 25 12:29:06.558530 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:06.558516    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a17746c5-1974-4937-9719-fe219bbb2c2f" containerName="storage-initializer"
May 25 12:29:06.558581 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:06.558532    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a17746c5-1974-4937-9719-fe219bbb2c2f" containerName="storage-initializer"
May 25 12:29:06.558581 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:06.558544    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a17746c5-1974-4937-9719-fe219bbb2c2f" containerName="kube-rbac-proxy"
May 25 12:29:06.558581 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:06.558549    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a17746c5-1974-4937-9719-fe219bbb2c2f" containerName="kube-rbac-proxy"
May 25 12:29:06.558581 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:06.558562    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a17746c5-1974-4937-9719-fe219bbb2c2f" containerName="kserve-container"
May 25 12:29:06.558581 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:06.558568    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a17746c5-1974-4937-9719-fe219bbb2c2f" containerName="kserve-container"
May 25 12:29:06.558581 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:06.558576    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a17746c5-1974-4937-9719-fe219bbb2c2f" containerName="agent"
May 25 12:29:06.558581 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:06.558581    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a17746c5-1974-4937-9719-fe219bbb2c2f" containerName="agent"
May 25 12:29:06.558834 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:06.558631    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="a17746c5-1974-4937-9719-fe219bbb2c2f" containerName="agent"
May 25 12:29:06.558834 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:06.558655    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="a17746c5-1974-4937-9719-fe219bbb2c2f" containerName="kserve-container"
May 25 12:29:06.558834 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:06.558667    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="a17746c5-1974-4937-9719-fe219bbb2c2f" containerName="kube-rbac-proxy"
May 25 12:29:06.562078 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:06.562045    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-h5dr4"
May 25 12:29:06.563903 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:06.563881    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
May 25 12:29:06.564067 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:06.564048    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-predictor-serving-cert\""
May 25 12:29:06.570852 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:06.570598    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-h5dr4"]
May 25 12:29:06.575689 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:06.575667    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-msxcd"]
May 25 12:29:06.575979 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:06.575920    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-msxcd" podUID="c80fbc59-fd33-487b-94f2-24ec039041b6" containerName="kserve-container" containerID="cri-o://66cf69eb511c53eba10f7defe3a3593505762826a3dddab92eef317a490b98ed" gracePeriod=30
May 25 12:29:06.576053 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:06.575997    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-msxcd" podUID="c80fbc59-fd33-487b-94f2-24ec039041b6" containerName="kube-rbac-proxy" containerID="cri-o://b1901ccc00bb67525c96bc49bcc7a879d9ab5d377c179610b2e25bfa882c44b9" gracePeriod=30
May 25 12:29:06.603635 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:06.603605    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c52n9\" (UniqueName: \"kubernetes.io/projected/fb17c480-0ff2-4ef0-9009-8acc07ed73fb-kube-api-access-c52n9\") pod \"isvc-lightgbm-predictor-bdf964bd-h5dr4\" (UID: \"fb17c480-0ff2-4ef0-9009-8acc07ed73fb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-h5dr4"
May 25 12:29:06.603753 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:06.603705    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb17c480-0ff2-4ef0-9009-8acc07ed73fb-kserve-provision-location\") pod \"isvc-lightgbm-predictor-bdf964bd-h5dr4\" (UID: \"fb17c480-0ff2-4ef0-9009-8acc07ed73fb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-h5dr4"
May 25 12:29:06.603753 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:06.603739    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb17c480-0ff2-4ef0-9009-8acc07ed73fb-isvc-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-bdf964bd-h5dr4\" (UID: \"fb17c480-0ff2-4ef0-9009-8acc07ed73fb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-h5dr4"
May 25 12:29:06.603893 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:06.603831    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb17c480-0ff2-4ef0-9009-8acc07ed73fb-proxy-tls\") pod \"isvc-lightgbm-predictor-bdf964bd-h5dr4\" (UID: \"fb17c480-0ff2-4ef0-9009-8acc07ed73fb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-h5dr4"
May 25 12:29:06.689857 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:06.689827    2555 generic.go:358] "Generic (PLEG): container finished" podID="83193704-9e17-4504-87f4-d7c8a668f2a2" containerID="58e75f46ae921fb83d5f44436a7c7d8bec7bfbd3d77d792ec93f06fd830599c0" exitCode=2
May 25 12:29:06.689979 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:06.689900    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8" event={"ID":"83193704-9e17-4504-87f4-d7c8a668f2a2","Type":"ContainerDied","Data":"58e75f46ae921fb83d5f44436a7c7d8bec7bfbd3d77d792ec93f06fd830599c0"}
May 25 12:29:06.704513 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:06.704487    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb17c480-0ff2-4ef0-9009-8acc07ed73fb-kserve-provision-location\") pod \"isvc-lightgbm-predictor-bdf964bd-h5dr4\" (UID: \"fb17c480-0ff2-4ef0-9009-8acc07ed73fb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-h5dr4"
May 25 12:29:06.704678 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:06.704547    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb17c480-0ff2-4ef0-9009-8acc07ed73fb-isvc-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-bdf964bd-h5dr4\" (UID: \"fb17c480-0ff2-4ef0-9009-8acc07ed73fb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-h5dr4"
May 25 12:29:06.704761 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:06.704698    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb17c480-0ff2-4ef0-9009-8acc07ed73fb-proxy-tls\") pod \"isvc-lightgbm-predictor-bdf964bd-h5dr4\" (UID: \"fb17c480-0ff2-4ef0-9009-8acc07ed73fb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-h5dr4"
May 25 12:29:06.704879 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:06.704855    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-c52n9\" (UniqueName: \"kubernetes.io/projected/fb17c480-0ff2-4ef0-9009-8acc07ed73fb-kube-api-access-c52n9\") pod \"isvc-lightgbm-predictor-bdf964bd-h5dr4\" (UID: \"fb17c480-0ff2-4ef0-9009-8acc07ed73fb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-h5dr4"
May 25 12:29:06.704941 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:06.704898    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb17c480-0ff2-4ef0-9009-8acc07ed73fb-kserve-provision-location\") pod \"isvc-lightgbm-predictor-bdf964bd-h5dr4\" (UID: \"fb17c480-0ff2-4ef0-9009-8acc07ed73fb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-h5dr4"
May 25 12:29:06.705439 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:06.705307    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb17c480-0ff2-4ef0-9009-8acc07ed73fb-isvc-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-bdf964bd-h5dr4\" (UID: \"fb17c480-0ff2-4ef0-9009-8acc07ed73fb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-h5dr4"
May 25 12:29:06.707944 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:06.707920    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb17c480-0ff2-4ef0-9009-8acc07ed73fb-proxy-tls\") pod \"isvc-lightgbm-predictor-bdf964bd-h5dr4\" (UID: \"fb17c480-0ff2-4ef0-9009-8acc07ed73fb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-h5dr4"
May 25 12:29:06.711676 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:06.711626    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c52n9\" (UniqueName: \"kubernetes.io/projected/fb17c480-0ff2-4ef0-9009-8acc07ed73fb-kube-api-access-c52n9\") pod \"isvc-lightgbm-predictor-bdf964bd-h5dr4\" (UID: \"fb17c480-0ff2-4ef0-9009-8acc07ed73fb\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-h5dr4"
May 25 12:29:06.817275 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:06.817253    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-msxcd"
May 25 12:29:06.876284 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:06.876225    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-h5dr4"
May 25 12:29:06.906952 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:06.906924    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c80fbc59-fd33-487b-94f2-24ec039041b6-proxy-tls\") pod \"c80fbc59-fd33-487b-94f2-24ec039041b6\" (UID: \"c80fbc59-fd33-487b-94f2-24ec039041b6\") "
May 25 12:29:06.907070 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:06.907014    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"message-dumper-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c80fbc59-fd33-487b-94f2-24ec039041b6-message-dumper-kube-rbac-proxy-sar-config\") pod \"c80fbc59-fd33-487b-94f2-24ec039041b6\" (UID: \"c80fbc59-fd33-487b-94f2-24ec039041b6\") "
May 25 12:29:06.907134 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:06.907067    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4drh\" (UniqueName: \"kubernetes.io/projected/c80fbc59-fd33-487b-94f2-24ec039041b6-kube-api-access-l4drh\") pod \"c80fbc59-fd33-487b-94f2-24ec039041b6\" (UID: \"c80fbc59-fd33-487b-94f2-24ec039041b6\") "
May 25 12:29:06.907450 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:06.907417    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c80fbc59-fd33-487b-94f2-24ec039041b6-message-dumper-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "message-dumper-kube-rbac-proxy-sar-config") pod "c80fbc59-fd33-487b-94f2-24ec039041b6" (UID: "c80fbc59-fd33-487b-94f2-24ec039041b6"). InnerVolumeSpecName "message-dumper-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:29:06.908956 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:06.908933    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c80fbc59-fd33-487b-94f2-24ec039041b6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c80fbc59-fd33-487b-94f2-24ec039041b6" (UID: "c80fbc59-fd33-487b-94f2-24ec039041b6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:29:06.909050 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:06.909030    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c80fbc59-fd33-487b-94f2-24ec039041b6-kube-api-access-l4drh" (OuterVolumeSpecName: "kube-api-access-l4drh") pod "c80fbc59-fd33-487b-94f2-24ec039041b6" (UID: "c80fbc59-fd33-487b-94f2-24ec039041b6"). InnerVolumeSpecName "kube-api-access-l4drh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:29:07.007020 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:07.006996    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-h5dr4"]
May 25 12:29:07.007719 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:07.007700    2555 reconciler_common.go:299] "Volume detached for volume \"message-dumper-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c80fbc59-fd33-487b-94f2-24ec039041b6-message-dumper-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:29:07.007771 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:07.007725    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4drh\" (UniqueName: \"kubernetes.io/projected/c80fbc59-fd33-487b-94f2-24ec039041b6-kube-api-access-l4drh\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:29:07.007771 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:07.007738    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c80fbc59-fd33-487b-94f2-24ec039041b6-proxy-tls\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:29:07.008924 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:29:07.008902    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb17c480_0ff2_4ef0_9009_8acc07ed73fb.slice/crio-068a1ba39fab6f98d7b24a04932f0bcf54920aff6597d8b6fa06e35cda9e1654 WatchSource:0}: Error finding container 068a1ba39fab6f98d7b24a04932f0bcf54920aff6597d8b6fa06e35cda9e1654: Status 404 returned error can't find the container with id 068a1ba39fab6f98d7b24a04932f0bcf54920aff6597d8b6fa06e35cda9e1654
May 25 12:29:07.010721 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:07.010707    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 12:29:07.695057 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:07.695026    2555 generic.go:358] "Generic (PLEG): container finished" podID="c80fbc59-fd33-487b-94f2-24ec039041b6" containerID="b1901ccc00bb67525c96bc49bcc7a879d9ab5d377c179610b2e25bfa882c44b9" exitCode=2
May 25 12:29:07.695057 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:07.695053    2555 generic.go:358] "Generic (PLEG): container finished" podID="c80fbc59-fd33-487b-94f2-24ec039041b6" containerID="66cf69eb511c53eba10f7defe3a3593505762826a3dddab92eef317a490b98ed" exitCode=2
May 25 12:29:07.695516 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:07.695101    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-msxcd"
May 25 12:29:07.695516 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:07.695109    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-msxcd" event={"ID":"c80fbc59-fd33-487b-94f2-24ec039041b6","Type":"ContainerDied","Data":"b1901ccc00bb67525c96bc49bcc7a879d9ab5d377c179610b2e25bfa882c44b9"}
May 25 12:29:07.695516 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:07.695148    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-msxcd" event={"ID":"c80fbc59-fd33-487b-94f2-24ec039041b6","Type":"ContainerDied","Data":"66cf69eb511c53eba10f7defe3a3593505762826a3dddab92eef317a490b98ed"}
May 25 12:29:07.695516 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:07.695160    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-msxcd" event={"ID":"c80fbc59-fd33-487b-94f2-24ec039041b6","Type":"ContainerDied","Data":"23ca8f75daf209adc909753d2fb142d00254b6ef33dfc3014f8499c095a4e975"}
May 25 12:29:07.695516 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:07.695174    2555 scope.go:117] "RemoveContainer" containerID="b1901ccc00bb67525c96bc49bcc7a879d9ab5d377c179610b2e25bfa882c44b9"
May 25 12:29:07.696849 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:07.696826    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-h5dr4" event={"ID":"fb17c480-0ff2-4ef0-9009-8acc07ed73fb","Type":"ContainerStarted","Data":"44d4159ec92ce87f2614eac72624fccc43de493b8917a022c04ba94bd58cbb7e"}
May 25 12:29:07.696971 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:07.696860    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-h5dr4" event={"ID":"fb17c480-0ff2-4ef0-9009-8acc07ed73fb","Type":"ContainerStarted","Data":"068a1ba39fab6f98d7b24a04932f0bcf54920aff6597d8b6fa06e35cda9e1654"}
May 25 12:29:07.703544 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:07.703505    2555 scope.go:117] "RemoveContainer" containerID="66cf69eb511c53eba10f7defe3a3593505762826a3dddab92eef317a490b98ed"
May 25 12:29:07.710786 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:07.710766    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-msxcd"]
May 25 12:29:07.711870 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:07.711851    2555 scope.go:117] "RemoveContainer" containerID="b1901ccc00bb67525c96bc49bcc7a879d9ab5d377c179610b2e25bfa882c44b9"
May 25 12:29:07.712347 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:29:07.712297    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1901ccc00bb67525c96bc49bcc7a879d9ab5d377c179610b2e25bfa882c44b9\": container with ID starting with b1901ccc00bb67525c96bc49bcc7a879d9ab5d377c179610b2e25bfa882c44b9 not found: ID does not exist" containerID="b1901ccc00bb67525c96bc49bcc7a879d9ab5d377c179610b2e25bfa882c44b9"
May 25 12:29:07.712447 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:07.712354    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1901ccc00bb67525c96bc49bcc7a879d9ab5d377c179610b2e25bfa882c44b9"} err="failed to get container status \"b1901ccc00bb67525c96bc49bcc7a879d9ab5d377c179610b2e25bfa882c44b9\": rpc error: code = NotFound desc = could not find container \"b1901ccc00bb67525c96bc49bcc7a879d9ab5d377c179610b2e25bfa882c44b9\": container with ID starting with b1901ccc00bb67525c96bc49bcc7a879d9ab5d377c179610b2e25bfa882c44b9 not found: ID does not exist"
May 25 12:29:07.712447 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:07.712378    2555 scope.go:117] "RemoveContainer" containerID="66cf69eb511c53eba10f7defe3a3593505762826a3dddab92eef317a490b98ed"
May 25 12:29:07.712723 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:29:07.712696    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66cf69eb511c53eba10f7defe3a3593505762826a3dddab92eef317a490b98ed\": container with ID starting with 66cf69eb511c53eba10f7defe3a3593505762826a3dddab92eef317a490b98ed not found: ID does not exist" containerID="66cf69eb511c53eba10f7defe3a3593505762826a3dddab92eef317a490b98ed"
May 25 12:29:07.712832 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:07.712729    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66cf69eb511c53eba10f7defe3a3593505762826a3dddab92eef317a490b98ed"} err="failed to get container status \"66cf69eb511c53eba10f7defe3a3593505762826a3dddab92eef317a490b98ed\": rpc error: code = NotFound desc = could not find container \"66cf69eb511c53eba10f7defe3a3593505762826a3dddab92eef317a490b98ed\": container with ID starting with 66cf69eb511c53eba10f7defe3a3593505762826a3dddab92eef317a490b98ed not found: ID does not exist"
May 25 12:29:07.712832 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:07.712751    2555 scope.go:117] "RemoveContainer" containerID="b1901ccc00bb67525c96bc49bcc7a879d9ab5d377c179610b2e25bfa882c44b9"
May 25 12:29:07.713084 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:07.713045    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1901ccc00bb67525c96bc49bcc7a879d9ab5d377c179610b2e25bfa882c44b9"} err="failed to get container status \"b1901ccc00bb67525c96bc49bcc7a879d9ab5d377c179610b2e25bfa882c44b9\": rpc error: code = NotFound desc = could not find container \"b1901ccc00bb67525c96bc49bcc7a879d9ab5d377c179610b2e25bfa882c44b9\": container with ID starting with b1901ccc00bb67525c96bc49bcc7a879d9ab5d377c179610b2e25bfa882c44b9 not found: ID does not exist"
May 25 12:29:07.713171 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:07.713087    2555 scope.go:117] "RemoveContainer" containerID="66cf69eb511c53eba10f7defe3a3593505762826a3dddab92eef317a490b98ed"
May 25 12:29:07.713376 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:07.713326    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66cf69eb511c53eba10f7defe3a3593505762826a3dddab92eef317a490b98ed"} err="failed to get container status \"66cf69eb511c53eba10f7defe3a3593505762826a3dddab92eef317a490b98ed\": rpc error: code = NotFound desc = could not find container \"66cf69eb511c53eba10f7defe3a3593505762826a3dddab92eef317a490b98ed\": container with ID starting with 66cf69eb511c53eba10f7defe3a3593505762826a3dddab92eef317a490b98ed not found: ID does not exist"
May 25 12:29:07.713951 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:07.713929    2555 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-msxcd"]
May 25 12:29:09.417114 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:09.417072    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8" podUID="83193704-9e17-4504-87f4-d7c8a668f2a2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused"
May 25 12:29:09.599418 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:09.599385    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c80fbc59-fd33-487b-94f2-24ec039041b6" path="/var/lib/kubelet/pods/c80fbc59-fd33-487b-94f2-24ec039041b6/volumes"
May 25 12:29:10.710180 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:10.710148    2555 generic.go:358] "Generic (PLEG): container finished" podID="83193704-9e17-4504-87f4-d7c8a668f2a2" containerID="cfc87e1f1b1ffb6318abe200c134efdecf2eab856dc211d9c3e23053e4eb02a7" exitCode=0
May 25 12:29:10.710518 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:10.710216    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8" event={"ID":"83193704-9e17-4504-87f4-d7c8a668f2a2","Type":"ContainerDied","Data":"cfc87e1f1b1ffb6318abe200c134efdecf2eab856dc211d9c3e23053e4eb02a7"}
May 25 12:29:11.715041 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:11.715007    2555 generic.go:358] "Generic (PLEG): container finished" podID="fb17c480-0ff2-4ef0-9009-8acc07ed73fb" containerID="44d4159ec92ce87f2614eac72624fccc43de493b8917a022c04ba94bd58cbb7e" exitCode=0
May 25 12:29:11.715381 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:11.715079    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-h5dr4" event={"ID":"fb17c480-0ff2-4ef0-9009-8acc07ed73fb","Type":"ContainerDied","Data":"44d4159ec92ce87f2614eac72624fccc43de493b8917a022c04ba94bd58cbb7e"}
May 25 12:29:14.417500 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:14.417455    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8" podUID="83193704-9e17-4504-87f4-d7c8a668f2a2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused"
May 25 12:29:14.420968 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:14.420930    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8" podUID="83193704-9e17-4504-87f4-d7c8a668f2a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
May 25 12:29:14.421455 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:14.421426    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8" podUID="83193704-9e17-4504-87f4-d7c8a668f2a2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:29:18.745826 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:18.745789    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-h5dr4" event={"ID":"fb17c480-0ff2-4ef0-9009-8acc07ed73fb","Type":"ContainerStarted","Data":"2a09a57708a5eb05d4a8e8043b1455b91f4a1adaf54eb5f5f950b195fc776ff2"}
May 25 12:29:18.745826 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:18.745832    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-h5dr4" event={"ID":"fb17c480-0ff2-4ef0-9009-8acc07ed73fb","Type":"ContainerStarted","Data":"f84258eb850d6f28ef2b389fe39ea666a6d7d91ac8573682516b0d12f4835b90"}
May 25 12:29:18.746303 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:18.746017    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-h5dr4"
May 25 12:29:18.763765 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:18.763710    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-h5dr4" podStartSLOduration=6.224925695 podStartE2EDuration="12.763696051s" podCreationTimestamp="2026-05-25 12:29:06 +0000 UTC" firstStartedPulling="2026-05-25 12:29:11.716276279 +0000 UTC m=+964.676326663" lastFinishedPulling="2026-05-25 12:29:18.25504662 +0000 UTC m=+971.215097019" observedRunningTime="2026-05-25 12:29:18.761232289 +0000 UTC m=+971.721282696" watchObservedRunningTime="2026-05-25 12:29:18.763696051 +0000 UTC m=+971.723746462"
May 25 12:29:19.416904 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:19.416859    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8" podUID="83193704-9e17-4504-87f4-d7c8a668f2a2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused"
May 25 12:29:19.417090 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:19.417028    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8"
May 25 12:29:19.749541 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:19.749516    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-h5dr4"
May 25 12:29:19.750633 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:19.750600    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-h5dr4" podUID="fb17c480-0ff2-4ef0-9009-8acc07ed73fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
May 25 12:29:20.752519 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:20.752477    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-h5dr4" podUID="fb17c480-0ff2-4ef0-9009-8acc07ed73fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
May 25 12:29:24.417537 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:24.417491    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8" podUID="83193704-9e17-4504-87f4-d7c8a668f2a2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused"
May 25 12:29:24.421835 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:24.421808    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8" podUID="83193704-9e17-4504-87f4-d7c8a668f2a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
May 25 12:29:24.422237 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:24.422207    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8" podUID="83193704-9e17-4504-87f4-d7c8a668f2a2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:29:25.757017 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:25.756986    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-h5dr4"
May 25 12:29:25.757609 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:25.757581    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-h5dr4" podUID="fb17c480-0ff2-4ef0-9009-8acc07ed73fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
May 25 12:29:29.417213 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:29.417120    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8" podUID="83193704-9e17-4504-87f4-d7c8a668f2a2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused"
May 25 12:29:34.417090 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:34.417038    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8" podUID="83193704-9e17-4504-87f4-d7c8a668f2a2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused"
May 25 12:29:34.421409 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:34.421372    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8" podUID="83193704-9e17-4504-87f4-d7c8a668f2a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
May 25 12:29:34.421553 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:34.421529    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8"
May 25 12:29:34.421815 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:34.421781    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8" podUID="83193704-9e17-4504-87f4-d7c8a668f2a2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:29:34.421955 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:34.421940    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8"
May 25 12:29:35.758056 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:35.758016    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-h5dr4" podUID="fb17c480-0ff2-4ef0-9009-8acc07ed73fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
May 25 12:29:36.631622 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:36.631601    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8"
May 25 12:29:36.765195 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:36.765107    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sb9z8\" (UniqueName: \"kubernetes.io/projected/83193704-9e17-4504-87f4-d7c8a668f2a2-kube-api-access-sb9z8\") pod \"83193704-9e17-4504-87f4-d7c8a668f2a2\" (UID: \"83193704-9e17-4504-87f4-d7c8a668f2a2\") "
May 25 12:29:36.765195 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:36.765156    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83193704-9e17-4504-87f4-d7c8a668f2a2-kserve-provision-location\") pod \"83193704-9e17-4504-87f4-d7c8a668f2a2\" (UID: \"83193704-9e17-4504-87f4-d7c8a668f2a2\") "
May 25 12:29:36.765195 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:36.765195    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-logger-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83193704-9e17-4504-87f4-d7c8a668f2a2-isvc-logger-kube-rbac-proxy-sar-config\") pod \"83193704-9e17-4504-87f4-d7c8a668f2a2\" (UID: \"83193704-9e17-4504-87f4-d7c8a668f2a2\") "
May 25 12:29:36.765703 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:36.765230    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83193704-9e17-4504-87f4-d7c8a668f2a2-proxy-tls\") pod \"83193704-9e17-4504-87f4-d7c8a668f2a2\" (UID: \"83193704-9e17-4504-87f4-d7c8a668f2a2\") "
May 25 12:29:36.765703 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:36.765530    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83193704-9e17-4504-87f4-d7c8a668f2a2-isvc-logger-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-logger-kube-rbac-proxy-sar-config") pod "83193704-9e17-4504-87f4-d7c8a668f2a2" (UID: "83193704-9e17-4504-87f4-d7c8a668f2a2"). InnerVolumeSpecName "isvc-logger-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:29:36.765703 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:36.765542    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83193704-9e17-4504-87f4-d7c8a668f2a2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "83193704-9e17-4504-87f4-d7c8a668f2a2" (UID: "83193704-9e17-4504-87f4-d7c8a668f2a2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:29:36.767083 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:36.767061    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83193704-9e17-4504-87f4-d7c8a668f2a2-kube-api-access-sb9z8" (OuterVolumeSpecName: "kube-api-access-sb9z8") pod "83193704-9e17-4504-87f4-d7c8a668f2a2" (UID: "83193704-9e17-4504-87f4-d7c8a668f2a2"). InnerVolumeSpecName "kube-api-access-sb9z8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:29:36.767176 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:36.767098    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83193704-9e17-4504-87f4-d7c8a668f2a2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "83193704-9e17-4504-87f4-d7c8a668f2a2" (UID: "83193704-9e17-4504-87f4-d7c8a668f2a2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:29:36.808078 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:36.808052    2555 generic.go:358] "Generic (PLEG): container finished" podID="83193704-9e17-4504-87f4-d7c8a668f2a2" containerID="13b20bf771b6ba0b8205a998f3247489f01daf02cf0a4b4d189bcc616e22e692" exitCode=0
May 25 12:29:36.808195 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:36.808085    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8" event={"ID":"83193704-9e17-4504-87f4-d7c8a668f2a2","Type":"ContainerDied","Data":"13b20bf771b6ba0b8205a998f3247489f01daf02cf0a4b4d189bcc616e22e692"}
May 25 12:29:36.808195 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:36.808128    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8" event={"ID":"83193704-9e17-4504-87f4-d7c8a668f2a2","Type":"ContainerDied","Data":"01d28f64afeb8491e6c5c8d86ca2dbc9eba891bf855fe946772d19efdf05803a"}
May 25 12:29:36.808195 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:36.808149    2555 scope.go:117] "RemoveContainer" containerID="13b20bf771b6ba0b8205a998f3247489f01daf02cf0a4b4d189bcc616e22e692"
May 25 12:29:36.808195 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:36.808188    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8"
May 25 12:29:36.816943 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:36.816761    2555 scope.go:117] "RemoveContainer" containerID="58e75f46ae921fb83d5f44436a7c7d8bec7bfbd3d77d792ec93f06fd830599c0"
May 25 12:29:36.823551 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:36.823535    2555 scope.go:117] "RemoveContainer" containerID="cfc87e1f1b1ffb6318abe200c134efdecf2eab856dc211d9c3e23053e4eb02a7"
May 25 12:29:36.830122 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:36.830100    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8"]
May 25 12:29:36.830425 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:36.830403    2555 scope.go:117] "RemoveContainer" containerID="5f63ec91ab6a49b4a9793eac84bf4bfba7f452cea87be402a09fc797a3123160"
May 25 12:29:36.832837 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:36.832817    2555 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-gbxb8"]
May 25 12:29:36.837449 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:36.837434    2555 scope.go:117] "RemoveContainer" containerID="13b20bf771b6ba0b8205a998f3247489f01daf02cf0a4b4d189bcc616e22e692"
May 25 12:29:36.837723 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:29:36.837704    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13b20bf771b6ba0b8205a998f3247489f01daf02cf0a4b4d189bcc616e22e692\": container with ID starting with 13b20bf771b6ba0b8205a998f3247489f01daf02cf0a4b4d189bcc616e22e692 not found: ID does not exist" containerID="13b20bf771b6ba0b8205a998f3247489f01daf02cf0a4b4d189bcc616e22e692"
May 25 12:29:36.837793 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:36.837728    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13b20bf771b6ba0b8205a998f3247489f01daf02cf0a4b4d189bcc616e22e692"} err="failed to get container status \"13b20bf771b6ba0b8205a998f3247489f01daf02cf0a4b4d189bcc616e22e692\": rpc error: code = NotFound desc = could not find container \"13b20bf771b6ba0b8205a998f3247489f01daf02cf0a4b4d189bcc616e22e692\": container with ID starting with 13b20bf771b6ba0b8205a998f3247489f01daf02cf0a4b4d189bcc616e22e692 not found: ID does not exist"
May 25 12:29:36.837793 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:36.837743    2555 scope.go:117] "RemoveContainer" containerID="58e75f46ae921fb83d5f44436a7c7d8bec7bfbd3d77d792ec93f06fd830599c0"
May 25 12:29:36.837955 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:29:36.837934    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58e75f46ae921fb83d5f44436a7c7d8bec7bfbd3d77d792ec93f06fd830599c0\": container with ID starting with 58e75f46ae921fb83d5f44436a7c7d8bec7bfbd3d77d792ec93f06fd830599c0 not found: ID does not exist" containerID="58e75f46ae921fb83d5f44436a7c7d8bec7bfbd3d77d792ec93f06fd830599c0"
May 25 12:29:36.837995 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:36.837962    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58e75f46ae921fb83d5f44436a7c7d8bec7bfbd3d77d792ec93f06fd830599c0"} err="failed to get container status \"58e75f46ae921fb83d5f44436a7c7d8bec7bfbd3d77d792ec93f06fd830599c0\": rpc error: code = NotFound desc = could not find container \"58e75f46ae921fb83d5f44436a7c7d8bec7bfbd3d77d792ec93f06fd830599c0\": container with ID starting with 58e75f46ae921fb83d5f44436a7c7d8bec7bfbd3d77d792ec93f06fd830599c0 not found: ID does not exist"
May 25 12:29:36.837995 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:36.837978    2555 scope.go:117] "RemoveContainer" containerID="cfc87e1f1b1ffb6318abe200c134efdecf2eab856dc211d9c3e23053e4eb02a7"
May 25 12:29:36.838205 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:29:36.838185    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfc87e1f1b1ffb6318abe200c134efdecf2eab856dc211d9c3e23053e4eb02a7\": container with ID starting with cfc87e1f1b1ffb6318abe200c134efdecf2eab856dc211d9c3e23053e4eb02a7 not found: ID does not exist" containerID="cfc87e1f1b1ffb6318abe200c134efdecf2eab856dc211d9c3e23053e4eb02a7"
May 25 12:29:36.838249 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:36.838210    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfc87e1f1b1ffb6318abe200c134efdecf2eab856dc211d9c3e23053e4eb02a7"} err="failed to get container status \"cfc87e1f1b1ffb6318abe200c134efdecf2eab856dc211d9c3e23053e4eb02a7\": rpc error: code = NotFound desc = could not find container \"cfc87e1f1b1ffb6318abe200c134efdecf2eab856dc211d9c3e23053e4eb02a7\": container with ID starting with cfc87e1f1b1ffb6318abe200c134efdecf2eab856dc211d9c3e23053e4eb02a7 not found: ID does not exist"
May 25 12:29:36.838249 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:36.838224    2555 scope.go:117] "RemoveContainer" containerID="5f63ec91ab6a49b4a9793eac84bf4bfba7f452cea87be402a09fc797a3123160"
May 25 12:29:36.838403 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:29:36.838389    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f63ec91ab6a49b4a9793eac84bf4bfba7f452cea87be402a09fc797a3123160\": container with ID starting with 5f63ec91ab6a49b4a9793eac84bf4bfba7f452cea87be402a09fc797a3123160 not found: ID does not exist" containerID="5f63ec91ab6a49b4a9793eac84bf4bfba7f452cea87be402a09fc797a3123160"
May 25 12:29:36.838439 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:36.838413    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f63ec91ab6a49b4a9793eac84bf4bfba7f452cea87be402a09fc797a3123160"} err="failed to get container status \"5f63ec91ab6a49b4a9793eac84bf4bfba7f452cea87be402a09fc797a3123160\": rpc error: code = NotFound desc = could not find container \"5f63ec91ab6a49b4a9793eac84bf4bfba7f452cea87be402a09fc797a3123160\": container with ID starting with 5f63ec91ab6a49b4a9793eac84bf4bfba7f452cea87be402a09fc797a3123160 not found: ID does not exist"
May 25 12:29:36.865866 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:36.865844    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83193704-9e17-4504-87f4-d7c8a668f2a2-kserve-provision-location\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:29:36.865866 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:36.865865    2555 reconciler_common.go:299] "Volume detached for volume \"isvc-logger-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83193704-9e17-4504-87f4-d7c8a668f2a2-isvc-logger-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:29:36.865970 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:36.865876    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83193704-9e17-4504-87f4-d7c8a668f2a2-proxy-tls\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:29:36.865970 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:36.865885    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb9z8\" (UniqueName: \"kubernetes.io/projected/83193704-9e17-4504-87f4-d7c8a668f2a2-kube-api-access-sb9z8\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:29:37.598284 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:37.598255    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83193704-9e17-4504-87f4-d7c8a668f2a2" path="/var/lib/kubelet/pods/83193704-9e17-4504-87f4-d7c8a668f2a2/volumes"
May 25 12:29:45.757568 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:45.757523    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-h5dr4" podUID="fb17c480-0ff2-4ef0-9009-8acc07ed73fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
May 25 12:29:55.757549 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:29:55.757500    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-h5dr4" podUID="fb17c480-0ff2-4ef0-9009-8acc07ed73fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
May 25 12:30:05.757616 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:05.757574    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-h5dr4" podUID="fb17c480-0ff2-4ef0-9009-8acc07ed73fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
May 25 12:30:15.757761 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:15.757724    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-h5dr4" podUID="fb17c480-0ff2-4ef0-9009-8acc07ed73fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
May 25 12:30:25.758246 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:25.758203    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-h5dr4" podUID="fb17c480-0ff2-4ef0-9009-8acc07ed73fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
May 25 12:30:26.594869 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:26.594825    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-h5dr4" podUID="fb17c480-0ff2-4ef0-9009-8acc07ed73fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
May 25 12:30:36.595880 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:36.595846    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-h5dr4"
May 25 12:30:46.684825 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:46.684788    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-h5dr4"]
May 25 12:30:46.685399 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:46.685122    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-h5dr4" podUID="fb17c480-0ff2-4ef0-9009-8acc07ed73fb" containerName="kserve-container" containerID="cri-o://f84258eb850d6f28ef2b389fe39ea666a6d7d91ac8573682516b0d12f4835b90" gracePeriod=30
May 25 12:30:46.685399 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:46.685175    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-h5dr4" podUID="fb17c480-0ff2-4ef0-9009-8acc07ed73fb" containerName="kube-rbac-proxy" containerID="cri-o://2a09a57708a5eb05d4a8e8043b1455b91f4a1adaf54eb5f5f950b195fc776ff2" gracePeriod=30
May 25 12:30:46.808433 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:46.808401    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d"]
May 25 12:30:46.808801 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:46.808787    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83193704-9e17-4504-87f4-d7c8a668f2a2" containerName="kserve-container"
May 25 12:30:46.808856 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:46.808803    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="83193704-9e17-4504-87f4-d7c8a668f2a2" containerName="kserve-container"
May 25 12:30:46.808856 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:46.808815    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83193704-9e17-4504-87f4-d7c8a668f2a2" containerName="agent"
May 25 12:30:46.808856 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:46.808820    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="83193704-9e17-4504-87f4-d7c8a668f2a2" containerName="agent"
May 25 12:30:46.808856 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:46.808830    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83193704-9e17-4504-87f4-d7c8a668f2a2" containerName="kube-rbac-proxy"
May 25 12:30:46.808856 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:46.808835    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="83193704-9e17-4504-87f4-d7c8a668f2a2" containerName="kube-rbac-proxy"
May 25 12:30:46.808856 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:46.808846    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83193704-9e17-4504-87f4-d7c8a668f2a2" containerName="storage-initializer"
May 25 12:30:46.808856 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:46.808851    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="83193704-9e17-4504-87f4-d7c8a668f2a2" containerName="storage-initializer"
May 25 12:30:46.809075 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:46.808870    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c80fbc59-fd33-487b-94f2-24ec039041b6" containerName="kserve-container"
May 25 12:30:46.809075 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:46.808875    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c80fbc59-fd33-487b-94f2-24ec039041b6" containerName="kserve-container"
May 25 12:30:46.809075 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:46.808880    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c80fbc59-fd33-487b-94f2-24ec039041b6" containerName="kube-rbac-proxy"
May 25 12:30:46.809075 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:46.808885    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c80fbc59-fd33-487b-94f2-24ec039041b6" containerName="kube-rbac-proxy"
May 25 12:30:46.809075 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:46.808936    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="c80fbc59-fd33-487b-94f2-24ec039041b6" containerName="kserve-container"
May 25 12:30:46.809075 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:46.808943    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="c80fbc59-fd33-487b-94f2-24ec039041b6" containerName="kube-rbac-proxy"
May 25 12:30:46.809075 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:46.808949    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="83193704-9e17-4504-87f4-d7c8a668f2a2" containerName="kserve-container"
May 25 12:30:46.809075 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:46.808956    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="83193704-9e17-4504-87f4-d7c8a668f2a2" containerName="agent"
May 25 12:30:46.809075 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:46.808963    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="83193704-9e17-4504-87f4-d7c8a668f2a2" containerName="kube-rbac-proxy"
May 25 12:30:46.811752 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:46.811735    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d"
May 25 12:30:46.813541 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:46.813514    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
May 25 12:30:46.813680 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:46.813563    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
May 25 12:30:46.819758 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:46.819736    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d"]
May 25 12:30:46.938750 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:46.938663    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sjq8\" (UniqueName: \"kubernetes.io/projected/bc7fefdf-50ea-4e3d-b247-2845ede7858f-kube-api-access-4sjq8\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d\" (UID: \"bc7fefdf-50ea-4e3d-b247-2845ede7858f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d"
May 25 12:30:46.938750 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:46.938717    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc7fefdf-50ea-4e3d-b247-2845ede7858f-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d\" (UID: \"bc7fefdf-50ea-4e3d-b247-2845ede7858f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d"
May 25 12:30:46.938750 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:46.938744    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc7fefdf-50ea-4e3d-b247-2845ede7858f-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d\" (UID: \"bc7fefdf-50ea-4e3d-b247-2845ede7858f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d"
May 25 12:30:46.938971 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:46.938776    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc7fefdf-50ea-4e3d-b247-2845ede7858f-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d\" (UID: \"bc7fefdf-50ea-4e3d-b247-2845ede7858f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d"
May 25 12:30:47.039329 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:47.039286    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc7fefdf-50ea-4e3d-b247-2845ede7858f-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d\" (UID: \"bc7fefdf-50ea-4e3d-b247-2845ede7858f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d"
May 25 12:30:47.039531 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:47.039385    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4sjq8\" (UniqueName: \"kubernetes.io/projected/bc7fefdf-50ea-4e3d-b247-2845ede7858f-kube-api-access-4sjq8\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d\" (UID: \"bc7fefdf-50ea-4e3d-b247-2845ede7858f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d"
May 25 12:30:47.039531 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:47.039427    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc7fefdf-50ea-4e3d-b247-2845ede7858f-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d\" (UID: \"bc7fefdf-50ea-4e3d-b247-2845ede7858f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d"
May 25 12:30:47.039531 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:30:47.039440    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-serving-cert: secret "isvc-lightgbm-runtime-predictor-serving-cert" not found
May 25 12:30:47.039531 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:47.039460    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc7fefdf-50ea-4e3d-b247-2845ede7858f-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d\" (UID: \"bc7fefdf-50ea-4e3d-b247-2845ede7858f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d"
May 25 12:30:47.039531 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:30:47.039509    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc7fefdf-50ea-4e3d-b247-2845ede7858f-proxy-tls podName:bc7fefdf-50ea-4e3d-b247-2845ede7858f nodeName:}" failed. No retries permitted until 2026-05-25 12:30:47.539491958 +0000 UTC m=+1060.499542342 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bc7fefdf-50ea-4e3d-b247-2845ede7858f-proxy-tls") pod "isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d" (UID: "bc7fefdf-50ea-4e3d-b247-2845ede7858f") : secret "isvc-lightgbm-runtime-predictor-serving-cert" not found
May 25 12:30:47.039935 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:47.039858    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc7fefdf-50ea-4e3d-b247-2845ede7858f-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d\" (UID: \"bc7fefdf-50ea-4e3d-b247-2845ede7858f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d"
May 25 12:30:47.040126 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:47.040108    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc7fefdf-50ea-4e3d-b247-2845ede7858f-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d\" (UID: \"bc7fefdf-50ea-4e3d-b247-2845ede7858f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d"
May 25 12:30:47.047564 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:47.047541    2555 generic.go:358] "Generic (PLEG): container finished" podID="fb17c480-0ff2-4ef0-9009-8acc07ed73fb" containerID="2a09a57708a5eb05d4a8e8043b1455b91f4a1adaf54eb5f5f950b195fc776ff2" exitCode=2
May 25 12:30:47.047670 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:47.047589    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-h5dr4" event={"ID":"fb17c480-0ff2-4ef0-9009-8acc07ed73fb","Type":"ContainerDied","Data":"2a09a57708a5eb05d4a8e8043b1455b91f4a1adaf54eb5f5f950b195fc776ff2"}
May 25 12:30:47.049985 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:47.049968    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sjq8\" (UniqueName: \"kubernetes.io/projected/bc7fefdf-50ea-4e3d-b247-2845ede7858f-kube-api-access-4sjq8\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d\" (UID: \"bc7fefdf-50ea-4e3d-b247-2845ede7858f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d"
May 25 12:30:47.545296 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:47.545258    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc7fefdf-50ea-4e3d-b247-2845ede7858f-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d\" (UID: \"bc7fefdf-50ea-4e3d-b247-2845ede7858f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d"
May 25 12:30:47.547685 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:47.547657    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc7fefdf-50ea-4e3d-b247-2845ede7858f-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d\" (UID: \"bc7fefdf-50ea-4e3d-b247-2845ede7858f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d"
May 25 12:30:47.723274 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:47.723241    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d"
May 25 12:30:47.853012 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:47.852975    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d"]
May 25 12:30:47.855564 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:30:47.855529    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc7fefdf_50ea_4e3d_b247_2845ede7858f.slice/crio-ca0c3529bb7a1b1a0b5ba67c3c8eae7efeac1c00a119e8b6bb6acfd91ad29f2e WatchSource:0}: Error finding container ca0c3529bb7a1b1a0b5ba67c3c8eae7efeac1c00a119e8b6bb6acfd91ad29f2e: Status 404 returned error can't find the container with id ca0c3529bb7a1b1a0b5ba67c3c8eae7efeac1c00a119e8b6bb6acfd91ad29f2e
May 25 12:30:48.052249 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:48.052211    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d" event={"ID":"bc7fefdf-50ea-4e3d-b247-2845ede7858f","Type":"ContainerStarted","Data":"9d36cbb473bf377a789987c48fce813e68d69d412602286774615c0c67be1675"}
May 25 12:30:48.052249 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:48.052251    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d" event={"ID":"bc7fefdf-50ea-4e3d-b247-2845ede7858f","Type":"ContainerStarted","Data":"ca0c3529bb7a1b1a0b5ba67c3c8eae7efeac1c00a119e8b6bb6acfd91ad29f2e"}
May 25 12:30:50.753518 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:50.753473    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-h5dr4" podUID="fb17c480-0ff2-4ef0-9009-8acc07ed73fb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
May 25 12:30:51.635273 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:51.635247    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-h5dr4"
May 25 12:30:51.680523 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:51.680488    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb17c480-0ff2-4ef0-9009-8acc07ed73fb-kserve-provision-location\") pod \"fb17c480-0ff2-4ef0-9009-8acc07ed73fb\" (UID: \"fb17c480-0ff2-4ef0-9009-8acc07ed73fb\") "
May 25 12:30:51.680721 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:51.680554    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb17c480-0ff2-4ef0-9009-8acc07ed73fb-isvc-lightgbm-kube-rbac-proxy-sar-config\") pod \"fb17c480-0ff2-4ef0-9009-8acc07ed73fb\" (UID: \"fb17c480-0ff2-4ef0-9009-8acc07ed73fb\") "
May 25 12:30:51.680721 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:51.680595    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c52n9\" (UniqueName: \"kubernetes.io/projected/fb17c480-0ff2-4ef0-9009-8acc07ed73fb-kube-api-access-c52n9\") pod \"fb17c480-0ff2-4ef0-9009-8acc07ed73fb\" (UID: \"fb17c480-0ff2-4ef0-9009-8acc07ed73fb\") "
May 25 12:30:51.680721 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:51.680617    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb17c480-0ff2-4ef0-9009-8acc07ed73fb-proxy-tls\") pod \"fb17c480-0ff2-4ef0-9009-8acc07ed73fb\" (UID: \"fb17c480-0ff2-4ef0-9009-8acc07ed73fb\") "
May 25 12:30:51.680889 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:51.680835    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb17c480-0ff2-4ef0-9009-8acc07ed73fb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fb17c480-0ff2-4ef0-9009-8acc07ed73fb" (UID: "fb17c480-0ff2-4ef0-9009-8acc07ed73fb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:30:51.680940 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:51.680924    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb17c480-0ff2-4ef0-9009-8acc07ed73fb-kserve-provision-location\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:30:51.681018 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:51.680979    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb17c480-0ff2-4ef0-9009-8acc07ed73fb-isvc-lightgbm-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-lightgbm-kube-rbac-proxy-sar-config") pod "fb17c480-0ff2-4ef0-9009-8acc07ed73fb" (UID: "fb17c480-0ff2-4ef0-9009-8acc07ed73fb"). InnerVolumeSpecName "isvc-lightgbm-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:30:51.682757 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:51.682732    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb17c480-0ff2-4ef0-9009-8acc07ed73fb-kube-api-access-c52n9" (OuterVolumeSpecName: "kube-api-access-c52n9") pod "fb17c480-0ff2-4ef0-9009-8acc07ed73fb" (UID: "fb17c480-0ff2-4ef0-9009-8acc07ed73fb"). InnerVolumeSpecName "kube-api-access-c52n9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:30:51.682869 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:51.682846    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb17c480-0ff2-4ef0-9009-8acc07ed73fb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fb17c480-0ff2-4ef0-9009-8acc07ed73fb" (UID: "fb17c480-0ff2-4ef0-9009-8acc07ed73fb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:30:51.781619 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:51.781536    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c52n9\" (UniqueName: \"kubernetes.io/projected/fb17c480-0ff2-4ef0-9009-8acc07ed73fb-kube-api-access-c52n9\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:30:51.781619 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:51.781566    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb17c480-0ff2-4ef0-9009-8acc07ed73fb-proxy-tls\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:30:51.781619 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:51.781577    2555 reconciler_common.go:299] "Volume detached for volume \"isvc-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb17c480-0ff2-4ef0-9009-8acc07ed73fb-isvc-lightgbm-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:30:52.070624 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:52.070538    2555 generic.go:358] "Generic (PLEG): container finished" podID="fb17c480-0ff2-4ef0-9009-8acc07ed73fb" containerID="f84258eb850d6f28ef2b389fe39ea666a6d7d91ac8573682516b0d12f4835b90" exitCode=0
May 25 12:30:52.070800 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:52.070623    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-h5dr4"
May 25 12:30:52.070800 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:52.070621    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-h5dr4" event={"ID":"fb17c480-0ff2-4ef0-9009-8acc07ed73fb","Type":"ContainerDied","Data":"f84258eb850d6f28ef2b389fe39ea666a6d7d91ac8573682516b0d12f4835b90"}
May 25 12:30:52.070800 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:52.070680    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-h5dr4" event={"ID":"fb17c480-0ff2-4ef0-9009-8acc07ed73fb","Type":"ContainerDied","Data":"068a1ba39fab6f98d7b24a04932f0bcf54920aff6597d8b6fa06e35cda9e1654"}
May 25 12:30:52.070800 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:52.070703    2555 scope.go:117] "RemoveContainer" containerID="2a09a57708a5eb05d4a8e8043b1455b91f4a1adaf54eb5f5f950b195fc776ff2"
May 25 12:30:52.072025 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:52.072001    2555 generic.go:358] "Generic (PLEG): container finished" podID="bc7fefdf-50ea-4e3d-b247-2845ede7858f" containerID="9d36cbb473bf377a789987c48fce813e68d69d412602286774615c0c67be1675" exitCode=0
May 25 12:30:52.072092 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:52.072060    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d" event={"ID":"bc7fefdf-50ea-4e3d-b247-2845ede7858f","Type":"ContainerDied","Data":"9d36cbb473bf377a789987c48fce813e68d69d412602286774615c0c67be1675"}
May 25 12:30:52.079400 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:52.079373    2555 scope.go:117] "RemoveContainer" containerID="f84258eb850d6f28ef2b389fe39ea666a6d7d91ac8573682516b0d12f4835b90"
May 25 12:30:52.087273 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:52.087255    2555 scope.go:117] "RemoveContainer" containerID="44d4159ec92ce87f2614eac72624fccc43de493b8917a022c04ba94bd58cbb7e"
May 25 12:30:52.094469 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:52.094449    2555 scope.go:117] "RemoveContainer" containerID="2a09a57708a5eb05d4a8e8043b1455b91f4a1adaf54eb5f5f950b195fc776ff2"
May 25 12:30:52.094833 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:30:52.094806    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a09a57708a5eb05d4a8e8043b1455b91f4a1adaf54eb5f5f950b195fc776ff2\": container with ID starting with 2a09a57708a5eb05d4a8e8043b1455b91f4a1adaf54eb5f5f950b195fc776ff2 not found: ID does not exist" containerID="2a09a57708a5eb05d4a8e8043b1455b91f4a1adaf54eb5f5f950b195fc776ff2"
May 25 12:30:52.094833 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:52.094842    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a09a57708a5eb05d4a8e8043b1455b91f4a1adaf54eb5f5f950b195fc776ff2"} err="failed to get container status \"2a09a57708a5eb05d4a8e8043b1455b91f4a1adaf54eb5f5f950b195fc776ff2\": rpc error: code = NotFound desc = could not find container \"2a09a57708a5eb05d4a8e8043b1455b91f4a1adaf54eb5f5f950b195fc776ff2\": container with ID starting with 2a09a57708a5eb05d4a8e8043b1455b91f4a1adaf54eb5f5f950b195fc776ff2 not found: ID does not exist"
May 25 12:30:52.094833 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:52.094866    2555 scope.go:117] "RemoveContainer" containerID="f84258eb850d6f28ef2b389fe39ea666a6d7d91ac8573682516b0d12f4835b90"
May 25 12:30:52.095387 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:30:52.095359    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f84258eb850d6f28ef2b389fe39ea666a6d7d91ac8573682516b0d12f4835b90\": container with ID starting with f84258eb850d6f28ef2b389fe39ea666a6d7d91ac8573682516b0d12f4835b90 not found: ID does not exist" containerID="f84258eb850d6f28ef2b389fe39ea666a6d7d91ac8573682516b0d12f4835b90"
May 25 12:30:52.095463 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:52.095395    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f84258eb850d6f28ef2b389fe39ea666a6d7d91ac8573682516b0d12f4835b90"} err="failed to get container status \"f84258eb850d6f28ef2b389fe39ea666a6d7d91ac8573682516b0d12f4835b90\": rpc error: code = NotFound desc = could not find container \"f84258eb850d6f28ef2b389fe39ea666a6d7d91ac8573682516b0d12f4835b90\": container with ID starting with f84258eb850d6f28ef2b389fe39ea666a6d7d91ac8573682516b0d12f4835b90 not found: ID does not exist"
May 25 12:30:52.095463 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:52.095417    2555 scope.go:117] "RemoveContainer" containerID="44d4159ec92ce87f2614eac72624fccc43de493b8917a022c04ba94bd58cbb7e"
May 25 12:30:52.095730 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:30:52.095707    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44d4159ec92ce87f2614eac72624fccc43de493b8917a022c04ba94bd58cbb7e\": container with ID starting with 44d4159ec92ce87f2614eac72624fccc43de493b8917a022c04ba94bd58cbb7e not found: ID does not exist" containerID="44d4159ec92ce87f2614eac72624fccc43de493b8917a022c04ba94bd58cbb7e"
May 25 12:30:52.095825 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:52.095737    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44d4159ec92ce87f2614eac72624fccc43de493b8917a022c04ba94bd58cbb7e"} err="failed to get container status \"44d4159ec92ce87f2614eac72624fccc43de493b8917a022c04ba94bd58cbb7e\": rpc error: code = NotFound desc = could not find container \"44d4159ec92ce87f2614eac72624fccc43de493b8917a022c04ba94bd58cbb7e\": container with ID starting with 44d4159ec92ce87f2614eac72624fccc43de493b8917a022c04ba94bd58cbb7e not found: ID does not exist"
May 25 12:30:52.107786 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:52.107761    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-h5dr4"]
May 25 12:30:52.112394 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:52.112365    2555 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-h5dr4"]
May 25 12:30:53.078795 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:53.078761    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d" event={"ID":"bc7fefdf-50ea-4e3d-b247-2845ede7858f","Type":"ContainerStarted","Data":"123bb98f0e977c5828fcd2ed6776f45a907e6bcd33163c9e32e193647df7967c"}
May 25 12:30:53.078795 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:53.078798    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d" event={"ID":"bc7fefdf-50ea-4e3d-b247-2845ede7858f","Type":"ContainerStarted","Data":"02e6c8bc388a369d19a02191799c463dc2fc288ce2908a85a1818e776877e5fb"}
May 25 12:30:53.079270 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:53.079071    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d"
May 25 12:30:53.079270 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:53.079196    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d"
May 25 12:30:53.080593 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:53.080565    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d" podUID="bc7fefdf-50ea-4e3d-b247-2845ede7858f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
May 25 12:30:53.096351 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:53.096312    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d" podStartSLOduration=7.096300539 podStartE2EDuration="7.096300539s" podCreationTimestamp="2026-05-25 12:30:46 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:30:53.094379509 +0000 UTC m=+1066.054429915" watchObservedRunningTime="2026-05-25 12:30:53.096300539 +0000 UTC m=+1066.056350944"
May 25 12:30:53.599480 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:53.599450    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb17c480-0ff2-4ef0-9009-8acc07ed73fb" path="/var/lib/kubelet/pods/fb17c480-0ff2-4ef0-9009-8acc07ed73fb/volumes"
May 25 12:30:54.082089 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:54.082050    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d" podUID="bc7fefdf-50ea-4e3d-b247-2845ede7858f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
May 25 12:30:59.087090 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:59.087006    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d"
May 25 12:30:59.087619 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:30:59.087592    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d" podUID="bc7fefdf-50ea-4e3d-b247-2845ede7858f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
May 25 12:31:09.088137 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:31:09.088099    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d" podUID="bc7fefdf-50ea-4e3d-b247-2845ede7858f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
May 25 12:31:19.087824 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:31:19.087781    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d" podUID="bc7fefdf-50ea-4e3d-b247-2845ede7858f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
May 25 12:31:29.087489 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:31:29.087449    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d" podUID="bc7fefdf-50ea-4e3d-b247-2845ede7858f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
May 25 12:31:39.088029 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:31:39.087987    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d" podUID="bc7fefdf-50ea-4e3d-b247-2845ede7858f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
May 25 12:31:49.088264 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:31:49.088189    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d" podUID="bc7fefdf-50ea-4e3d-b247-2845ede7858f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
May 25 12:31:59.087708 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:31:59.087667    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d" podUID="bc7fefdf-50ea-4e3d-b247-2845ede7858f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
May 25 12:32:09.088331 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:09.088301    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d"
May 25 12:32:17.236935 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:17.236901    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d"]
May 25 12:32:17.237476 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:17.237240    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d" podUID="bc7fefdf-50ea-4e3d-b247-2845ede7858f" containerName="kserve-container" containerID="cri-o://02e6c8bc388a369d19a02191799c463dc2fc288ce2908a85a1818e776877e5fb" gracePeriod=30
May 25 12:32:17.237476 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:17.237271    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d" podUID="bc7fefdf-50ea-4e3d-b247-2845ede7858f" containerName="kube-rbac-proxy" containerID="cri-o://123bb98f0e977c5828fcd2ed6776f45a907e6bcd33163c9e32e193647df7967c" gracePeriod=30
May 25 12:32:17.352667 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:17.352609    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-csb95"]
May 25 12:32:17.353087 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:17.353049    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb17c480-0ff2-4ef0-9009-8acc07ed73fb" containerName="kube-rbac-proxy"
May 25 12:32:17.353087 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:17.353071    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb17c480-0ff2-4ef0-9009-8acc07ed73fb" containerName="kube-rbac-proxy"
May 25 12:32:17.353240 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:17.353093    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb17c480-0ff2-4ef0-9009-8acc07ed73fb" containerName="storage-initializer"
May 25 12:32:17.353240 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:17.353102    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb17c480-0ff2-4ef0-9009-8acc07ed73fb" containerName="storage-initializer"
May 25 12:32:17.353240 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:17.353126    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb17c480-0ff2-4ef0-9009-8acc07ed73fb" containerName="kserve-container"
May 25 12:32:17.353240 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:17.353134    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb17c480-0ff2-4ef0-9009-8acc07ed73fb" containerName="kserve-container"
May 25 12:32:17.353240 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:17.353237    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="fb17c480-0ff2-4ef0-9009-8acc07ed73fb" containerName="kube-rbac-proxy"
May 25 12:32:17.353489 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:17.353255    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="fb17c480-0ff2-4ef0-9009-8acc07ed73fb" containerName="kserve-container"
May 25 12:32:17.356477 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:17.356440    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-csb95"
May 25 12:32:17.359226 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:17.359200    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
May 25 12:32:17.359335 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:17.359249    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
May 25 12:32:17.361414 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:17.361389    2555 generic.go:358] "Generic (PLEG): container finished" podID="bc7fefdf-50ea-4e3d-b247-2845ede7858f" containerID="123bb98f0e977c5828fcd2ed6776f45a907e6bcd33163c9e32e193647df7967c" exitCode=2
May 25 12:32:17.361516 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:17.361456    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d" event={"ID":"bc7fefdf-50ea-4e3d-b247-2845ede7858f","Type":"ContainerDied","Data":"123bb98f0e977c5828fcd2ed6776f45a907e6bcd33163c9e32e193647df7967c"}
May 25 12:32:17.374626 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:17.374605    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-csb95"]
May 25 12:32:17.482347 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:17.482316    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a54d9e9c-68d3-498b-ab6c-89259961d753-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-csb95\" (UID: \"a54d9e9c-68d3-498b-ab6c-89259961d753\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-csb95"
May 25 12:32:17.482585 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:17.482416    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a54d9e9c-68d3-498b-ab6c-89259961d753-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-csb95\" (UID: \"a54d9e9c-68d3-498b-ab6c-89259961d753\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-csb95"
May 25 12:32:17.482585 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:17.482456    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq2fk\" (UniqueName: \"kubernetes.io/projected/a54d9e9c-68d3-498b-ab6c-89259961d753-kube-api-access-vq2fk\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-csb95\" (UID: \"a54d9e9c-68d3-498b-ab6c-89259961d753\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-csb95"
May 25 12:32:17.482585 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:17.482501    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a54d9e9c-68d3-498b-ab6c-89259961d753-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-csb95\" (UID: \"a54d9e9c-68d3-498b-ab6c-89259961d753\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-csb95"
May 25 12:32:17.583459 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:17.583385    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a54d9e9c-68d3-498b-ab6c-89259961d753-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-csb95\" (UID: \"a54d9e9c-68d3-498b-ab6c-89259961d753\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-csb95"
May 25 12:32:17.583459 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:17.583420    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vq2fk\" (UniqueName: \"kubernetes.io/projected/a54d9e9c-68d3-498b-ab6c-89259961d753-kube-api-access-vq2fk\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-csb95\" (UID: \"a54d9e9c-68d3-498b-ab6c-89259961d753\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-csb95"
May 25 12:32:17.583667 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:17.583459    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a54d9e9c-68d3-498b-ab6c-89259961d753-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-csb95\" (UID: \"a54d9e9c-68d3-498b-ab6c-89259961d753\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-csb95"
May 25 12:32:17.583667 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:17.583488    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a54d9e9c-68d3-498b-ab6c-89259961d753-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-csb95\" (UID: \"a54d9e9c-68d3-498b-ab6c-89259961d753\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-csb95"
May 25 12:32:17.583826 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:17.583804    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a54d9e9c-68d3-498b-ab6c-89259961d753-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-csb95\" (UID: \"a54d9e9c-68d3-498b-ab6c-89259961d753\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-csb95"
May 25 12:32:17.584138 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:17.584115    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a54d9e9c-68d3-498b-ab6c-89259961d753-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-csb95\" (UID: \"a54d9e9c-68d3-498b-ab6c-89259961d753\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-csb95"
May 25 12:32:17.585937 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:17.585920    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a54d9e9c-68d3-498b-ab6c-89259961d753-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-csb95\" (UID: \"a54d9e9c-68d3-498b-ab6c-89259961d753\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-csb95"
May 25 12:32:17.590793 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:17.590770    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq2fk\" (UniqueName: \"kubernetes.io/projected/a54d9e9c-68d3-498b-ab6c-89259961d753-kube-api-access-vq2fk\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-csb95\" (UID: \"a54d9e9c-68d3-498b-ab6c-89259961d753\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-csb95"
May 25 12:32:17.667658 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:17.667623    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-csb95"
May 25 12:32:17.801208 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:17.801170    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-csb95"]
May 25 12:32:17.804744 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:32:17.804715    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda54d9e9c_68d3_498b_ab6c_89259961d753.slice/crio-da5170dd528dccca999dbeef9e1eadd07555186ee854c824a56e1c324b3d4ba6 WatchSource:0}: Error finding container da5170dd528dccca999dbeef9e1eadd07555186ee854c824a56e1c324b3d4ba6: Status 404 returned error can't find the container with id da5170dd528dccca999dbeef9e1eadd07555186ee854c824a56e1c324b3d4ba6
May 25 12:32:18.366991 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:18.366953    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-csb95" event={"ID":"a54d9e9c-68d3-498b-ab6c-89259961d753","Type":"ContainerStarted","Data":"63c69f26510cb1455d2470e8ad31ae6939fcfcf3f65990b65059102a99db315e"}
May 25 12:32:18.366991 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:18.366996    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-csb95" event={"ID":"a54d9e9c-68d3-498b-ab6c-89259961d753","Type":"ContainerStarted","Data":"da5170dd528dccca999dbeef9e1eadd07555186ee854c824a56e1c324b3d4ba6"}
May 25 12:32:19.082359 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:19.082314    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d" podUID="bc7fefdf-50ea-4e3d-b247-2845ede7858f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
May 25 12:32:19.088263 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:19.088238    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d" podUID="bc7fefdf-50ea-4e3d-b247-2845ede7858f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
May 25 12:32:21.281546 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:21.281524    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d"
May 25 12:32:21.379065 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:21.378984    2555 generic.go:358] "Generic (PLEG): container finished" podID="bc7fefdf-50ea-4e3d-b247-2845ede7858f" containerID="02e6c8bc388a369d19a02191799c463dc2fc288ce2908a85a1818e776877e5fb" exitCode=0
May 25 12:32:21.379214 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:21.379066    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d" event={"ID":"bc7fefdf-50ea-4e3d-b247-2845ede7858f","Type":"ContainerDied","Data":"02e6c8bc388a369d19a02191799c463dc2fc288ce2908a85a1818e776877e5fb"}
May 25 12:32:21.379214 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:21.379104    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d" event={"ID":"bc7fefdf-50ea-4e3d-b247-2845ede7858f","Type":"ContainerDied","Data":"ca0c3529bb7a1b1a0b5ba67c3c8eae7efeac1c00a119e8b6bb6acfd91ad29f2e"}
May 25 12:32:21.379214 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:21.379120    2555 scope.go:117] "RemoveContainer" containerID="123bb98f0e977c5828fcd2ed6776f45a907e6bcd33163c9e32e193647df7967c"
May 25 12:32:21.379214 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:21.379070    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d"
May 25 12:32:21.386627 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:21.386611    2555 scope.go:117] "RemoveContainer" containerID="02e6c8bc388a369d19a02191799c463dc2fc288ce2908a85a1818e776877e5fb"
May 25 12:32:21.393422 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:21.393405    2555 scope.go:117] "RemoveContainer" containerID="9d36cbb473bf377a789987c48fce813e68d69d412602286774615c0c67be1675"
May 25 12:32:21.399662 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:21.399614    2555 scope.go:117] "RemoveContainer" containerID="123bb98f0e977c5828fcd2ed6776f45a907e6bcd33163c9e32e193647df7967c"
May 25 12:32:21.399881 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:32:21.399864    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"123bb98f0e977c5828fcd2ed6776f45a907e6bcd33163c9e32e193647df7967c\": container with ID starting with 123bb98f0e977c5828fcd2ed6776f45a907e6bcd33163c9e32e193647df7967c not found: ID does not exist" containerID="123bb98f0e977c5828fcd2ed6776f45a907e6bcd33163c9e32e193647df7967c"
May 25 12:32:21.399930 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:21.399888    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"123bb98f0e977c5828fcd2ed6776f45a907e6bcd33163c9e32e193647df7967c"} err="failed to get container status \"123bb98f0e977c5828fcd2ed6776f45a907e6bcd33163c9e32e193647df7967c\": rpc error: code = NotFound desc = could not find container \"123bb98f0e977c5828fcd2ed6776f45a907e6bcd33163c9e32e193647df7967c\": container with ID starting with 123bb98f0e977c5828fcd2ed6776f45a907e6bcd33163c9e32e193647df7967c not found: ID does not exist"
May 25 12:32:21.399930 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:21.399909    2555 scope.go:117] "RemoveContainer" containerID="02e6c8bc388a369d19a02191799c463dc2fc288ce2908a85a1818e776877e5fb"
May 25 12:32:21.400150 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:32:21.400133    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02e6c8bc388a369d19a02191799c463dc2fc288ce2908a85a1818e776877e5fb\": container with ID starting with 02e6c8bc388a369d19a02191799c463dc2fc288ce2908a85a1818e776877e5fb not found: ID does not exist" containerID="02e6c8bc388a369d19a02191799c463dc2fc288ce2908a85a1818e776877e5fb"
May 25 12:32:21.400199 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:21.400155    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02e6c8bc388a369d19a02191799c463dc2fc288ce2908a85a1818e776877e5fb"} err="failed to get container status \"02e6c8bc388a369d19a02191799c463dc2fc288ce2908a85a1818e776877e5fb\": rpc error: code = NotFound desc = could not find container \"02e6c8bc388a369d19a02191799c463dc2fc288ce2908a85a1818e776877e5fb\": container with ID starting with 02e6c8bc388a369d19a02191799c463dc2fc288ce2908a85a1818e776877e5fb not found: ID does not exist"
May 25 12:32:21.400199 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:21.400170    2555 scope.go:117] "RemoveContainer" containerID="9d36cbb473bf377a789987c48fce813e68d69d412602286774615c0c67be1675"
May 25 12:32:21.400374 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:32:21.400356    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d36cbb473bf377a789987c48fce813e68d69d412602286774615c0c67be1675\": container with ID starting with 9d36cbb473bf377a789987c48fce813e68d69d412602286774615c0c67be1675 not found: ID does not exist" containerID="9d36cbb473bf377a789987c48fce813e68d69d412602286774615c0c67be1675"
May 25 12:32:21.400413 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:21.400380    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d36cbb473bf377a789987c48fce813e68d69d412602286774615c0c67be1675"} err="failed to get container status \"9d36cbb473bf377a789987c48fce813e68d69d412602286774615c0c67be1675\": rpc error: code = NotFound desc = could not find container \"9d36cbb473bf377a789987c48fce813e68d69d412602286774615c0c67be1675\": container with ID starting with 9d36cbb473bf377a789987c48fce813e68d69d412602286774615c0c67be1675 not found: ID does not exist"
May 25 12:32:21.413871 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:21.413852    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc7fefdf-50ea-4e3d-b247-2845ede7858f-kserve-provision-location\") pod \"bc7fefdf-50ea-4e3d-b247-2845ede7858f\" (UID: \"bc7fefdf-50ea-4e3d-b247-2845ede7858f\") "
May 25 12:32:21.413954 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:21.413886    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc7fefdf-50ea-4e3d-b247-2845ede7858f-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\") pod \"bc7fefdf-50ea-4e3d-b247-2845ede7858f\" (UID: \"bc7fefdf-50ea-4e3d-b247-2845ede7858f\") "
May 25 12:32:21.413954 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:21.413939    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc7fefdf-50ea-4e3d-b247-2845ede7858f-proxy-tls\") pod \"bc7fefdf-50ea-4e3d-b247-2845ede7858f\" (UID: \"bc7fefdf-50ea-4e3d-b247-2845ede7858f\") "
May 25 12:32:21.414067 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:21.413996    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4sjq8\" (UniqueName: \"kubernetes.io/projected/bc7fefdf-50ea-4e3d-b247-2845ede7858f-kube-api-access-4sjq8\") pod \"bc7fefdf-50ea-4e3d-b247-2845ede7858f\" (UID: \"bc7fefdf-50ea-4e3d-b247-2845ede7858f\") "
May 25 12:32:21.414232 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:21.414207    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc7fefdf-50ea-4e3d-b247-2845ede7858f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bc7fefdf-50ea-4e3d-b247-2845ede7858f" (UID: "bc7fefdf-50ea-4e3d-b247-2845ede7858f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:32:21.414393 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:21.414350    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc7fefdf-50ea-4e3d-b247-2845ede7858f-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-lightgbm-runtime-kube-rbac-proxy-sar-config") pod "bc7fefdf-50ea-4e3d-b247-2845ede7858f" (UID: "bc7fefdf-50ea-4e3d-b247-2845ede7858f"). InnerVolumeSpecName "isvc-lightgbm-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:32:21.415769 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:21.415745    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc7fefdf-50ea-4e3d-b247-2845ede7858f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bc7fefdf-50ea-4e3d-b247-2845ede7858f" (UID: "bc7fefdf-50ea-4e3d-b247-2845ede7858f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:32:21.415921 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:21.415904    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc7fefdf-50ea-4e3d-b247-2845ede7858f-kube-api-access-4sjq8" (OuterVolumeSpecName: "kube-api-access-4sjq8") pod "bc7fefdf-50ea-4e3d-b247-2845ede7858f" (UID: "bc7fefdf-50ea-4e3d-b247-2845ede7858f"). InnerVolumeSpecName "kube-api-access-4sjq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:32:21.514811 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:21.514783    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc7fefdf-50ea-4e3d-b247-2845ede7858f-proxy-tls\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:32:21.514978 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:21.514817    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sjq8\" (UniqueName: \"kubernetes.io/projected/bc7fefdf-50ea-4e3d-b247-2845ede7858f-kube-api-access-4sjq8\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:32:21.514978 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:21.514832    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc7fefdf-50ea-4e3d-b247-2845ede7858f-kserve-provision-location\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:32:21.514978 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:21.514846    2555 reconciler_common.go:299] "Volume detached for volume \"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc7fefdf-50ea-4e3d-b247-2845ede7858f-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:32:21.693961 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:21.693933    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d"]
May 25 12:32:21.697747 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:21.697720    2555 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-4gj7d"]
May 25 12:32:22.385021 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:22.384917    2555 generic.go:358] "Generic (PLEG): container finished" podID="a54d9e9c-68d3-498b-ab6c-89259961d753" containerID="63c69f26510cb1455d2470e8ad31ae6939fcfcf3f65990b65059102a99db315e" exitCode=0
May 25 12:32:22.385021 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:22.384979    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-csb95" event={"ID":"a54d9e9c-68d3-498b-ab6c-89259961d753","Type":"ContainerDied","Data":"63c69f26510cb1455d2470e8ad31ae6939fcfcf3f65990b65059102a99db315e"}
May 25 12:32:23.601859 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:32:23.601346    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc7fefdf-50ea-4e3d-b247-2845ede7858f" path="/var/lib/kubelet/pods/bc7fefdf-50ea-4e3d-b247-2845ede7858f/volumes"
May 25 12:33:07.553005 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:33:07.552960    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-2tqtm_00e6f79c-8f05-4330-aa16-1ada2ae9e9cc/console-operator/1.log"
May 25 12:33:07.555870 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:33:07.555847    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-2tqtm_00e6f79c-8f05-4330-aa16-1ada2ae9e9cc/console-operator/1.log"
May 25 12:34:31.118178 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:34:31.118152    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 12:34:31.865337 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:34:31.865296    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-csb95" event={"ID":"a54d9e9c-68d3-498b-ab6c-89259961d753","Type":"ContainerStarted","Data":"a0b00a3d15f7e8cb086375f0e4ee17db4d590fbd0895a5d53eb7b37fe58b795e"}
May 25 12:34:31.865520 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:34:31.865344    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-csb95" event={"ID":"a54d9e9c-68d3-498b-ab6c-89259961d753","Type":"ContainerStarted","Data":"541c64247e77d75a213c1fc4a28d347819ba7470c0a0c7e583ff961cfac77c4e"}
May 25 12:34:31.865520 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:34:31.865467    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-csb95"
May 25 12:34:31.865520 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:34:31.865499    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-csb95"
May 25 12:34:31.888261 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:34:31.888201    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-csb95" podStartSLOduration=6.261817213 podStartE2EDuration="2m14.888181669s" podCreationTimestamp="2026-05-25 12:32:17 +0000 UTC" firstStartedPulling="2026-05-25 12:32:22.386125756 +0000 UTC m=+1155.346176140" lastFinishedPulling="2026-05-25 12:34:31.012490212 +0000 UTC m=+1283.972540596" observedRunningTime="2026-05-25 12:34:31.886328116 +0000 UTC m=+1284.846378523" watchObservedRunningTime="2026-05-25 12:34:31.888181669 +0000 UTC m=+1284.848232075"
May 25 12:34:37.873101 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:34:37.873074    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-csb95"
May 25 12:35:07.876786 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:07.876754    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-csb95"
May 25 12:35:17.520472 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:17.520436    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-csb95"]
May 25 12:35:17.521061 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:17.520866    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-csb95" podUID="a54d9e9c-68d3-498b-ab6c-89259961d753" containerName="kserve-container" containerID="cri-o://541c64247e77d75a213c1fc4a28d347819ba7470c0a0c7e583ff961cfac77c4e" gracePeriod=30
May 25 12:35:17.521061 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:17.520911    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-csb95" podUID="a54d9e9c-68d3-498b-ab6c-89259961d753" containerName="kube-rbac-proxy" containerID="cri-o://a0b00a3d15f7e8cb086375f0e4ee17db4d590fbd0895a5d53eb7b37fe58b795e" gracePeriod=30
May 25 12:35:17.615687 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:17.615633    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-9xcdr"]
May 25 12:35:17.616077 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:17.616059    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc7fefdf-50ea-4e3d-b247-2845ede7858f" containerName="storage-initializer"
May 25 12:35:17.616162 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:17.616080    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc7fefdf-50ea-4e3d-b247-2845ede7858f" containerName="storage-initializer"
May 25 12:35:17.616162 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:17.616109    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc7fefdf-50ea-4e3d-b247-2845ede7858f" containerName="kserve-container"
May 25 12:35:17.616162 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:17.616119    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc7fefdf-50ea-4e3d-b247-2845ede7858f" containerName="kserve-container"
May 25 12:35:17.616162 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:17.616129    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc7fefdf-50ea-4e3d-b247-2845ede7858f" containerName="kube-rbac-proxy"
May 25 12:35:17.616162 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:17.616137    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc7fefdf-50ea-4e3d-b247-2845ede7858f" containerName="kube-rbac-proxy"
May 25 12:35:17.616418 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:17.616230    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="bc7fefdf-50ea-4e3d-b247-2845ede7858f" containerName="kserve-container"
May 25 12:35:17.616418 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:17.616245    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="bc7fefdf-50ea-4e3d-b247-2845ede7858f" containerName="kube-rbac-proxy"
May 25 12:35:17.619027 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:17.618993    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-9xcdr"
May 25 12:35:17.621326 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:17.621299    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
May 25 12:35:17.621634 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:17.621610    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
May 25 12:35:17.630325 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:17.630299    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-9xcdr"]
May 25 12:35:17.695426 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:17.695380    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-649p9\" (UniqueName: \"kubernetes.io/projected/5e812af3-80d2-490c-8915-6150569a1fe9-kube-api-access-649p9\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-9xcdr\" (UID: \"5e812af3-80d2-490c-8915-6150569a1fe9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-9xcdr"
May 25 12:35:17.695426 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:17.695424    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e812af3-80d2-490c-8915-6150569a1fe9-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-9xcdr\" (UID: \"5e812af3-80d2-490c-8915-6150569a1fe9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-9xcdr"
May 25 12:35:17.695744 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:17.695447    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e812af3-80d2-490c-8915-6150569a1fe9-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-9xcdr\" (UID: \"5e812af3-80d2-490c-8915-6150569a1fe9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-9xcdr"
May 25 12:35:17.695744 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:17.695486    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e812af3-80d2-490c-8915-6150569a1fe9-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-9xcdr\" (UID: \"5e812af3-80d2-490c-8915-6150569a1fe9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-9xcdr"
May 25 12:35:17.796493 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:17.796415    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-649p9\" (UniqueName: \"kubernetes.io/projected/5e812af3-80d2-490c-8915-6150569a1fe9-kube-api-access-649p9\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-9xcdr\" (UID: \"5e812af3-80d2-490c-8915-6150569a1fe9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-9xcdr"
May 25 12:35:17.796493 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:17.796452    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e812af3-80d2-490c-8915-6150569a1fe9-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-9xcdr\" (UID: \"5e812af3-80d2-490c-8915-6150569a1fe9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-9xcdr"
May 25 12:35:17.796493 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:17.796474    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e812af3-80d2-490c-8915-6150569a1fe9-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-9xcdr\" (UID: \"5e812af3-80d2-490c-8915-6150569a1fe9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-9xcdr"
May 25 12:35:17.796828 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:17.796504    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e812af3-80d2-490c-8915-6150569a1fe9-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-9xcdr\" (UID: \"5e812af3-80d2-490c-8915-6150569a1fe9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-9xcdr"
May 25 12:35:17.796989 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:17.796961    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e812af3-80d2-490c-8915-6150569a1fe9-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-9xcdr\" (UID: \"5e812af3-80d2-490c-8915-6150569a1fe9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-9xcdr"
May 25 12:35:17.797192 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:17.797169    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e812af3-80d2-490c-8915-6150569a1fe9-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-9xcdr\" (UID: \"5e812af3-80d2-490c-8915-6150569a1fe9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-9xcdr"
May 25 12:35:17.799279 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:17.799250    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e812af3-80d2-490c-8915-6150569a1fe9-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-9xcdr\" (UID: \"5e812af3-80d2-490c-8915-6150569a1fe9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-9xcdr"
May 25 12:35:17.803958 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:17.803934    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-649p9\" (UniqueName: \"kubernetes.io/projected/5e812af3-80d2-490c-8915-6150569a1fe9-kube-api-access-649p9\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-9xcdr\" (UID: \"5e812af3-80d2-490c-8915-6150569a1fe9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-9xcdr"
May 25 12:35:17.869067 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:17.869024    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-csb95" podUID="a54d9e9c-68d3-498b-ab6c-89259961d753" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
May 25 12:35:17.938370 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:17.938341    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-9xcdr"
May 25 12:35:18.017030 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:18.016995    2555 generic.go:358] "Generic (PLEG): container finished" podID="a54d9e9c-68d3-498b-ab6c-89259961d753" containerID="a0b00a3d15f7e8cb086375f0e4ee17db4d590fbd0895a5d53eb7b37fe58b795e" exitCode=2
May 25 12:35:18.017170 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:18.017078    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-csb95" event={"ID":"a54d9e9c-68d3-498b-ab6c-89259961d753","Type":"ContainerDied","Data":"a0b00a3d15f7e8cb086375f0e4ee17db4d590fbd0895a5d53eb7b37fe58b795e"}
May 25 12:35:18.067809 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:18.067779    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-9xcdr"]
May 25 12:35:18.070473 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:35:18.070446    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e812af3_80d2_490c_8915_6150569a1fe9.slice/crio-fe075b5cbc25a228b6b611d43edf6f6923c07bd7b01557db973160061de25adc WatchSource:0}: Error finding container fe075b5cbc25a228b6b611d43edf6f6923c07bd7b01557db973160061de25adc: Status 404 returned error can't find the container with id fe075b5cbc25a228b6b611d43edf6f6923c07bd7b01557db973160061de25adc
May 25 12:35:18.568784 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:18.568760    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-csb95"
May 25 12:35:18.601510 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:18.601419    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a54d9e9c-68d3-498b-ab6c-89259961d753-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\") pod \"a54d9e9c-68d3-498b-ab6c-89259961d753\" (UID: \"a54d9e9c-68d3-498b-ab6c-89259961d753\") "
May 25 12:35:18.601510 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:18.601501    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a54d9e9c-68d3-498b-ab6c-89259961d753-kserve-provision-location\") pod \"a54d9e9c-68d3-498b-ab6c-89259961d753\" (UID: \"a54d9e9c-68d3-498b-ab6c-89259961d753\") "
May 25 12:35:18.601772 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:18.601532    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vq2fk\" (UniqueName: \"kubernetes.io/projected/a54d9e9c-68d3-498b-ab6c-89259961d753-kube-api-access-vq2fk\") pod \"a54d9e9c-68d3-498b-ab6c-89259961d753\" (UID: \"a54d9e9c-68d3-498b-ab6c-89259961d753\") "
May 25 12:35:18.601772 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:18.601580    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a54d9e9c-68d3-498b-ab6c-89259961d753-proxy-tls\") pod \"a54d9e9c-68d3-498b-ab6c-89259961d753\" (UID: \"a54d9e9c-68d3-498b-ab6c-89259961d753\") "
May 25 12:35:18.601891 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:18.601787    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a54d9e9c-68d3-498b-ab6c-89259961d753-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config") pod "a54d9e9c-68d3-498b-ab6c-89259961d753" (UID: "a54d9e9c-68d3-498b-ab6c-89259961d753"). InnerVolumeSpecName "isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:35:18.601958 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:18.601898    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a54d9e9c-68d3-498b-ab6c-89259961d753-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a54d9e9c-68d3-498b-ab6c-89259961d753" (UID: "a54d9e9c-68d3-498b-ab6c-89259961d753"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:35:18.603784 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:18.603757    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a54d9e9c-68d3-498b-ab6c-89259961d753-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a54d9e9c-68d3-498b-ab6c-89259961d753" (UID: "a54d9e9c-68d3-498b-ab6c-89259961d753"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:35:18.603915 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:18.603853    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a54d9e9c-68d3-498b-ab6c-89259961d753-kube-api-access-vq2fk" (OuterVolumeSpecName: "kube-api-access-vq2fk") pod "a54d9e9c-68d3-498b-ab6c-89259961d753" (UID: "a54d9e9c-68d3-498b-ab6c-89259961d753"). InnerVolumeSpecName "kube-api-access-vq2fk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:35:18.703018 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:18.702976    2555 reconciler_common.go:299] "Volume detached for volume \"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a54d9e9c-68d3-498b-ab6c-89259961d753-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:35:18.703018 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:18.703015    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a54d9e9c-68d3-498b-ab6c-89259961d753-kserve-provision-location\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:35:18.703018 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:18.703026    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq2fk\" (UniqueName: \"kubernetes.io/projected/a54d9e9c-68d3-498b-ab6c-89259961d753-kube-api-access-vq2fk\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:35:18.703230 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:18.703036    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a54d9e9c-68d3-498b-ab6c-89259961d753-proxy-tls\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:35:19.021857 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:19.021826    2555 generic.go:358] "Generic (PLEG): container finished" podID="a54d9e9c-68d3-498b-ab6c-89259961d753" containerID="541c64247e77d75a213c1fc4a28d347819ba7470c0a0c7e583ff961cfac77c4e" exitCode=0
May 25 12:35:19.022030 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:19.021891    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-csb95" event={"ID":"a54d9e9c-68d3-498b-ab6c-89259961d753","Type":"ContainerDied","Data":"541c64247e77d75a213c1fc4a28d347819ba7470c0a0c7e583ff961cfac77c4e"}
May 25 12:35:19.022030 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:19.021920    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-csb95" event={"ID":"a54d9e9c-68d3-498b-ab6c-89259961d753","Type":"ContainerDied","Data":"da5170dd528dccca999dbeef9e1eadd07555186ee854c824a56e1c324b3d4ba6"}
May 25 12:35:19.022030 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:19.021918    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-csb95"
May 25 12:35:19.022030 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:19.021941    2555 scope.go:117] "RemoveContainer" containerID="a0b00a3d15f7e8cb086375f0e4ee17db4d590fbd0895a5d53eb7b37fe58b795e"
May 25 12:35:19.023410 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:19.023388    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-9xcdr" event={"ID":"5e812af3-80d2-490c-8915-6150569a1fe9","Type":"ContainerStarted","Data":"899e71a228df7a096dbb6735942e5449febc113beb332040cf98e9b646b6712d"}
May 25 12:35:19.023506 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:19.023416    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-9xcdr" event={"ID":"5e812af3-80d2-490c-8915-6150569a1fe9","Type":"ContainerStarted","Data":"fe075b5cbc25a228b6b611d43edf6f6923c07bd7b01557db973160061de25adc"}
May 25 12:35:19.031841 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:19.031821    2555 scope.go:117] "RemoveContainer" containerID="541c64247e77d75a213c1fc4a28d347819ba7470c0a0c7e583ff961cfac77c4e"
May 25 12:35:19.039335 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:19.039305    2555 scope.go:117] "RemoveContainer" containerID="63c69f26510cb1455d2470e8ad31ae6939fcfcf3f65990b65059102a99db315e"
May 25 12:35:19.047046 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:19.047029    2555 scope.go:117] "RemoveContainer" containerID="a0b00a3d15f7e8cb086375f0e4ee17db4d590fbd0895a5d53eb7b37fe58b795e"
May 25 12:35:19.047301 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:35:19.047283    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0b00a3d15f7e8cb086375f0e4ee17db4d590fbd0895a5d53eb7b37fe58b795e\": container with ID starting with a0b00a3d15f7e8cb086375f0e4ee17db4d590fbd0895a5d53eb7b37fe58b795e not found: ID does not exist" containerID="a0b00a3d15f7e8cb086375f0e4ee17db4d590fbd0895a5d53eb7b37fe58b795e"
May 25 12:35:19.047356 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:19.047308    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0b00a3d15f7e8cb086375f0e4ee17db4d590fbd0895a5d53eb7b37fe58b795e"} err="failed to get container status \"a0b00a3d15f7e8cb086375f0e4ee17db4d590fbd0895a5d53eb7b37fe58b795e\": rpc error: code = NotFound desc = could not find container \"a0b00a3d15f7e8cb086375f0e4ee17db4d590fbd0895a5d53eb7b37fe58b795e\": container with ID starting with a0b00a3d15f7e8cb086375f0e4ee17db4d590fbd0895a5d53eb7b37fe58b795e not found: ID does not exist"
May 25 12:35:19.047356 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:19.047325    2555 scope.go:117] "RemoveContainer" containerID="541c64247e77d75a213c1fc4a28d347819ba7470c0a0c7e583ff961cfac77c4e"
May 25 12:35:19.047580 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:35:19.047563    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"541c64247e77d75a213c1fc4a28d347819ba7470c0a0c7e583ff961cfac77c4e\": container with ID starting with 541c64247e77d75a213c1fc4a28d347819ba7470c0a0c7e583ff961cfac77c4e not found: ID does not exist" containerID="541c64247e77d75a213c1fc4a28d347819ba7470c0a0c7e583ff961cfac77c4e"
May 25 12:35:19.047651 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:19.047586    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"541c64247e77d75a213c1fc4a28d347819ba7470c0a0c7e583ff961cfac77c4e"} err="failed to get container status \"541c64247e77d75a213c1fc4a28d347819ba7470c0a0c7e583ff961cfac77c4e\": rpc error: code = NotFound desc = could not find container \"541c64247e77d75a213c1fc4a28d347819ba7470c0a0c7e583ff961cfac77c4e\": container with ID starting with 541c64247e77d75a213c1fc4a28d347819ba7470c0a0c7e583ff961cfac77c4e not found: ID does not exist"
May 25 12:35:19.047748 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:19.047719    2555 scope.go:117] "RemoveContainer" containerID="63c69f26510cb1455d2470e8ad31ae6939fcfcf3f65990b65059102a99db315e"
May 25 12:35:19.047987 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:35:19.047965    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63c69f26510cb1455d2470e8ad31ae6939fcfcf3f65990b65059102a99db315e\": container with ID starting with 63c69f26510cb1455d2470e8ad31ae6939fcfcf3f65990b65059102a99db315e not found: ID does not exist" containerID="63c69f26510cb1455d2470e8ad31ae6939fcfcf3f65990b65059102a99db315e"
May 25 12:35:19.048055 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:19.047997    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63c69f26510cb1455d2470e8ad31ae6939fcfcf3f65990b65059102a99db315e"} err="failed to get container status \"63c69f26510cb1455d2470e8ad31ae6939fcfcf3f65990b65059102a99db315e\": rpc error: code = NotFound desc = could not find container \"63c69f26510cb1455d2470e8ad31ae6939fcfcf3f65990b65059102a99db315e\": container with ID starting with 63c69f26510cb1455d2470e8ad31ae6939fcfcf3f65990b65059102a99db315e not found: ID does not exist"
May 25 12:35:19.054407 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:19.054386    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-csb95"]
May 25 12:35:19.058033 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:19.058012    2555 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-csb95"]
May 25 12:35:19.599267 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:19.599221    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a54d9e9c-68d3-498b-ab6c-89259961d753" path="/var/lib/kubelet/pods/a54d9e9c-68d3-498b-ab6c-89259961d753/volumes"
May 25 12:35:22.034746 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:22.034707    2555 generic.go:358] "Generic (PLEG): container finished" podID="5e812af3-80d2-490c-8915-6150569a1fe9" containerID="899e71a228df7a096dbb6735942e5449febc113beb332040cf98e9b646b6712d" exitCode=0
May 25 12:35:22.035161 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:22.034790    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-9xcdr" event={"ID":"5e812af3-80d2-490c-8915-6150569a1fe9","Type":"ContainerDied","Data":"899e71a228df7a096dbb6735942e5449febc113beb332040cf98e9b646b6712d"}
May 25 12:35:23.040706 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:23.040674    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-9xcdr" event={"ID":"5e812af3-80d2-490c-8915-6150569a1fe9","Type":"ContainerStarted","Data":"8d4dcc39a83196202f46a58cc1cba61bb05724825e9f466bed5c0a98b426f6d9"}
May 25 12:35:23.041184 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:23.040714    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-9xcdr" event={"ID":"5e812af3-80d2-490c-8915-6150569a1fe9","Type":"ContainerStarted","Data":"d13cb269fede4488cfdb5142a94d98a746a9a89ee0a02439f62243cd968bd92b"}
May 25 12:35:23.041184 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:23.040921    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-9xcdr"
May 25 12:35:23.057511 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:23.057449    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-9xcdr" podStartSLOduration=6.057432579 podStartE2EDuration="6.057432579s" podCreationTimestamp="2026-05-25 12:35:17 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:35:23.056427285 +0000 UTC m=+1336.016477691" watchObservedRunningTime="2026-05-25 12:35:23.057432579 +0000 UTC m=+1336.017482984"
May 25 12:35:24.044460 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:24.044427    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-9xcdr"
May 25 12:35:24.045729 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:24.045694    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-9xcdr" podUID="5e812af3-80d2-490c-8915-6150569a1fe9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
May 25 12:35:25.047771 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:25.047730    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-9xcdr" podUID="5e812af3-80d2-490c-8915-6150569a1fe9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
May 25 12:35:30.051697 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:30.051608    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-9xcdr"
May 25 12:35:30.052654 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:30.052624    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-9xcdr"
May 25 12:35:37.663435 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:37.663396    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-9xcdr"]
May 25 12:35:37.663818 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:37.663764    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-9xcdr" podUID="5e812af3-80d2-490c-8915-6150569a1fe9" containerName="kserve-container" containerID="cri-o://d13cb269fede4488cfdb5142a94d98a746a9a89ee0a02439f62243cd968bd92b" gracePeriod=30
May 25 12:35:37.663898 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:37.663825    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-9xcdr" podUID="5e812af3-80d2-490c-8915-6150569a1fe9" containerName="kube-rbac-proxy" containerID="cri-o://8d4dcc39a83196202f46a58cc1cba61bb05724825e9f466bed5c0a98b426f6d9" gracePeriod=30
May 25 12:35:37.756620 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:37.756587    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-9x6qf"]
May 25 12:35:37.757093 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:37.757077    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a54d9e9c-68d3-498b-ab6c-89259961d753" containerName="kube-rbac-proxy"
May 25 12:35:37.757177 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:37.757096    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a54d9e9c-68d3-498b-ab6c-89259961d753" containerName="kube-rbac-proxy"
May 25 12:35:37.757177 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:37.757117    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a54d9e9c-68d3-498b-ab6c-89259961d753" containerName="kserve-container"
May 25 12:35:37.757177 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:37.757123    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a54d9e9c-68d3-498b-ab6c-89259961d753" containerName="kserve-container"
May 25 12:35:37.757177 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:37.757132    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a54d9e9c-68d3-498b-ab6c-89259961d753" containerName="storage-initializer"
May 25 12:35:37.757177 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:37.757138    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a54d9e9c-68d3-498b-ab6c-89259961d753" containerName="storage-initializer"
May 25 12:35:37.757332 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:37.757191    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="a54d9e9c-68d3-498b-ab6c-89259961d753" containerName="kube-rbac-proxy"
May 25 12:35:37.757332 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:37.757201    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="a54d9e9c-68d3-498b-ab6c-89259961d753" containerName="kserve-container"
May 25 12:35:37.759316 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:37.759301    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-9x6qf"
May 25 12:35:37.761062 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:37.761038    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
May 25 12:35:37.761062 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:37.761050    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
May 25 12:35:37.771064 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:37.771044    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-9x6qf"]
May 25 12:35:37.858171 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:37.858134    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56a13387-cda8-4551-bf33-3a3e49be3217-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-9x6qf\" (UID: \"56a13387-cda8-4551-bf33-3a3e49be3217\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-9x6qf"
May 25 12:35:37.858322 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:37.858195    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56a13387-cda8-4551-bf33-3a3e49be3217-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-9x6qf\" (UID: \"56a13387-cda8-4551-bf33-3a3e49be3217\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-9x6qf"
May 25 12:35:37.858322 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:37.858295    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56a13387-cda8-4551-bf33-3a3e49be3217-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-9x6qf\" (UID: \"56a13387-cda8-4551-bf33-3a3e49be3217\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-9x6qf"
May 25 12:35:37.858415 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:37.858360    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7wvj\" (UniqueName: \"kubernetes.io/projected/56a13387-cda8-4551-bf33-3a3e49be3217-kube-api-access-l7wvj\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-9x6qf\" (UID: \"56a13387-cda8-4551-bf33-3a3e49be3217\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-9x6qf"
May 25 12:35:37.959349 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:37.959306    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56a13387-cda8-4551-bf33-3a3e49be3217-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-9x6qf\" (UID: \"56a13387-cda8-4551-bf33-3a3e49be3217\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-9x6qf"
May 25 12:35:37.959587 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:37.959415    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-l7wvj\" (UniqueName: \"kubernetes.io/projected/56a13387-cda8-4551-bf33-3a3e49be3217-kube-api-access-l7wvj\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-9x6qf\" (UID: \"56a13387-cda8-4551-bf33-3a3e49be3217\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-9x6qf"
May 25 12:35:37.959587 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:37.959459    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56a13387-cda8-4551-bf33-3a3e49be3217-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-9x6qf\" (UID: \"56a13387-cda8-4551-bf33-3a3e49be3217\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-9x6qf"
May 25 12:35:37.959587 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:37.959497    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56a13387-cda8-4551-bf33-3a3e49be3217-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-9x6qf\" (UID: \"56a13387-cda8-4551-bf33-3a3e49be3217\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-9x6qf"
May 25 12:35:37.959911 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:37.959891    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56a13387-cda8-4551-bf33-3a3e49be3217-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-9x6qf\" (UID: \"56a13387-cda8-4551-bf33-3a3e49be3217\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-9x6qf"
May 25 12:35:37.960308 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:37.960282    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56a13387-cda8-4551-bf33-3a3e49be3217-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-9x6qf\" (UID: \"56a13387-cda8-4551-bf33-3a3e49be3217\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-9x6qf"
May 25 12:35:37.961778 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:37.961759    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56a13387-cda8-4551-bf33-3a3e49be3217-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-9x6qf\" (UID: \"56a13387-cda8-4551-bf33-3a3e49be3217\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-9x6qf"
May 25 12:35:37.966619 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:37.966597    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7wvj\" (UniqueName: \"kubernetes.io/projected/56a13387-cda8-4551-bf33-3a3e49be3217-kube-api-access-l7wvj\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-9x6qf\" (UID: \"56a13387-cda8-4551-bf33-3a3e49be3217\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-9x6qf"
May 25 12:35:38.070180 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:38.070144    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-9x6qf"
May 25 12:35:38.093854 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:38.093814    2555 generic.go:358] "Generic (PLEG): container finished" podID="5e812af3-80d2-490c-8915-6150569a1fe9" containerID="8d4dcc39a83196202f46a58cc1cba61bb05724825e9f466bed5c0a98b426f6d9" exitCode=2
May 25 12:35:38.094070 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:38.093869    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-9xcdr" event={"ID":"5e812af3-80d2-490c-8915-6150569a1fe9","Type":"ContainerDied","Data":"8d4dcc39a83196202f46a58cc1cba61bb05724825e9f466bed5c0a98b426f6d9"}
May 25 12:35:38.199004 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:38.198959    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-9x6qf"]
May 25 12:35:38.201600 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:35:38.201571    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56a13387_cda8_4551_bf33_3a3e49be3217.slice/crio-fc007f4399445ba3932452d3ca1b244c50a06bf2443532d2d11aca4b5652d716 WatchSource:0}: Error finding container fc007f4399445ba3932452d3ca1b244c50a06bf2443532d2d11aca4b5652d716: Status 404 returned error can't find the container with id fc007f4399445ba3932452d3ca1b244c50a06bf2443532d2d11aca4b5652d716
May 25 12:35:38.300229 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:38.300203    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-9xcdr"
May 25 12:35:38.363721 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:38.363692    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e812af3-80d2-490c-8915-6150569a1fe9-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\") pod \"5e812af3-80d2-490c-8915-6150569a1fe9\" (UID: \"5e812af3-80d2-490c-8915-6150569a1fe9\") "
May 25 12:35:38.363847 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:38.363737    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e812af3-80d2-490c-8915-6150569a1fe9-proxy-tls\") pod \"5e812af3-80d2-490c-8915-6150569a1fe9\" (UID: \"5e812af3-80d2-490c-8915-6150569a1fe9\") "
May 25 12:35:38.363847 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:38.363825    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-649p9\" (UniqueName: \"kubernetes.io/projected/5e812af3-80d2-490c-8915-6150569a1fe9-kube-api-access-649p9\") pod \"5e812af3-80d2-490c-8915-6150569a1fe9\" (UID: \"5e812af3-80d2-490c-8915-6150569a1fe9\") "
May 25 12:35:38.363950 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:38.363873    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e812af3-80d2-490c-8915-6150569a1fe9-kserve-provision-location\") pod \"5e812af3-80d2-490c-8915-6150569a1fe9\" (UID: \"5e812af3-80d2-490c-8915-6150569a1fe9\") "
May 25 12:35:38.364056 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:38.364032    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e812af3-80d2-490c-8915-6150569a1fe9-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config") pod "5e812af3-80d2-490c-8915-6150569a1fe9" (UID: "5e812af3-80d2-490c-8915-6150569a1fe9"). InnerVolumeSpecName "isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:35:38.364154 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:38.364121    2555 reconciler_common.go:299] "Volume detached for volume \"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e812af3-80d2-490c-8915-6150569a1fe9-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:35:38.364357 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:38.364331    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e812af3-80d2-490c-8915-6150569a1fe9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5e812af3-80d2-490c-8915-6150569a1fe9" (UID: "5e812af3-80d2-490c-8915-6150569a1fe9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:35:38.365873 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:38.365851    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e812af3-80d2-490c-8915-6150569a1fe9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5e812af3-80d2-490c-8915-6150569a1fe9" (UID: "5e812af3-80d2-490c-8915-6150569a1fe9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:35:38.365953 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:38.365931    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e812af3-80d2-490c-8915-6150569a1fe9-kube-api-access-649p9" (OuterVolumeSpecName: "kube-api-access-649p9") pod "5e812af3-80d2-490c-8915-6150569a1fe9" (UID: "5e812af3-80d2-490c-8915-6150569a1fe9"). InnerVolumeSpecName "kube-api-access-649p9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:35:38.464788 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:38.464754    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-649p9\" (UniqueName: \"kubernetes.io/projected/5e812af3-80d2-490c-8915-6150569a1fe9-kube-api-access-649p9\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:35:38.464788 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:38.464783    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e812af3-80d2-490c-8915-6150569a1fe9-kserve-provision-location\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:35:38.464963 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:38.464797    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e812af3-80d2-490c-8915-6150569a1fe9-proxy-tls\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:35:39.098882 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:39.098845    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-9x6qf" event={"ID":"56a13387-cda8-4551-bf33-3a3e49be3217","Type":"ContainerStarted","Data":"a1e18082faf4657d383570034b9b47f6524ede14a3ccd6435792789ed36acef4"}
May 25 12:35:39.098882 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:39.098888    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-9x6qf" event={"ID":"56a13387-cda8-4551-bf33-3a3e49be3217","Type":"ContainerStarted","Data":"fc007f4399445ba3932452d3ca1b244c50a06bf2443532d2d11aca4b5652d716"}
May 25 12:35:39.100489 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:39.100465    2555 generic.go:358] "Generic (PLEG): container finished" podID="5e812af3-80d2-490c-8915-6150569a1fe9" containerID="d13cb269fede4488cfdb5142a94d98a746a9a89ee0a02439f62243cd968bd92b" exitCode=0
May 25 12:35:39.100620 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:39.100523    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-9xcdr" event={"ID":"5e812af3-80d2-490c-8915-6150569a1fe9","Type":"ContainerDied","Data":"d13cb269fede4488cfdb5142a94d98a746a9a89ee0a02439f62243cd968bd92b"}
May 25 12:35:39.100620 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:39.100543    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-9xcdr" event={"ID":"5e812af3-80d2-490c-8915-6150569a1fe9","Type":"ContainerDied","Data":"fe075b5cbc25a228b6b611d43edf6f6923c07bd7b01557db973160061de25adc"}
May 25 12:35:39.100620 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:39.100558    2555 scope.go:117] "RemoveContainer" containerID="8d4dcc39a83196202f46a58cc1cba61bb05724825e9f466bed5c0a98b426f6d9"
May 25 12:35:39.100620 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:39.100526    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-9xcdr"
May 25 12:35:39.108479 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:39.108460    2555 scope.go:117] "RemoveContainer" containerID="d13cb269fede4488cfdb5142a94d98a746a9a89ee0a02439f62243cd968bd92b"
May 25 12:35:39.115491 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:39.115473    2555 scope.go:117] "RemoveContainer" containerID="899e71a228df7a096dbb6735942e5449febc113beb332040cf98e9b646b6712d"
May 25 12:35:39.123171 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:39.123157    2555 scope.go:117] "RemoveContainer" containerID="8d4dcc39a83196202f46a58cc1cba61bb05724825e9f466bed5c0a98b426f6d9"
May 25 12:35:39.123405 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:35:39.123387    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d4dcc39a83196202f46a58cc1cba61bb05724825e9f466bed5c0a98b426f6d9\": container with ID starting with 8d4dcc39a83196202f46a58cc1cba61bb05724825e9f466bed5c0a98b426f6d9 not found: ID does not exist" containerID="8d4dcc39a83196202f46a58cc1cba61bb05724825e9f466bed5c0a98b426f6d9"
May 25 12:35:39.123469 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:39.123411    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d4dcc39a83196202f46a58cc1cba61bb05724825e9f466bed5c0a98b426f6d9"} err="failed to get container status \"8d4dcc39a83196202f46a58cc1cba61bb05724825e9f466bed5c0a98b426f6d9\": rpc error: code = NotFound desc = could not find container \"8d4dcc39a83196202f46a58cc1cba61bb05724825e9f466bed5c0a98b426f6d9\": container with ID starting with 8d4dcc39a83196202f46a58cc1cba61bb05724825e9f466bed5c0a98b426f6d9 not found: ID does not exist"
May 25 12:35:39.123469 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:39.123426    2555 scope.go:117] "RemoveContainer" containerID="d13cb269fede4488cfdb5142a94d98a746a9a89ee0a02439f62243cd968bd92b"
May 25 12:35:39.123669 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:35:39.123634    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d13cb269fede4488cfdb5142a94d98a746a9a89ee0a02439f62243cd968bd92b\": container with ID starting with d13cb269fede4488cfdb5142a94d98a746a9a89ee0a02439f62243cd968bd92b not found: ID does not exist" containerID="d13cb269fede4488cfdb5142a94d98a746a9a89ee0a02439f62243cd968bd92b"
May 25 12:35:39.123708 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:39.123677    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d13cb269fede4488cfdb5142a94d98a746a9a89ee0a02439f62243cd968bd92b"} err="failed to get container status \"d13cb269fede4488cfdb5142a94d98a746a9a89ee0a02439f62243cd968bd92b\": rpc error: code = NotFound desc = could not find container \"d13cb269fede4488cfdb5142a94d98a746a9a89ee0a02439f62243cd968bd92b\": container with ID starting with d13cb269fede4488cfdb5142a94d98a746a9a89ee0a02439f62243cd968bd92b not found: ID does not exist"
May 25 12:35:39.123708 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:39.123697    2555 scope.go:117] "RemoveContainer" containerID="899e71a228df7a096dbb6735942e5449febc113beb332040cf98e9b646b6712d"
May 25 12:35:39.123926 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:35:39.123911    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"899e71a228df7a096dbb6735942e5449febc113beb332040cf98e9b646b6712d\": container with ID starting with 899e71a228df7a096dbb6735942e5449febc113beb332040cf98e9b646b6712d not found: ID does not exist" containerID="899e71a228df7a096dbb6735942e5449febc113beb332040cf98e9b646b6712d"
May 25 12:35:39.123963 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:39.123932    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"899e71a228df7a096dbb6735942e5449febc113beb332040cf98e9b646b6712d"} err="failed to get container status \"899e71a228df7a096dbb6735942e5449febc113beb332040cf98e9b646b6712d\": rpc error: code = NotFound desc = could not find container \"899e71a228df7a096dbb6735942e5449febc113beb332040cf98e9b646b6712d\": container with ID starting with 899e71a228df7a096dbb6735942e5449febc113beb332040cf98e9b646b6712d not found: ID does not exist"
May 25 12:35:39.129275 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:39.129252    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-9xcdr"]
May 25 12:35:39.132419 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:39.132389    2555 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-9xcdr"]
May 25 12:35:39.599109 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:39.599080    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e812af3-80d2-490c-8915-6150569a1fe9" path="/var/lib/kubelet/pods/5e812af3-80d2-490c-8915-6150569a1fe9/volumes"
May 25 12:35:43.116582 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:43.116546    2555 generic.go:358] "Generic (PLEG): container finished" podID="56a13387-cda8-4551-bf33-3a3e49be3217" containerID="a1e18082faf4657d383570034b9b47f6524ede14a3ccd6435792789ed36acef4" exitCode=0
May 25 12:35:43.116994 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:43.116629    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-9x6qf" event={"ID":"56a13387-cda8-4551-bf33-3a3e49be3217","Type":"ContainerDied","Data":"a1e18082faf4657d383570034b9b47f6524ede14a3ccd6435792789ed36acef4"}
May 25 12:35:44.122436 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:44.122400    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-9x6qf" event={"ID":"56a13387-cda8-4551-bf33-3a3e49be3217","Type":"ContainerStarted","Data":"d223382f51619370ea0560c7304f59b35e800f8e1995b37a181a55cc632c65e1"}
May 25 12:35:44.122862 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:44.122443    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-9x6qf" event={"ID":"56a13387-cda8-4551-bf33-3a3e49be3217","Type":"ContainerStarted","Data":"7fca194193cbf089e8325f0aa23df7a79e8fc146bb9664b06112f6dc08577523"}
May 25 12:35:44.122862 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:44.122690    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-9x6qf"
May 25 12:35:44.139744 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:44.139700    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-9x6qf" podStartSLOduration=7.139686636 podStartE2EDuration="7.139686636s" podCreationTimestamp="2026-05-25 12:35:37 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:35:44.137679082 +0000 UTC m=+1357.097729489" watchObservedRunningTime="2026-05-25 12:35:44.139686636 +0000 UTC m=+1357.099737042"
May 25 12:35:45.126709 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:45.126674    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-9x6qf"
May 25 12:35:51.137232 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:35:51.137204    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-9x6qf"
May 25 12:36:21.141247 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:21.141209    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-9x6qf"
May 25 12:36:27.828946 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:27.828910    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-9x6qf"]
May 25 12:36:27.829378 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:27.829219    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-9x6qf" podUID="56a13387-cda8-4551-bf33-3a3e49be3217" containerName="kserve-container" containerID="cri-o://7fca194193cbf089e8325f0aa23df7a79e8fc146bb9664b06112f6dc08577523" gracePeriod=30
May 25 12:36:27.829378 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:27.829258    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-9x6qf" podUID="56a13387-cda8-4551-bf33-3a3e49be3217" containerName="kube-rbac-proxy" containerID="cri-o://d223382f51619370ea0560c7304f59b35e800f8e1995b37a181a55cc632c65e1" gracePeriod=30
May 25 12:36:27.920994 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:27.920957    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh"]
May 25 12:36:27.921308 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:27.921295    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e812af3-80d2-490c-8915-6150569a1fe9" containerName="kserve-container"
May 25 12:36:27.921369 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:27.921309    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e812af3-80d2-490c-8915-6150569a1fe9" containerName="kserve-container"
May 25 12:36:27.921369 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:27.921320    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e812af3-80d2-490c-8915-6150569a1fe9" containerName="storage-initializer"
May 25 12:36:27.921369 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:27.921326    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e812af3-80d2-490c-8915-6150569a1fe9" containerName="storage-initializer"
May 25 12:36:27.921369 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:27.921344    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e812af3-80d2-490c-8915-6150569a1fe9" containerName="kube-rbac-proxy"
May 25 12:36:27.921369 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:27.921351    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e812af3-80d2-490c-8915-6150569a1fe9" containerName="kube-rbac-proxy"
May 25 12:36:27.921595 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:27.921407    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="5e812af3-80d2-490c-8915-6150569a1fe9" containerName="kube-rbac-proxy"
May 25 12:36:27.921595 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:27.921420    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="5e812af3-80d2-490c-8915-6150569a1fe9" containerName="kserve-container"
May 25 12:36:27.924337 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:27.924319    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh"
May 25 12:36:27.926069 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:27.926048    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
May 25 12:36:27.926175 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:27.926145    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
May 25 12:36:27.935576 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:27.935531    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh"]
May 25 12:36:27.961751 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:27.961713    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cc3c087-83c6-41d7-8e55-dc8756f9aff7-isvc-sklearn-mcp-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh\" (UID: \"5cc3c087-83c6-41d7-8e55-dc8756f9aff7\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh"
May 25 12:36:27.961888 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:27.961755    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5cc3c087-83c6-41d7-8e55-dc8756f9aff7-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh\" (UID: \"5cc3c087-83c6-41d7-8e55-dc8756f9aff7\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh"
May 25 12:36:27.961888 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:27.961781    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cc3c087-83c6-41d7-8e55-dc8756f9aff7-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh\" (UID: \"5cc3c087-83c6-41d7-8e55-dc8756f9aff7\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh"
May 25 12:36:27.961888 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:27.961828    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kttr2\" (UniqueName: \"kubernetes.io/projected/5cc3c087-83c6-41d7-8e55-dc8756f9aff7-kube-api-access-kttr2\") pod \"isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh\" (UID: \"5cc3c087-83c6-41d7-8e55-dc8756f9aff7\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh"
May 25 12:36:28.062847 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:28.062801    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cc3c087-83c6-41d7-8e55-dc8756f9aff7-isvc-sklearn-mcp-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh\" (UID: \"5cc3c087-83c6-41d7-8e55-dc8756f9aff7\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh"
May 25 12:36:28.062847 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:28.062849    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5cc3c087-83c6-41d7-8e55-dc8756f9aff7-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh\" (UID: \"5cc3c087-83c6-41d7-8e55-dc8756f9aff7\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh"
May 25 12:36:28.063124 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:28.062880    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cc3c087-83c6-41d7-8e55-dc8756f9aff7-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh\" (UID: \"5cc3c087-83c6-41d7-8e55-dc8756f9aff7\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh"
May 25 12:36:28.063124 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:28.062920    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kttr2\" (UniqueName: \"kubernetes.io/projected/5cc3c087-83c6-41d7-8e55-dc8756f9aff7-kube-api-access-kttr2\") pod \"isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh\" (UID: \"5cc3c087-83c6-41d7-8e55-dc8756f9aff7\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh"
May 25 12:36:28.063124 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:36:28.063039    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-serving-cert: secret "isvc-sklearn-mcp-predictor-serving-cert" not found
May 25 12:36:28.063124 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:36:28.063109    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cc3c087-83c6-41d7-8e55-dc8756f9aff7-proxy-tls podName:5cc3c087-83c6-41d7-8e55-dc8756f9aff7 nodeName:}" failed. No retries permitted until 2026-05-25 12:36:28.56308876 +0000 UTC m=+1401.523139157 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5cc3c087-83c6-41d7-8e55-dc8756f9aff7-proxy-tls") pod "isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh" (UID: "5cc3c087-83c6-41d7-8e55-dc8756f9aff7") : secret "isvc-sklearn-mcp-predictor-serving-cert" not found
May 25 12:36:28.063345 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:28.063304    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5cc3c087-83c6-41d7-8e55-dc8756f9aff7-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh\" (UID: \"5cc3c087-83c6-41d7-8e55-dc8756f9aff7\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh"
May 25 12:36:28.063528 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:28.063505    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cc3c087-83c6-41d7-8e55-dc8756f9aff7-isvc-sklearn-mcp-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh\" (UID: \"5cc3c087-83c6-41d7-8e55-dc8756f9aff7\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh"
May 25 12:36:28.070896 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:28.070872    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kttr2\" (UniqueName: \"kubernetes.io/projected/5cc3c087-83c6-41d7-8e55-dc8756f9aff7-kube-api-access-kttr2\") pod \"isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh\" (UID: \"5cc3c087-83c6-41d7-8e55-dc8756f9aff7\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh"
May 25 12:36:28.278039 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:28.278004    2555 generic.go:358] "Generic (PLEG): container finished" podID="56a13387-cda8-4551-bf33-3a3e49be3217" containerID="d223382f51619370ea0560c7304f59b35e800f8e1995b37a181a55cc632c65e1" exitCode=2
May 25 12:36:28.278205 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:28.278070    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-9x6qf" event={"ID":"56a13387-cda8-4551-bf33-3a3e49be3217","Type":"ContainerDied","Data":"d223382f51619370ea0560c7304f59b35e800f8e1995b37a181a55cc632c65e1"}
May 25 12:36:28.567247 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:28.567161    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cc3c087-83c6-41d7-8e55-dc8756f9aff7-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh\" (UID: \"5cc3c087-83c6-41d7-8e55-dc8756f9aff7\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh"
May 25 12:36:28.569792 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:28.569758    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cc3c087-83c6-41d7-8e55-dc8756f9aff7-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh\" (UID: \"5cc3c087-83c6-41d7-8e55-dc8756f9aff7\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh"
May 25 12:36:28.835563 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:28.835508    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh"
May 25 12:36:28.973020 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:28.972993    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh"]
May 25 12:36:28.975250 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:36:28.975222    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cc3c087_83c6_41d7_8e55_dc8756f9aff7.slice/crio-e3601c397ff6e2a0034e2eb5f1b0ffc62006e47f3ce06a8db055661f681e823f WatchSource:0}: Error finding container e3601c397ff6e2a0034e2eb5f1b0ffc62006e47f3ce06a8db055661f681e823f: Status 404 returned error can't find the container with id e3601c397ff6e2a0034e2eb5f1b0ffc62006e47f3ce06a8db055661f681e823f
May 25 12:36:28.989676 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:28.989631    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-9x6qf"
May 25 12:36:29.071658 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:29.071620    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56a13387-cda8-4551-bf33-3a3e49be3217-proxy-tls\") pod \"56a13387-cda8-4551-bf33-3a3e49be3217\" (UID: \"56a13387-cda8-4551-bf33-3a3e49be3217\") "
May 25 12:36:29.071757 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:29.071733    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56a13387-cda8-4551-bf33-3a3e49be3217-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\") pod \"56a13387-cda8-4551-bf33-3a3e49be3217\" (UID: \"56a13387-cda8-4551-bf33-3a3e49be3217\") "
May 25 12:36:29.071814 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:29.071781    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7wvj\" (UniqueName: \"kubernetes.io/projected/56a13387-cda8-4551-bf33-3a3e49be3217-kube-api-access-l7wvj\") pod \"56a13387-cda8-4551-bf33-3a3e49be3217\" (UID: \"56a13387-cda8-4551-bf33-3a3e49be3217\") "
May 25 12:36:29.071864 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:29.071809    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56a13387-cda8-4551-bf33-3a3e49be3217-kserve-provision-location\") pod \"56a13387-cda8-4551-bf33-3a3e49be3217\" (UID: \"56a13387-cda8-4551-bf33-3a3e49be3217\") "
May 25 12:36:29.072145 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:29.072042    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56a13387-cda8-4551-bf33-3a3e49be3217-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config") pod "56a13387-cda8-4551-bf33-3a3e49be3217" (UID: "56a13387-cda8-4551-bf33-3a3e49be3217"). InnerVolumeSpecName "isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:36:29.072223 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:29.072181    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56a13387-cda8-4551-bf33-3a3e49be3217-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "56a13387-cda8-4551-bf33-3a3e49be3217" (UID: "56a13387-cda8-4551-bf33-3a3e49be3217"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:36:29.073568 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:29.073550    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56a13387-cda8-4551-bf33-3a3e49be3217-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "56a13387-cda8-4551-bf33-3a3e49be3217" (UID: "56a13387-cda8-4551-bf33-3a3e49be3217"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:36:29.073822 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:29.073794    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56a13387-cda8-4551-bf33-3a3e49be3217-kube-api-access-l7wvj" (OuterVolumeSpecName: "kube-api-access-l7wvj") pod "56a13387-cda8-4551-bf33-3a3e49be3217" (UID: "56a13387-cda8-4551-bf33-3a3e49be3217"). InnerVolumeSpecName "kube-api-access-l7wvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:36:29.173293 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:29.173236    2555 reconciler_common.go:299] "Volume detached for volume \"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56a13387-cda8-4551-bf33-3a3e49be3217-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:36:29.173293 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:29.173260    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7wvj\" (UniqueName: \"kubernetes.io/projected/56a13387-cda8-4551-bf33-3a3e49be3217-kube-api-access-l7wvj\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:36:29.173293 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:29.173271    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56a13387-cda8-4551-bf33-3a3e49be3217-kserve-provision-location\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:36:29.173293 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:29.173281    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56a13387-cda8-4551-bf33-3a3e49be3217-proxy-tls\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:36:29.282892 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:29.282862    2555 generic.go:358] "Generic (PLEG): container finished" podID="56a13387-cda8-4551-bf33-3a3e49be3217" containerID="7fca194193cbf089e8325f0aa23df7a79e8fc146bb9664b06112f6dc08577523" exitCode=0
May 25 12:36:29.283065 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:29.282950    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-9x6qf"
May 25 12:36:29.283065 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:29.282946    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-9x6qf" event={"ID":"56a13387-cda8-4551-bf33-3a3e49be3217","Type":"ContainerDied","Data":"7fca194193cbf089e8325f0aa23df7a79e8fc146bb9664b06112f6dc08577523"}
May 25 12:36:29.283065 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:29.282999    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-9x6qf" event={"ID":"56a13387-cda8-4551-bf33-3a3e49be3217","Type":"ContainerDied","Data":"fc007f4399445ba3932452d3ca1b244c50a06bf2443532d2d11aca4b5652d716"}
May 25 12:36:29.283065 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:29.283020    2555 scope.go:117] "RemoveContainer" containerID="d223382f51619370ea0560c7304f59b35e800f8e1995b37a181a55cc632c65e1"
May 25 12:36:29.284533 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:29.284511    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh" event={"ID":"5cc3c087-83c6-41d7-8e55-dc8756f9aff7","Type":"ContainerStarted","Data":"3b255f167c26a210eec95267815135c92204cd8666dfe8f2819a3d7d98002289"}
May 25 12:36:29.284632 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:29.284545    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh" event={"ID":"5cc3c087-83c6-41d7-8e55-dc8756f9aff7","Type":"ContainerStarted","Data":"e3601c397ff6e2a0034e2eb5f1b0ffc62006e47f3ce06a8db055661f681e823f"}
May 25 12:36:29.291803 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:29.291781    2555 scope.go:117] "RemoveContainer" containerID="7fca194193cbf089e8325f0aa23df7a79e8fc146bb9664b06112f6dc08577523"
May 25 12:36:29.299254 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:29.299236    2555 scope.go:117] "RemoveContainer" containerID="a1e18082faf4657d383570034b9b47f6524ede14a3ccd6435792789ed36acef4"
May 25 12:36:29.307637 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:29.307618    2555 scope.go:117] "RemoveContainer" containerID="d223382f51619370ea0560c7304f59b35e800f8e1995b37a181a55cc632c65e1"
May 25 12:36:29.307921 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:36:29.307900    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d223382f51619370ea0560c7304f59b35e800f8e1995b37a181a55cc632c65e1\": container with ID starting with d223382f51619370ea0560c7304f59b35e800f8e1995b37a181a55cc632c65e1 not found: ID does not exist" containerID="d223382f51619370ea0560c7304f59b35e800f8e1995b37a181a55cc632c65e1"
May 25 12:36:29.307985 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:29.307929    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d223382f51619370ea0560c7304f59b35e800f8e1995b37a181a55cc632c65e1"} err="failed to get container status \"d223382f51619370ea0560c7304f59b35e800f8e1995b37a181a55cc632c65e1\": rpc error: code = NotFound desc = could not find container \"d223382f51619370ea0560c7304f59b35e800f8e1995b37a181a55cc632c65e1\": container with ID starting with d223382f51619370ea0560c7304f59b35e800f8e1995b37a181a55cc632c65e1 not found: ID does not exist"
May 25 12:36:29.307985 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:29.307944    2555 scope.go:117] "RemoveContainer" containerID="7fca194193cbf089e8325f0aa23df7a79e8fc146bb9664b06112f6dc08577523"
May 25 12:36:29.308193 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:36:29.308176    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fca194193cbf089e8325f0aa23df7a79e8fc146bb9664b06112f6dc08577523\": container with ID starting with 7fca194193cbf089e8325f0aa23df7a79e8fc146bb9664b06112f6dc08577523 not found: ID does not exist" containerID="7fca194193cbf089e8325f0aa23df7a79e8fc146bb9664b06112f6dc08577523"
May 25 12:36:29.308234 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:29.308201    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fca194193cbf089e8325f0aa23df7a79e8fc146bb9664b06112f6dc08577523"} err="failed to get container status \"7fca194193cbf089e8325f0aa23df7a79e8fc146bb9664b06112f6dc08577523\": rpc error: code = NotFound desc = could not find container \"7fca194193cbf089e8325f0aa23df7a79e8fc146bb9664b06112f6dc08577523\": container with ID starting with 7fca194193cbf089e8325f0aa23df7a79e8fc146bb9664b06112f6dc08577523 not found: ID does not exist"
May 25 12:36:29.308234 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:29.308217    2555 scope.go:117] "RemoveContainer" containerID="a1e18082faf4657d383570034b9b47f6524ede14a3ccd6435792789ed36acef4"
May 25 12:36:29.308421 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:36:29.308405    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1e18082faf4657d383570034b9b47f6524ede14a3ccd6435792789ed36acef4\": container with ID starting with a1e18082faf4657d383570034b9b47f6524ede14a3ccd6435792789ed36acef4 not found: ID does not exist" containerID="a1e18082faf4657d383570034b9b47f6524ede14a3ccd6435792789ed36acef4"
May 25 12:36:29.308456 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:29.308424    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1e18082faf4657d383570034b9b47f6524ede14a3ccd6435792789ed36acef4"} err="failed to get container status \"a1e18082faf4657d383570034b9b47f6524ede14a3ccd6435792789ed36acef4\": rpc error: code = NotFound desc = could not find container \"a1e18082faf4657d383570034b9b47f6524ede14a3ccd6435792789ed36acef4\": container with ID starting with a1e18082faf4657d383570034b9b47f6524ede14a3ccd6435792789ed36acef4 not found: ID does not exist"
May 25 12:36:29.316992 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:29.316969    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-9x6qf"]
May 25 12:36:29.320085 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:29.320066    2555 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-9x6qf"]
May 25 12:36:29.603937 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:29.600096    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56a13387-cda8-4551-bf33-3a3e49be3217" path="/var/lib/kubelet/pods/56a13387-cda8-4551-bf33-3a3e49be3217/volumes"
May 25 12:36:33.299182 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:33.299150    2555 generic.go:358] "Generic (PLEG): container finished" podID="5cc3c087-83c6-41d7-8e55-dc8756f9aff7" containerID="3b255f167c26a210eec95267815135c92204cd8666dfe8f2819a3d7d98002289" exitCode=0
May 25 12:36:33.299545 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:33.299224    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh" event={"ID":"5cc3c087-83c6-41d7-8e55-dc8756f9aff7","Type":"ContainerDied","Data":"3b255f167c26a210eec95267815135c92204cd8666dfe8f2819a3d7d98002289"}
May 25 12:36:34.303993 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:34.303961    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh" event={"ID":"5cc3c087-83c6-41d7-8e55-dc8756f9aff7","Type":"ContainerStarted","Data":"daa1b331f39a6a172a2fe289f51879ebc78be64a9259904a3065591db7509eae"}
May 25 12:36:36.317148 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:36.317110    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh" event={"ID":"5cc3c087-83c6-41d7-8e55-dc8756f9aff7","Type":"ContainerStarted","Data":"ee52c4dad45e87c52d4fd29b9fd5b468a14cf17daf8b2f7087d753daaf1973be"}
May 25 12:36:36.317604 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:36.317154    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh" event={"ID":"5cc3c087-83c6-41d7-8e55-dc8756f9aff7","Type":"ContainerStarted","Data":"29b0a7e7a62db0ba83667d1e5e45c510ed3522335d834817d6bcac35e72c41e9"}
May 25 12:36:36.317604 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:36.317544    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh"
May 25 12:36:36.335722 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:36.335676    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh" podStartSLOduration=7.348359481 podStartE2EDuration="9.335664354s" podCreationTimestamp="2026-05-25 12:36:27 +0000 UTC" firstStartedPulling="2026-05-25 12:36:33.356823198 +0000 UTC m=+1406.316873589" lastFinishedPulling="2026-05-25 12:36:35.344128075 +0000 UTC m=+1408.304178462" observedRunningTime="2026-05-25 12:36:36.334238345 +0000 UTC m=+1409.294288751" watchObservedRunningTime="2026-05-25 12:36:36.335664354 +0000 UTC m=+1409.295714759"
May 25 12:36:37.320474 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:37.320443    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh"
May 25 12:36:37.320474 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:37.320478    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh"
May 25 12:36:43.328908 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:36:43.328876    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh"
May 25 12:37:03.330910 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:37:03.330838    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh"
May 25 12:37:43.331815 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:37:43.331786    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh"
May 25 12:37:48.001372 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:37:48.001336    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh"]
May 25 12:37:48.001768 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:37:48.001722    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh" podUID="5cc3c087-83c6-41d7-8e55-dc8756f9aff7" containerName="kserve-agent" containerID="cri-o://29b0a7e7a62db0ba83667d1e5e45c510ed3522335d834817d6bcac35e72c41e9" gracePeriod=30
May 25 12:37:48.001768 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:37:48.001728    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh" podUID="5cc3c087-83c6-41d7-8e55-dc8756f9aff7" containerName="kube-rbac-proxy" containerID="cri-o://ee52c4dad45e87c52d4fd29b9fd5b468a14cf17daf8b2f7087d753daaf1973be" gracePeriod=30
May 25 12:37:48.001945 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:37:48.001715    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh" podUID="5cc3c087-83c6-41d7-8e55-dc8756f9aff7" containerName="kserve-container" containerID="cri-o://daa1b331f39a6a172a2fe289f51879ebc78be64a9259904a3065591db7509eae" gracePeriod=30
May 25 12:37:48.324554 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:37:48.324471    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh" podUID="5cc3c087-83c6-41d7-8e55-dc8756f9aff7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.46:8643/healthz\": dial tcp 10.134.0.46:8643: connect: connection refused"
May 25 12:37:48.560883 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:37:48.560849    2555 generic.go:358] "Generic (PLEG): container finished" podID="5cc3c087-83c6-41d7-8e55-dc8756f9aff7" containerID="ee52c4dad45e87c52d4fd29b9fd5b468a14cf17daf8b2f7087d753daaf1973be" exitCode=2
May 25 12:37:48.561048 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:37:48.560922    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh" event={"ID":"5cc3c087-83c6-41d7-8e55-dc8756f9aff7","Type":"ContainerDied","Data":"ee52c4dad45e87c52d4fd29b9fd5b468a14cf17daf8b2f7087d753daaf1973be"}
May 25 12:37:50.569187 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:37:50.569155    2555 generic.go:358] "Generic (PLEG): container finished" podID="5cc3c087-83c6-41d7-8e55-dc8756f9aff7" containerID="daa1b331f39a6a172a2fe289f51879ebc78be64a9259904a3065591db7509eae" exitCode=0
May 25 12:37:50.569572 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:37:50.569234    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh" event={"ID":"5cc3c087-83c6-41d7-8e55-dc8756f9aff7","Type":"ContainerDied","Data":"daa1b331f39a6a172a2fe289f51879ebc78be64a9259904a3065591db7509eae"}
May 25 12:37:53.323945 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:37:53.323906    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh" podUID="5cc3c087-83c6-41d7-8e55-dc8756f9aff7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.46:8643/healthz\": dial tcp 10.134.0.46:8643: connect: connection refused"
May 25 12:37:53.330102 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:37:53.330068    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh" podUID="5cc3c087-83c6-41d7-8e55-dc8756f9aff7" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.46:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.46:8080: connect: connection refused"
May 25 12:37:58.324833 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:37:58.324794    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh" podUID="5cc3c087-83c6-41d7-8e55-dc8756f9aff7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.46:8643/healthz\": dial tcp 10.134.0.46:8643: connect: connection refused"
May 25 12:37:58.325198 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:37:58.324935    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh"
May 25 12:38:03.324697 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:38:03.324629    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh" podUID="5cc3c087-83c6-41d7-8e55-dc8756f9aff7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.46:8643/healthz\": dial tcp 10.134.0.46:8643: connect: connection refused"
May 25 12:38:03.330083 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:38:03.330055    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh" podUID="5cc3c087-83c6-41d7-8e55-dc8756f9aff7" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.46:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.46:8080: connect: connection refused"
May 25 12:38:07.578060 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:38:07.578033    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-2tqtm_00e6f79c-8f05-4330-aa16-1ada2ae9e9cc/console-operator/1.log"
May 25 12:38:07.581359 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:38:07.581338    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-2tqtm_00e6f79c-8f05-4330-aa16-1ada2ae9e9cc/console-operator/1.log"
May 25 12:38:08.323902 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:38:08.323859    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh" podUID="5cc3c087-83c6-41d7-8e55-dc8756f9aff7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.46:8643/healthz\": dial tcp 10.134.0.46:8643: connect: connection refused"
May 25 12:38:13.324053 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:38:13.324010    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh" podUID="5cc3c087-83c6-41d7-8e55-dc8756f9aff7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.46:8643/healthz\": dial tcp 10.134.0.46:8643: connect: connection refused"
May 25 12:38:13.330339 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:38:13.330306    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh" podUID="5cc3c087-83c6-41d7-8e55-dc8756f9aff7" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.46:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.46:8080: connect: connection refused"
May 25 12:38:13.330466 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:38:13.330436    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh"
May 25 12:38:18.199386 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:38:18.199360    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh"
May 25 12:38:18.309220 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:38:18.309192    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5cc3c087-83c6-41d7-8e55-dc8756f9aff7-kserve-provision-location\") pod \"5cc3c087-83c6-41d7-8e55-dc8756f9aff7\" (UID: \"5cc3c087-83c6-41d7-8e55-dc8756f9aff7\") "
May 25 12:38:18.309376 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:38:18.309237    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cc3c087-83c6-41d7-8e55-dc8756f9aff7-proxy-tls\") pod \"5cc3c087-83c6-41d7-8e55-dc8756f9aff7\" (UID: \"5cc3c087-83c6-41d7-8e55-dc8756f9aff7\") "
May 25 12:38:18.309376 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:38:18.309268    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cc3c087-83c6-41d7-8e55-dc8756f9aff7-isvc-sklearn-mcp-kube-rbac-proxy-sar-config\") pod \"5cc3c087-83c6-41d7-8e55-dc8756f9aff7\" (UID: \"5cc3c087-83c6-41d7-8e55-dc8756f9aff7\") "
May 25 12:38:18.309376 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:38:18.309346    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kttr2\" (UniqueName: \"kubernetes.io/projected/5cc3c087-83c6-41d7-8e55-dc8756f9aff7-kube-api-access-kttr2\") pod \"5cc3c087-83c6-41d7-8e55-dc8756f9aff7\" (UID: \"5cc3c087-83c6-41d7-8e55-dc8756f9aff7\") "
May 25 12:38:18.309633 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:38:18.309598    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cc3c087-83c6-41d7-8e55-dc8756f9aff7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5cc3c087-83c6-41d7-8e55-dc8756f9aff7" (UID: "5cc3c087-83c6-41d7-8e55-dc8756f9aff7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:38:18.309770 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:38:18.309701    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cc3c087-83c6-41d7-8e55-dc8756f9aff7-isvc-sklearn-mcp-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-mcp-kube-rbac-proxy-sar-config") pod "5cc3c087-83c6-41d7-8e55-dc8756f9aff7" (UID: "5cc3c087-83c6-41d7-8e55-dc8756f9aff7"). InnerVolumeSpecName "isvc-sklearn-mcp-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:38:18.311229 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:38:18.311204    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cc3c087-83c6-41d7-8e55-dc8756f9aff7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5cc3c087-83c6-41d7-8e55-dc8756f9aff7" (UID: "5cc3c087-83c6-41d7-8e55-dc8756f9aff7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:38:18.311315 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:38:18.311296    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cc3c087-83c6-41d7-8e55-dc8756f9aff7-kube-api-access-kttr2" (OuterVolumeSpecName: "kube-api-access-kttr2") pod "5cc3c087-83c6-41d7-8e55-dc8756f9aff7" (UID: "5cc3c087-83c6-41d7-8e55-dc8756f9aff7"). InnerVolumeSpecName "kube-api-access-kttr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:38:18.410691 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:38:18.410663    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kttr2\" (UniqueName: \"kubernetes.io/projected/5cc3c087-83c6-41d7-8e55-dc8756f9aff7-kube-api-access-kttr2\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:38:18.410691 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:38:18.410687    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5cc3c087-83c6-41d7-8e55-dc8756f9aff7-kserve-provision-location\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:38:18.410857 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:38:18.410697    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cc3c087-83c6-41d7-8e55-dc8756f9aff7-proxy-tls\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:38:18.410857 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:38:18.410707    2555 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cc3c087-83c6-41d7-8e55-dc8756f9aff7-isvc-sklearn-mcp-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:38:18.660977 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:38:18.660888    2555 generic.go:358] "Generic (PLEG): container finished" podID="5cc3c087-83c6-41d7-8e55-dc8756f9aff7" containerID="29b0a7e7a62db0ba83667d1e5e45c510ed3522335d834817d6bcac35e72c41e9" exitCode=137
May 25 12:38:18.661113 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:38:18.660979    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh"
May 25 12:38:18.661113 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:38:18.660970    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh" event={"ID":"5cc3c087-83c6-41d7-8e55-dc8756f9aff7","Type":"ContainerDied","Data":"29b0a7e7a62db0ba83667d1e5e45c510ed3522335d834817d6bcac35e72c41e9"}
May 25 12:38:18.661113 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:38:18.661088    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh" event={"ID":"5cc3c087-83c6-41d7-8e55-dc8756f9aff7","Type":"ContainerDied","Data":"e3601c397ff6e2a0034e2eb5f1b0ffc62006e47f3ce06a8db055661f681e823f"}
May 25 12:38:18.661113 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:38:18.661104    2555 scope.go:117] "RemoveContainer" containerID="ee52c4dad45e87c52d4fd29b9fd5b468a14cf17daf8b2f7087d753daaf1973be"
May 25 12:38:18.669381 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:38:18.669365    2555 scope.go:117] "RemoveContainer" containerID="29b0a7e7a62db0ba83667d1e5e45c510ed3522335d834817d6bcac35e72c41e9"
May 25 12:38:18.676243 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:38:18.676223    2555 scope.go:117] "RemoveContainer" containerID="daa1b331f39a6a172a2fe289f51879ebc78be64a9259904a3065591db7509eae"
May 25 12:38:18.680824 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:38:18.680805    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh"]
May 25 12:38:18.684146 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:38:18.684131    2555 scope.go:117] "RemoveContainer" containerID="3b255f167c26a210eec95267815135c92204cd8666dfe8f2819a3d7d98002289"
May 25 12:38:18.684367 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:38:18.684345    2555 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cchhh"]
May 25 12:38:18.690848 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:38:18.690831    2555 scope.go:117] "RemoveContainer" containerID="ee52c4dad45e87c52d4fd29b9fd5b468a14cf17daf8b2f7087d753daaf1973be"
May 25 12:38:18.691079 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:38:18.691062    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee52c4dad45e87c52d4fd29b9fd5b468a14cf17daf8b2f7087d753daaf1973be\": container with ID starting with ee52c4dad45e87c52d4fd29b9fd5b468a14cf17daf8b2f7087d753daaf1973be not found: ID does not exist" containerID="ee52c4dad45e87c52d4fd29b9fd5b468a14cf17daf8b2f7087d753daaf1973be"
May 25 12:38:18.691148 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:38:18.691091    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee52c4dad45e87c52d4fd29b9fd5b468a14cf17daf8b2f7087d753daaf1973be"} err="failed to get container status \"ee52c4dad45e87c52d4fd29b9fd5b468a14cf17daf8b2f7087d753daaf1973be\": rpc error: code = NotFound desc = could not find container \"ee52c4dad45e87c52d4fd29b9fd5b468a14cf17daf8b2f7087d753daaf1973be\": container with ID starting with ee52c4dad45e87c52d4fd29b9fd5b468a14cf17daf8b2f7087d753daaf1973be not found: ID does not exist"
May 25 12:38:18.691148 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:38:18.691116    2555 scope.go:117] "RemoveContainer" containerID="29b0a7e7a62db0ba83667d1e5e45c510ed3522335d834817d6bcac35e72c41e9"
May 25 12:38:18.691369 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:38:18.691351    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29b0a7e7a62db0ba83667d1e5e45c510ed3522335d834817d6bcac35e72c41e9\": container with ID starting with 29b0a7e7a62db0ba83667d1e5e45c510ed3522335d834817d6bcac35e72c41e9 not found: ID does not exist" containerID="29b0a7e7a62db0ba83667d1e5e45c510ed3522335d834817d6bcac35e72c41e9"
May 25 12:38:18.691416 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:38:18.691375    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29b0a7e7a62db0ba83667d1e5e45c510ed3522335d834817d6bcac35e72c41e9"} err="failed to get container status \"29b0a7e7a62db0ba83667d1e5e45c510ed3522335d834817d6bcac35e72c41e9\": rpc error: code = NotFound desc = could not find container \"29b0a7e7a62db0ba83667d1e5e45c510ed3522335d834817d6bcac35e72c41e9\": container with ID starting with 29b0a7e7a62db0ba83667d1e5e45c510ed3522335d834817d6bcac35e72c41e9 not found: ID does not exist"
May 25 12:38:18.691416 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:38:18.691390    2555 scope.go:117] "RemoveContainer" containerID="daa1b331f39a6a172a2fe289f51879ebc78be64a9259904a3065591db7509eae"
May 25 12:38:18.691594 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:38:18.691577    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daa1b331f39a6a172a2fe289f51879ebc78be64a9259904a3065591db7509eae\": container with ID starting with daa1b331f39a6a172a2fe289f51879ebc78be64a9259904a3065591db7509eae not found: ID does not exist" containerID="daa1b331f39a6a172a2fe289f51879ebc78be64a9259904a3065591db7509eae"
May 25 12:38:18.691636 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:38:18.691599    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daa1b331f39a6a172a2fe289f51879ebc78be64a9259904a3065591db7509eae"} err="failed to get container status \"daa1b331f39a6a172a2fe289f51879ebc78be64a9259904a3065591db7509eae\": rpc error: code = NotFound desc = could not find container \"daa1b331f39a6a172a2fe289f51879ebc78be64a9259904a3065591db7509eae\": container with ID starting with daa1b331f39a6a172a2fe289f51879ebc78be64a9259904a3065591db7509eae not found: ID does not exist"
May 25 12:38:18.691636 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:38:18.691613    2555 scope.go:117] "RemoveContainer" containerID="3b255f167c26a210eec95267815135c92204cd8666dfe8f2819a3d7d98002289"
May 25 12:38:18.691863 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:38:18.691845    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b255f167c26a210eec95267815135c92204cd8666dfe8f2819a3d7d98002289\": container with ID starting with 3b255f167c26a210eec95267815135c92204cd8666dfe8f2819a3d7d98002289 not found: ID does not exist" containerID="3b255f167c26a210eec95267815135c92204cd8666dfe8f2819a3d7d98002289"
May 25 12:38:18.691922 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:38:18.691869    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b255f167c26a210eec95267815135c92204cd8666dfe8f2819a3d7d98002289"} err="failed to get container status \"3b255f167c26a210eec95267815135c92204cd8666dfe8f2819a3d7d98002289\": rpc error: code = NotFound desc = could not find container \"3b255f167c26a210eec95267815135c92204cd8666dfe8f2819a3d7d98002289\": container with ID starting with 3b255f167c26a210eec95267815135c92204cd8666dfe8f2819a3d7d98002289 not found: ID does not exist"
May 25 12:38:19.600277 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:38:19.600233    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cc3c087-83c6-41d7-8e55-dc8756f9aff7" path="/var/lib/kubelet/pods/5cc3c087-83c6-41d7-8e55-dc8756f9aff7/volumes"
May 25 12:42:54.468078 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:42:54.468043    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ffv9n"]
May 25 12:42:54.468704 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:42:54.468515    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56a13387-cda8-4551-bf33-3a3e49be3217" containerName="kserve-container"
May 25 12:42:54.468704 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:42:54.468535    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="56a13387-cda8-4551-bf33-3a3e49be3217" containerName="kserve-container"
May 25 12:42:54.468704 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:42:54.468548    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cc3c087-83c6-41d7-8e55-dc8756f9aff7" containerName="kserve-agent"
May 25 12:42:54.468704 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:42:54.468556    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cc3c087-83c6-41d7-8e55-dc8756f9aff7" containerName="kserve-agent"
May 25 12:42:54.468704 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:42:54.468569    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56a13387-cda8-4551-bf33-3a3e49be3217" containerName="storage-initializer"
May 25 12:42:54.468704 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:42:54.468579    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="56a13387-cda8-4551-bf33-3a3e49be3217" containerName="storage-initializer"
May 25 12:42:54.468704 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:42:54.468596    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cc3c087-83c6-41d7-8e55-dc8756f9aff7" containerName="storage-initializer"
May 25 12:42:54.468704 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:42:54.468605    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cc3c087-83c6-41d7-8e55-dc8756f9aff7" containerName="storage-initializer"
May 25 12:42:54.468704 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:42:54.468621    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cc3c087-83c6-41d7-8e55-dc8756f9aff7" containerName="kserve-container"
May 25 12:42:54.468704 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:42:54.468629    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cc3c087-83c6-41d7-8e55-dc8756f9aff7" containerName="kserve-container"
May 25 12:42:54.468704 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:42:54.468657    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cc3c087-83c6-41d7-8e55-dc8756f9aff7" containerName="kube-rbac-proxy"
May 25 12:42:54.468704 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:42:54.468665    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cc3c087-83c6-41d7-8e55-dc8756f9aff7" containerName="kube-rbac-proxy"
May 25 12:42:54.468704 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:42:54.468677    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56a13387-cda8-4551-bf33-3a3e49be3217" containerName="kube-rbac-proxy"
May 25 12:42:54.468704 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:42:54.468685    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="56a13387-cda8-4551-bf33-3a3e49be3217" containerName="kube-rbac-proxy"
May 25 12:42:54.469431 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:42:54.468782    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="56a13387-cda8-4551-bf33-3a3e49be3217" containerName="kserve-container"
May 25 12:42:54.469431 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:42:54.468797    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="5cc3c087-83c6-41d7-8e55-dc8756f9aff7" containerName="kserve-agent"
May 25 12:42:54.469431 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:42:54.468808    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="56a13387-cda8-4551-bf33-3a3e49be3217" containerName="kube-rbac-proxy"
May 25 12:42:54.469431 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:42:54.468819    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="5cc3c087-83c6-41d7-8e55-dc8756f9aff7" containerName="kserve-container"
May 25 12:42:54.469431 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:42:54.468830    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="5cc3c087-83c6-41d7-8e55-dc8756f9aff7" containerName="kube-rbac-proxy"
May 25 12:42:54.472131 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:42:54.472107    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ffv9n"
May 25 12:42:54.473969 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:42:54.473946    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
May 25 12:42:54.474703 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:42:54.474682    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 25 12:42:54.474825 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:42:54.474705    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 25 12:42:54.474825 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:42:54.474793    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-predictor-serving-cert\""
May 25 12:42:54.474942 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:42:54.474824    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6qbkc\""
May 25 12:42:54.478885 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:42:54.478863    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ffv9n"]
May 25 12:42:54.648860 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:42:54.648831    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnh6v\" (UniqueName: \"kubernetes.io/projected/b60e0c71-303a-4160-bba1-d5199133de27-kube-api-access-jnh6v\") pod \"isvc-pmml-runtime-predictor-67bc544947-ffv9n\" (UID: \"b60e0c71-303a-4160-bba1-d5199133de27\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ffv9n"
May 25 12:42:54.649002 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:42:54.648866    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b60e0c71-303a-4160-bba1-d5199133de27-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-67bc544947-ffv9n\" (UID: \"b60e0c71-303a-4160-bba1-d5199133de27\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ffv9n"
May 25 12:42:54.649002 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:42:54.648920    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b60e0c71-303a-4160-bba1-d5199133de27-proxy-tls\") pod \"isvc-pmml-runtime-predictor-67bc544947-ffv9n\" (UID: \"b60e0c71-303a-4160-bba1-d5199133de27\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ffv9n"
May 25 12:42:54.649002 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:42:54.648950    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-pmml-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b60e0c71-303a-4160-bba1-d5199133de27-isvc-pmml-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-67bc544947-ffv9n\" (UID: \"b60e0c71-303a-4160-bba1-d5199133de27\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ffv9n"
May 25 12:42:54.749618 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:42:54.749552    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b60e0c71-303a-4160-bba1-d5199133de27-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-67bc544947-ffv9n\" (UID: \"b60e0c71-303a-4160-bba1-d5199133de27\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ffv9n"
May 25 12:42:54.749741 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:42:54.749668    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b60e0c71-303a-4160-bba1-d5199133de27-proxy-tls\") pod \"isvc-pmml-runtime-predictor-67bc544947-ffv9n\" (UID: \"b60e0c71-303a-4160-bba1-d5199133de27\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ffv9n"
May 25 12:42:54.749741 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:42:54.749729    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-pmml-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b60e0c71-303a-4160-bba1-d5199133de27-isvc-pmml-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-67bc544947-ffv9n\" (UID: \"b60e0c71-303a-4160-bba1-d5199133de27\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ffv9n"
May 25 12:42:54.749813 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:42:54.749764    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jnh6v\" (UniqueName: \"kubernetes.io/projected/b60e0c71-303a-4160-bba1-d5199133de27-kube-api-access-jnh6v\") pod \"isvc-pmml-runtime-predictor-67bc544947-ffv9n\" (UID: \"b60e0c71-303a-4160-bba1-d5199133de27\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ffv9n"
May 25 12:42:54.749972 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:42:54.749949    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b60e0c71-303a-4160-bba1-d5199133de27-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-67bc544947-ffv9n\" (UID: \"b60e0c71-303a-4160-bba1-d5199133de27\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ffv9n"
May 25 12:42:54.750410 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:42:54.750390    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-pmml-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b60e0c71-303a-4160-bba1-d5199133de27-isvc-pmml-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-67bc544947-ffv9n\" (UID: \"b60e0c71-303a-4160-bba1-d5199133de27\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ffv9n"
May 25 12:42:54.752186 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:42:54.752168    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b60e0c71-303a-4160-bba1-d5199133de27-proxy-tls\") pod \"isvc-pmml-runtime-predictor-67bc544947-ffv9n\" (UID: \"b60e0c71-303a-4160-bba1-d5199133de27\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ffv9n"
May 25 12:42:54.756819 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:42:54.756802    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnh6v\" (UniqueName: \"kubernetes.io/projected/b60e0c71-303a-4160-bba1-d5199133de27-kube-api-access-jnh6v\") pod \"isvc-pmml-runtime-predictor-67bc544947-ffv9n\" (UID: \"b60e0c71-303a-4160-bba1-d5199133de27\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ffv9n"
May 25 12:42:54.783770 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:42:54.783744    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ffv9n"
May 25 12:42:54.907314 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:42:54.907287    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ffv9n"]
May 25 12:42:54.909608 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:42:54.909578    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb60e0c71_303a_4160_bba1_d5199133de27.slice/crio-7e9629947f3fc850ff88ecef7890a4dc0eeb5a97d6c4d4e16db5e2e38b15c40d WatchSource:0}: Error finding container 7e9629947f3fc850ff88ecef7890a4dc0eeb5a97d6c4d4e16db5e2e38b15c40d: Status 404 returned error can't find the container with id 7e9629947f3fc850ff88ecef7890a4dc0eeb5a97d6c4d4e16db5e2e38b15c40d
May 25 12:42:54.911550 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:42:54.911531    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 12:42:55.568368 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:42:55.568334    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ffv9n" event={"ID":"b60e0c71-303a-4160-bba1-d5199133de27","Type":"ContainerStarted","Data":"cbc356eb9e76baeb05ea857c50e472cbed1b134c2ffc4559e2beab035a57fb8e"}
May 25 12:42:55.568368 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:42:55.568373    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ffv9n" event={"ID":"b60e0c71-303a-4160-bba1-d5199133de27","Type":"ContainerStarted","Data":"7e9629947f3fc850ff88ecef7890a4dc0eeb5a97d6c4d4e16db5e2e38b15c40d"}
May 25 12:42:59.583818 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:42:59.583780    2555 generic.go:358] "Generic (PLEG): container finished" podID="b60e0c71-303a-4160-bba1-d5199133de27" containerID="cbc356eb9e76baeb05ea857c50e472cbed1b134c2ffc4559e2beab035a57fb8e" exitCode=0
May 25 12:42:59.584184 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:42:59.583839    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ffv9n" event={"ID":"b60e0c71-303a-4160-bba1-d5199133de27","Type":"ContainerDied","Data":"cbc356eb9e76baeb05ea857c50e472cbed1b134c2ffc4559e2beab035a57fb8e"}
May 25 12:43:06.616796 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:43:06.616767    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ffv9n" event={"ID":"b60e0c71-303a-4160-bba1-d5199133de27","Type":"ContainerStarted","Data":"03cd07d609610e9b44ae8b29c473dac8a9a2dfd4df1e0b4f4216dc62c64b5581"}
May 25 12:43:07.607326 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:43:07.607300    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-2tqtm_00e6f79c-8f05-4330-aa16-1ada2ae9e9cc/console-operator/1.log"
May 25 12:43:07.609523 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:43:07.609500    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-2tqtm_00e6f79c-8f05-4330-aa16-1ada2ae9e9cc/console-operator/1.log"
May 25 12:43:07.620809 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:43:07.620779    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ffv9n" event={"ID":"b60e0c71-303a-4160-bba1-d5199133de27","Type":"ContainerStarted","Data":"f201f4437dea95a57fb410285288a7466136ad80d696b340b159e5195eb53533"}
May 25 12:43:07.621137 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:43:07.620838    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ffv9n"
May 25 12:43:07.636200 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:43:07.636161    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ffv9n" podStartSLOduration=6.706373545 podStartE2EDuration="13.63615019s" podCreationTimestamp="2026-05-25 12:42:54 +0000 UTC" firstStartedPulling="2026-05-25 12:42:59.585060899 +0000 UTC m=+1792.545111286" lastFinishedPulling="2026-05-25 12:43:06.514837547 +0000 UTC m=+1799.474887931" observedRunningTime="2026-05-25 12:43:07.634759467 +0000 UTC m=+1800.594809874" watchObservedRunningTime="2026-05-25 12:43:07.63615019 +0000 UTC m=+1800.596200596"
May 25 12:43:08.624577 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:43:08.624541    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ffv9n"
May 25 12:43:08.625945 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:43:08.625917    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ffv9n" podUID="b60e0c71-303a-4160-bba1-d5199133de27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
May 25 12:43:09.627897 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:43:09.627854    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ffv9n" podUID="b60e0c71-303a-4160-bba1-d5199133de27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
May 25 12:43:14.631854 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:43:14.631821    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ffv9n"
May 25 12:43:14.632444 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:43:14.632414    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ffv9n" podUID="b60e0c71-303a-4160-bba1-d5199133de27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
May 25 12:43:24.632585 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:43:24.632544    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ffv9n" podUID="b60e0c71-303a-4160-bba1-d5199133de27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
May 25 12:43:34.632772 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:43:34.632721    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ffv9n" podUID="b60e0c71-303a-4160-bba1-d5199133de27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
May 25 12:43:44.632872 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:43:44.632825    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ffv9n" podUID="b60e0c71-303a-4160-bba1-d5199133de27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
May 25 12:43:54.632956 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:43:54.632919    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ffv9n" podUID="b60e0c71-303a-4160-bba1-d5199133de27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
May 25 12:44:04.632862 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:44:04.632826    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ffv9n" podUID="b60e0c71-303a-4160-bba1-d5199133de27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
May 25 12:44:14.632860 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:44:14.632818    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ffv9n" podUID="b60e0c71-303a-4160-bba1-d5199133de27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
May 25 12:44:24.632887 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:44:24.632851    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ffv9n"
May 25 12:44:35.530000 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:44:35.529968    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ffv9n"]
May 25 12:44:35.530403 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:44:35.530264    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ffv9n" podUID="b60e0c71-303a-4160-bba1-d5199133de27" containerName="kserve-container" containerID="cri-o://03cd07d609610e9b44ae8b29c473dac8a9a2dfd4df1e0b4f4216dc62c64b5581" gracePeriod=30
May 25 12:44:35.530403 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:44:35.530297    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ffv9n" podUID="b60e0c71-303a-4160-bba1-d5199133de27" containerName="kube-rbac-proxy" containerID="cri-o://f201f4437dea95a57fb410285288a7466136ad80d696b340b159e5195eb53533" gracePeriod=30
May 25 12:44:35.900330 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:44:35.900245    2555 generic.go:358] "Generic (PLEG): container finished" podID="b60e0c71-303a-4160-bba1-d5199133de27" containerID="f201f4437dea95a57fb410285288a7466136ad80d696b340b159e5195eb53533" exitCode=2
May 25 12:44:35.900490 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:44:35.900330    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ffv9n" event={"ID":"b60e0c71-303a-4160-bba1-d5199133de27","Type":"ContainerDied","Data":"f201f4437dea95a57fb410285288a7466136ad80d696b340b159e5195eb53533"}
May 25 12:44:39.581429 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:44:39.581403    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ffv9n"
May 25 12:44:39.659534 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:44:39.659478    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b60e0c71-303a-4160-bba1-d5199133de27-proxy-tls\") pod \"b60e0c71-303a-4160-bba1-d5199133de27\" (UID: \"b60e0c71-303a-4160-bba1-d5199133de27\") "
May 25 12:44:39.659673 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:44:39.659551    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-pmml-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b60e0c71-303a-4160-bba1-d5199133de27-isvc-pmml-runtime-kube-rbac-proxy-sar-config\") pod \"b60e0c71-303a-4160-bba1-d5199133de27\" (UID: \"b60e0c71-303a-4160-bba1-d5199133de27\") "
May 25 12:44:39.659673 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:44:39.659594    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jnh6v\" (UniqueName: \"kubernetes.io/projected/b60e0c71-303a-4160-bba1-d5199133de27-kube-api-access-jnh6v\") pod \"b60e0c71-303a-4160-bba1-d5199133de27\" (UID: \"b60e0c71-303a-4160-bba1-d5199133de27\") "
May 25 12:44:39.659673 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:44:39.659626    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b60e0c71-303a-4160-bba1-d5199133de27-kserve-provision-location\") pod \"b60e0c71-303a-4160-bba1-d5199133de27\" (UID: \"b60e0c71-303a-4160-bba1-d5199133de27\") "
May 25 12:44:39.660030 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:44:39.659994    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b60e0c71-303a-4160-bba1-d5199133de27-isvc-pmml-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-pmml-runtime-kube-rbac-proxy-sar-config") pod "b60e0c71-303a-4160-bba1-d5199133de27" (UID: "b60e0c71-303a-4160-bba1-d5199133de27"). InnerVolumeSpecName "isvc-pmml-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:44:39.660154 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:44:39.660005    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b60e0c71-303a-4160-bba1-d5199133de27-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b60e0c71-303a-4160-bba1-d5199133de27" (UID: "b60e0c71-303a-4160-bba1-d5199133de27"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:44:39.661618 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:44:39.661595    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b60e0c71-303a-4160-bba1-d5199133de27-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b60e0c71-303a-4160-bba1-d5199133de27" (UID: "b60e0c71-303a-4160-bba1-d5199133de27"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:44:39.661716 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:44:39.661638    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b60e0c71-303a-4160-bba1-d5199133de27-kube-api-access-jnh6v" (OuterVolumeSpecName: "kube-api-access-jnh6v") pod "b60e0c71-303a-4160-bba1-d5199133de27" (UID: "b60e0c71-303a-4160-bba1-d5199133de27"). InnerVolumeSpecName "kube-api-access-jnh6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:44:39.760602 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:44:39.760580    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b60e0c71-303a-4160-bba1-d5199133de27-kserve-provision-location\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:44:39.760602 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:44:39.760601    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b60e0c71-303a-4160-bba1-d5199133de27-proxy-tls\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:44:39.760749 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:44:39.760611    2555 reconciler_common.go:299] "Volume detached for volume \"isvc-pmml-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b60e0c71-303a-4160-bba1-d5199133de27-isvc-pmml-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:44:39.760749 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:44:39.760622    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnh6v\" (UniqueName: \"kubernetes.io/projected/b60e0c71-303a-4160-bba1-d5199133de27-kube-api-access-jnh6v\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:44:39.917978 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:44:39.917912    2555 generic.go:358] "Generic (PLEG): container finished" podID="b60e0c71-303a-4160-bba1-d5199133de27" containerID="03cd07d609610e9b44ae8b29c473dac8a9a2dfd4df1e0b4f4216dc62c64b5581" exitCode=0
May 25 12:44:39.918063 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:44:39.917995    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ffv9n" event={"ID":"b60e0c71-303a-4160-bba1-d5199133de27","Type":"ContainerDied","Data":"03cd07d609610e9b44ae8b29c473dac8a9a2dfd4df1e0b4f4216dc62c64b5581"}
May 25 12:44:39.918063 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:44:39.918038    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ffv9n" event={"ID":"b60e0c71-303a-4160-bba1-d5199133de27","Type":"ContainerDied","Data":"7e9629947f3fc850ff88ecef7890a4dc0eeb5a97d6c4d4e16db5e2e38b15c40d"}
May 25 12:44:39.918063 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:44:39.918002    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ffv9n"
May 25 12:44:39.918063 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:44:39.918051    2555 scope.go:117] "RemoveContainer" containerID="f201f4437dea95a57fb410285288a7466136ad80d696b340b159e5195eb53533"
May 25 12:44:39.926192 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:44:39.926175    2555 scope.go:117] "RemoveContainer" containerID="03cd07d609610e9b44ae8b29c473dac8a9a2dfd4df1e0b4f4216dc62c64b5581"
May 25 12:44:39.933921 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:44:39.933906    2555 scope.go:117] "RemoveContainer" containerID="cbc356eb9e76baeb05ea857c50e472cbed1b134c2ffc4559e2beab035a57fb8e"
May 25 12:44:39.938038 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:44:39.938017    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ffv9n"]
May 25 12:44:39.940865 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:44:39.940850    2555 scope.go:117] "RemoveContainer" containerID="f201f4437dea95a57fb410285288a7466136ad80d696b340b159e5195eb53533"
May 25 12:44:39.941093 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:44:39.941077    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f201f4437dea95a57fb410285288a7466136ad80d696b340b159e5195eb53533\": container with ID starting with f201f4437dea95a57fb410285288a7466136ad80d696b340b159e5195eb53533 not found: ID does not exist" containerID="f201f4437dea95a57fb410285288a7466136ad80d696b340b159e5195eb53533"
May 25 12:44:39.941139 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:44:39.941100    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f201f4437dea95a57fb410285288a7466136ad80d696b340b159e5195eb53533"} err="failed to get container status \"f201f4437dea95a57fb410285288a7466136ad80d696b340b159e5195eb53533\": rpc error: code = NotFound desc = could not find container \"f201f4437dea95a57fb410285288a7466136ad80d696b340b159e5195eb53533\": container with ID starting with f201f4437dea95a57fb410285288a7466136ad80d696b340b159e5195eb53533 not found: ID does not exist"
May 25 12:44:39.941139 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:44:39.941122    2555 scope.go:117] "RemoveContainer" containerID="03cd07d609610e9b44ae8b29c473dac8a9a2dfd4df1e0b4f4216dc62c64b5581"
May 25 12:44:39.941327 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:44:39.941311    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03cd07d609610e9b44ae8b29c473dac8a9a2dfd4df1e0b4f4216dc62c64b5581\": container with ID starting with 03cd07d609610e9b44ae8b29c473dac8a9a2dfd4df1e0b4f4216dc62c64b5581 not found: ID does not exist" containerID="03cd07d609610e9b44ae8b29c473dac8a9a2dfd4df1e0b4f4216dc62c64b5581"
May 25 12:44:39.941403 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:44:39.941330    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03cd07d609610e9b44ae8b29c473dac8a9a2dfd4df1e0b4f4216dc62c64b5581"} err="failed to get container status \"03cd07d609610e9b44ae8b29c473dac8a9a2dfd4df1e0b4f4216dc62c64b5581\": rpc error: code = NotFound desc = could not find container \"03cd07d609610e9b44ae8b29c473dac8a9a2dfd4df1e0b4f4216dc62c64b5581\": container with ID starting with 03cd07d609610e9b44ae8b29c473dac8a9a2dfd4df1e0b4f4216dc62c64b5581 not found: ID does not exist"
May 25 12:44:39.941403 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:44:39.941344    2555 scope.go:117] "RemoveContainer" containerID="cbc356eb9e76baeb05ea857c50e472cbed1b134c2ffc4559e2beab035a57fb8e"
May 25 12:44:39.941588 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:44:39.941563    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbc356eb9e76baeb05ea857c50e472cbed1b134c2ffc4559e2beab035a57fb8e\": container with ID starting with cbc356eb9e76baeb05ea857c50e472cbed1b134c2ffc4559e2beab035a57fb8e not found: ID does not exist" containerID="cbc356eb9e76baeb05ea857c50e472cbed1b134c2ffc4559e2beab035a57fb8e"
May 25 12:44:39.941588 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:44:39.941590    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbc356eb9e76baeb05ea857c50e472cbed1b134c2ffc4559e2beab035a57fb8e"} err="failed to get container status \"cbc356eb9e76baeb05ea857c50e472cbed1b134c2ffc4559e2beab035a57fb8e\": rpc error: code = NotFound desc = could not find container \"cbc356eb9e76baeb05ea857c50e472cbed1b134c2ffc4559e2beab035a57fb8e\": container with ID starting with cbc356eb9e76baeb05ea857c50e472cbed1b134c2ffc4559e2beab035a57fb8e not found: ID does not exist"
May 25 12:44:39.943682 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:44:39.943637    2555 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-ffv9n"]
May 25 12:44:41.598600 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:44:41.598565    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b60e0c71-303a-4160-bba1-d5199133de27" path="/var/lib/kubelet/pods/b60e0c71-303a-4160-bba1-d5199133de27/volumes"
May 25 12:46:16.879878 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:46:16.879794    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-f32ae1-predictor-7f478b4d68-nv59n"]
May 25 12:46:16.880309 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:46:16.880104    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b60e0c71-303a-4160-bba1-d5199133de27" containerName="storage-initializer"
May 25 12:46:16.880309 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:46:16.880115    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b60e0c71-303a-4160-bba1-d5199133de27" containerName="storage-initializer"
May 25 12:46:16.880309 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:46:16.880127    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b60e0c71-303a-4160-bba1-d5199133de27" containerName="kserve-container"
May 25 12:46:16.880309 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:46:16.880132    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b60e0c71-303a-4160-bba1-d5199133de27" containerName="kserve-container"
May 25 12:46:16.880309 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:46:16.880145    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b60e0c71-303a-4160-bba1-d5199133de27" containerName="kube-rbac-proxy"
May 25 12:46:16.880309 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:46:16.880150    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b60e0c71-303a-4160-bba1-d5199133de27" containerName="kube-rbac-proxy"
May 25 12:46:16.880309 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:46:16.880207    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="b60e0c71-303a-4160-bba1-d5199133de27" containerName="kserve-container"
May 25 12:46:16.880309 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:46:16.880218    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="b60e0c71-303a-4160-bba1-d5199133de27" containerName="kube-rbac-proxy"
May 25 12:46:16.883195 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:46:16.883178    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-f32ae1-predictor-7f478b4d68-nv59n"
May 25 12:46:16.885165 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:46:16.885134    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 25 12:46:16.885503 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:46:16.885477    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 25 12:46:16.885503 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:46:16.885489    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6qbkc\""
May 25 12:46:16.885662 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:46:16.885581    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-f32ae1-predictor-serving-cert\""
May 25 12:46:16.885662 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:46:16.885620    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-f32ae1-kube-rbac-proxy-sar-config\""
May 25 12:46:16.893092 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:46:16.893072    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-f32ae1-predictor-7f478b4d68-nv59n"]
May 25 12:46:16.995996 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:46:16.995967    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61ae41ef-8448-4eba-aec7-92c15d8e0a39-proxy-tls\") pod \"isvc-primary-f32ae1-predictor-7f478b4d68-nv59n\" (UID: \"61ae41ef-8448-4eba-aec7-92c15d8e0a39\") " pod="kserve-ci-e2e-test/isvc-primary-f32ae1-predictor-7f478b4d68-nv59n"
May 25 12:46:16.996143 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:46:16.996018    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61ae41ef-8448-4eba-aec7-92c15d8e0a39-kserve-provision-location\") pod \"isvc-primary-f32ae1-predictor-7f478b4d68-nv59n\" (UID: \"61ae41ef-8448-4eba-aec7-92c15d8e0a39\") " pod="kserve-ci-e2e-test/isvc-primary-f32ae1-predictor-7f478b4d68-nv59n"
May 25 12:46:16.996143 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:46:16.996108    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnbdm\" (UniqueName: \"kubernetes.io/projected/61ae41ef-8448-4eba-aec7-92c15d8e0a39-kube-api-access-dnbdm\") pod \"isvc-primary-f32ae1-predictor-7f478b4d68-nv59n\" (UID: \"61ae41ef-8448-4eba-aec7-92c15d8e0a39\") " pod="kserve-ci-e2e-test/isvc-primary-f32ae1-predictor-7f478b4d68-nv59n"
May 25 12:46:16.996234 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:46:16.996156    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-primary-f32ae1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/61ae41ef-8448-4eba-aec7-92c15d8e0a39-isvc-primary-f32ae1-kube-rbac-proxy-sar-config\") pod \"isvc-primary-f32ae1-predictor-7f478b4d68-nv59n\" (UID: \"61ae41ef-8448-4eba-aec7-92c15d8e0a39\") " pod="kserve-ci-e2e-test/isvc-primary-f32ae1-predictor-7f478b4d68-nv59n"
May 25 12:46:17.096680 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:46:17.096632    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61ae41ef-8448-4eba-aec7-92c15d8e0a39-kserve-provision-location\") pod \"isvc-primary-f32ae1-predictor-7f478b4d68-nv59n\" (UID: \"61ae41ef-8448-4eba-aec7-92c15d8e0a39\") " pod="kserve-ci-e2e-test/isvc-primary-f32ae1-predictor-7f478b4d68-nv59n"
May 25 12:46:17.096820 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:46:17.096701    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dnbdm\" (UniqueName: \"kubernetes.io/projected/61ae41ef-8448-4eba-aec7-92c15d8e0a39-kube-api-access-dnbdm\") pod \"isvc-primary-f32ae1-predictor-7f478b4d68-nv59n\" (UID: \"61ae41ef-8448-4eba-aec7-92c15d8e0a39\") " pod="kserve-ci-e2e-test/isvc-primary-f32ae1-predictor-7f478b4d68-nv59n"
May 25 12:46:17.096820 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:46:17.096733    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-primary-f32ae1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/61ae41ef-8448-4eba-aec7-92c15d8e0a39-isvc-primary-f32ae1-kube-rbac-proxy-sar-config\") pod \"isvc-primary-f32ae1-predictor-7f478b4d68-nv59n\" (UID: \"61ae41ef-8448-4eba-aec7-92c15d8e0a39\") " pod="kserve-ci-e2e-test/isvc-primary-f32ae1-predictor-7f478b4d68-nv59n"
May 25 12:46:17.096820 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:46:17.096784    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61ae41ef-8448-4eba-aec7-92c15d8e0a39-proxy-tls\") pod \"isvc-primary-f32ae1-predictor-7f478b4d68-nv59n\" (UID: \"61ae41ef-8448-4eba-aec7-92c15d8e0a39\") " pod="kserve-ci-e2e-test/isvc-primary-f32ae1-predictor-7f478b4d68-nv59n"
May 25 12:46:17.097126 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:46:17.097099    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61ae41ef-8448-4eba-aec7-92c15d8e0a39-kserve-provision-location\") pod \"isvc-primary-f32ae1-predictor-7f478b4d68-nv59n\" (UID: \"61ae41ef-8448-4eba-aec7-92c15d8e0a39\") " pod="kserve-ci-e2e-test/isvc-primary-f32ae1-predictor-7f478b4d68-nv59n"
May 25 12:46:17.097357 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:46:17.097338    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-primary-f32ae1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/61ae41ef-8448-4eba-aec7-92c15d8e0a39-isvc-primary-f32ae1-kube-rbac-proxy-sar-config\") pod \"isvc-primary-f32ae1-predictor-7f478b4d68-nv59n\" (UID: \"61ae41ef-8448-4eba-aec7-92c15d8e0a39\") " pod="kserve-ci-e2e-test/isvc-primary-f32ae1-predictor-7f478b4d68-nv59n"
May 25 12:46:17.099213 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:46:17.099190    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61ae41ef-8448-4eba-aec7-92c15d8e0a39-proxy-tls\") pod \"isvc-primary-f32ae1-predictor-7f478b4d68-nv59n\" (UID: \"61ae41ef-8448-4eba-aec7-92c15d8e0a39\") " pod="kserve-ci-e2e-test/isvc-primary-f32ae1-predictor-7f478b4d68-nv59n"
May 25 12:46:17.104207 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:46:17.104180    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnbdm\" (UniqueName: \"kubernetes.io/projected/61ae41ef-8448-4eba-aec7-92c15d8e0a39-kube-api-access-dnbdm\") pod \"isvc-primary-f32ae1-predictor-7f478b4d68-nv59n\" (UID: \"61ae41ef-8448-4eba-aec7-92c15d8e0a39\") " pod="kserve-ci-e2e-test/isvc-primary-f32ae1-predictor-7f478b4d68-nv59n"
May 25 12:46:17.194489 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:46:17.194455    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-f32ae1-predictor-7f478b4d68-nv59n"
May 25 12:46:17.323150 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:46:17.323123    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-f32ae1-predictor-7f478b4d68-nv59n"]
May 25 12:46:17.324758 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:46:17.324720    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61ae41ef_8448_4eba_aec7_92c15d8e0a39.slice/crio-3bd3f131110f726641b11195ccb7bb2a90b9be65dece562d6972911527d1b74f WatchSource:0}: Error finding container 3bd3f131110f726641b11195ccb7bb2a90b9be65dece562d6972911527d1b74f: Status 404 returned error can't find the container with id 3bd3f131110f726641b11195ccb7bb2a90b9be65dece562d6972911527d1b74f
May 25 12:46:18.241757 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:46:18.241721    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-f32ae1-predictor-7f478b4d68-nv59n" event={"ID":"61ae41ef-8448-4eba-aec7-92c15d8e0a39","Type":"ContainerStarted","Data":"466c7b02dad80b3e8d6f107e36987da8366f1d83c9a825bb7820da6bde43c60c"}
May 25 12:46:18.242116 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:46:18.241765    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-f32ae1-predictor-7f478b4d68-nv59n" event={"ID":"61ae41ef-8448-4eba-aec7-92c15d8e0a39","Type":"ContainerStarted","Data":"3bd3f131110f726641b11195ccb7bb2a90b9be65dece562d6972911527d1b74f"}
May 25 12:46:21.023958 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:46:21.023927    2555 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61ae41ef_8448_4eba_aec7_92c15d8e0a39.slice/crio-conmon-466c7b02dad80b3e8d6f107e36987da8366f1d83c9a825bb7820da6bde43c60c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61ae41ef_8448_4eba_aec7_92c15d8e0a39.slice/crio-466c7b02dad80b3e8d6f107e36987da8366f1d83c9a825bb7820da6bde43c60c.scope\": RecentStats: unable to find data in memory cache]"
May 25 12:46:21.252274 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:46:21.252195    2555 generic.go:358] "Generic (PLEG): container finished" podID="61ae41ef-8448-4eba-aec7-92c15d8e0a39" containerID="466c7b02dad80b3e8d6f107e36987da8366f1d83c9a825bb7820da6bde43c60c" exitCode=0
May 25 12:46:21.252408 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:46:21.252268    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-f32ae1-predictor-7f478b4d68-nv59n" event={"ID":"61ae41ef-8448-4eba-aec7-92c15d8e0a39","Type":"ContainerDied","Data":"466c7b02dad80b3e8d6f107e36987da8366f1d83c9a825bb7820da6bde43c60c"}
May 25 12:46:22.258145 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:46:22.258114    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-f32ae1-predictor-7f478b4d68-nv59n" event={"ID":"61ae41ef-8448-4eba-aec7-92c15d8e0a39","Type":"ContainerStarted","Data":"38a07528613ebc025d65cd7a1c5cb3e40398ca5a0677925e89c6d2a18b508197"}
May 25 12:46:22.258145 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:46:22.258151    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-f32ae1-predictor-7f478b4d68-nv59n" event={"ID":"61ae41ef-8448-4eba-aec7-92c15d8e0a39","Type":"ContainerStarted","Data":"3279db93ade2fadf71a457de74396c15118ecf8c57f9826a8494a096b727c73b"}
May 25 12:46:22.258567 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:46:22.258431    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-f32ae1-predictor-7f478b4d68-nv59n"
May 25 12:46:22.258625 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:46:22.258562    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-f32ae1-predictor-7f478b4d68-nv59n"
May 25 12:46:22.259712 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:46:22.259686    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f32ae1-predictor-7f478b4d68-nv59n" podUID="61ae41ef-8448-4eba-aec7-92c15d8e0a39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
May 25 12:46:22.275505 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:46:22.275466    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-f32ae1-predictor-7f478b4d68-nv59n" podStartSLOduration=6.275455459 podStartE2EDuration="6.275455459s" podCreationTimestamp="2026-05-25 12:46:16 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:46:22.273402503 +0000 UTC m=+1995.233452909" watchObservedRunningTime="2026-05-25 12:46:22.275455459 +0000 UTC m=+1995.235505865"
May 25 12:46:23.261791 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:46:23.261749    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f32ae1-predictor-7f478b4d68-nv59n" podUID="61ae41ef-8448-4eba-aec7-92c15d8e0a39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
May 25 12:46:28.266706 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:46:28.266676    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-f32ae1-predictor-7f478b4d68-nv59n"
May 25 12:46:28.267255 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:46:28.267227    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f32ae1-predictor-7f478b4d68-nv59n" podUID="61ae41ef-8448-4eba-aec7-92c15d8e0a39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
May 25 12:46:38.268051 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:46:38.268011    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f32ae1-predictor-7f478b4d68-nv59n" podUID="61ae41ef-8448-4eba-aec7-92c15d8e0a39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
May 25 12:46:48.268019 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:46:48.267981    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f32ae1-predictor-7f478b4d68-nv59n" podUID="61ae41ef-8448-4eba-aec7-92c15d8e0a39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
May 25 12:46:58.267438 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:46:58.267393    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f32ae1-predictor-7f478b4d68-nv59n" podUID="61ae41ef-8448-4eba-aec7-92c15d8e0a39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
May 25 12:47:08.267358 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:08.267317    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f32ae1-predictor-7f478b4d68-nv59n" podUID="61ae41ef-8448-4eba-aec7-92c15d8e0a39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
May 25 12:47:18.267291 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:18.267239    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f32ae1-predictor-7f478b4d68-nv59n" podUID="61ae41ef-8448-4eba-aec7-92c15d8e0a39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
May 25 12:47:28.268749 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:28.268631    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-f32ae1-predictor-7f478b4d68-nv59n"
May 25 12:47:37.009305 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:37.009273    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-f32ae1-predictor-546d9cc87-dtmgp"]
May 25 12:47:37.013038 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:37.013016    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-f32ae1-predictor-546d9cc87-dtmgp"
May 25 12:47:37.014847 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:37.014824    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-f32ae1-predictor-serving-cert\""
May 25 12:47:37.014979 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:37.014842    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
May 25 12:47:37.014979 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:37.014857    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-f32ae1-dockercfg-wnnkn\""
May 25 12:47:37.014979 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:37.014907    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-f32ae1-kube-rbac-proxy-sar-config\""
May 25 12:47:37.015173 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:37.015159    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-f32ae1\""
May 25 12:47:37.023464 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:37.023441    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-f32ae1-predictor-546d9cc87-dtmgp"]
May 25 12:47:37.115809 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:37.115784    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53-proxy-tls\") pod \"isvc-secondary-f32ae1-predictor-546d9cc87-dtmgp\" (UID: \"47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53\") " pod="kserve-ci-e2e-test/isvc-secondary-f32ae1-predictor-546d9cc87-dtmgp"
May 25 12:47:37.115975 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:37.115816    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8rqc\" (UniqueName: \"kubernetes.io/projected/47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53-kube-api-access-h8rqc\") pod \"isvc-secondary-f32ae1-predictor-546d9cc87-dtmgp\" (UID: \"47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53\") " pod="kserve-ci-e2e-test/isvc-secondary-f32ae1-predictor-546d9cc87-dtmgp"
May 25 12:47:37.115975 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:37.115850    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53-cabundle-cert\") pod \"isvc-secondary-f32ae1-predictor-546d9cc87-dtmgp\" (UID: \"47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53\") " pod="kserve-ci-e2e-test/isvc-secondary-f32ae1-predictor-546d9cc87-dtmgp"
May 25 12:47:37.115975 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:37.115918    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-secondary-f32ae1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53-isvc-secondary-f32ae1-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-f32ae1-predictor-546d9cc87-dtmgp\" (UID: \"47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53\") " pod="kserve-ci-e2e-test/isvc-secondary-f32ae1-predictor-546d9cc87-dtmgp"
May 25 12:47:37.116111 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:37.116011    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53-kserve-provision-location\") pod \"isvc-secondary-f32ae1-predictor-546d9cc87-dtmgp\" (UID: \"47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53\") " pod="kserve-ci-e2e-test/isvc-secondary-f32ae1-predictor-546d9cc87-dtmgp"
May 25 12:47:37.216457 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:37.216396    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53-proxy-tls\") pod \"isvc-secondary-f32ae1-predictor-546d9cc87-dtmgp\" (UID: \"47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53\") " pod="kserve-ci-e2e-test/isvc-secondary-f32ae1-predictor-546d9cc87-dtmgp"
May 25 12:47:37.216619 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:37.216480    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-h8rqc\" (UniqueName: \"kubernetes.io/projected/47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53-kube-api-access-h8rqc\") pod \"isvc-secondary-f32ae1-predictor-546d9cc87-dtmgp\" (UID: \"47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53\") " pod="kserve-ci-e2e-test/isvc-secondary-f32ae1-predictor-546d9cc87-dtmgp"
May 25 12:47:37.216619 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:37.216527    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53-cabundle-cert\") pod \"isvc-secondary-f32ae1-predictor-546d9cc87-dtmgp\" (UID: \"47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53\") " pod="kserve-ci-e2e-test/isvc-secondary-f32ae1-predictor-546d9cc87-dtmgp"
May 25 12:47:37.216619 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:37.216553    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-secondary-f32ae1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53-isvc-secondary-f32ae1-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-f32ae1-predictor-546d9cc87-dtmgp\" (UID: \"47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53\") " pod="kserve-ci-e2e-test/isvc-secondary-f32ae1-predictor-546d9cc87-dtmgp"
May 25 12:47:37.216619 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:37.216603    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53-kserve-provision-location\") pod \"isvc-secondary-f32ae1-predictor-546d9cc87-dtmgp\" (UID: \"47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53\") " pod="kserve-ci-e2e-test/isvc-secondary-f32ae1-predictor-546d9cc87-dtmgp"
May 25 12:47:37.217028 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:37.217006    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53-kserve-provision-location\") pod \"isvc-secondary-f32ae1-predictor-546d9cc87-dtmgp\" (UID: \"47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53\") " pod="kserve-ci-e2e-test/isvc-secondary-f32ae1-predictor-546d9cc87-dtmgp"
May 25 12:47:37.217275 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:37.217254    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53-cabundle-cert\") pod \"isvc-secondary-f32ae1-predictor-546d9cc87-dtmgp\" (UID: \"47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53\") " pod="kserve-ci-e2e-test/isvc-secondary-f32ae1-predictor-546d9cc87-dtmgp"
May 25 12:47:37.217340 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:37.217254    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-secondary-f32ae1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53-isvc-secondary-f32ae1-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-f32ae1-predictor-546d9cc87-dtmgp\" (UID: \"47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53\") " pod="kserve-ci-e2e-test/isvc-secondary-f32ae1-predictor-546d9cc87-dtmgp"
May 25 12:47:37.218773 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:37.218752    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53-proxy-tls\") pod \"isvc-secondary-f32ae1-predictor-546d9cc87-dtmgp\" (UID: \"47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53\") " pod="kserve-ci-e2e-test/isvc-secondary-f32ae1-predictor-546d9cc87-dtmgp"
May 25 12:47:37.223080 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:37.223059    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8rqc\" (UniqueName: \"kubernetes.io/projected/47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53-kube-api-access-h8rqc\") pod \"isvc-secondary-f32ae1-predictor-546d9cc87-dtmgp\" (UID: \"47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53\") " pod="kserve-ci-e2e-test/isvc-secondary-f32ae1-predictor-546d9cc87-dtmgp"
May 25 12:47:37.325215 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:37.325147    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-f32ae1-predictor-546d9cc87-dtmgp"
May 25 12:47:37.455618 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:37.455573    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-f32ae1-predictor-546d9cc87-dtmgp"]
May 25 12:47:37.457932 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:47:37.457901    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47d40cf7_5ec2_41e1_9ec4_7a961ffc4a53.slice/crio-722e712e087e3e8ca51e7543bb25e1b0645ec9a0f01e52676297bb188444dfef WatchSource:0}: Error finding container 722e712e087e3e8ca51e7543bb25e1b0645ec9a0f01e52676297bb188444dfef: Status 404 returned error can't find the container with id 722e712e087e3e8ca51e7543bb25e1b0645ec9a0f01e52676297bb188444dfef
May 25 12:47:37.505705 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:37.505658    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-f32ae1-predictor-546d9cc87-dtmgp" event={"ID":"47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53","Type":"ContainerStarted","Data":"722e712e087e3e8ca51e7543bb25e1b0645ec9a0f01e52676297bb188444dfef"}
May 25 12:47:38.510789 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:38.510750    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-f32ae1-predictor-546d9cc87-dtmgp" event={"ID":"47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53","Type":"ContainerStarted","Data":"f86002d6ed0bedbab209c3b0e26ceade7424110297090416d1ca5cb4d363eaf4"}
May 25 12:47:42.526012 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:42.525935    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-f32ae1-predictor-546d9cc87-dtmgp_47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53/storage-initializer/0.log"
May 25 12:47:42.526012 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:42.525977    2555 generic.go:358] "Generic (PLEG): container finished" podID="47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53" containerID="f86002d6ed0bedbab209c3b0e26ceade7424110297090416d1ca5cb4d363eaf4" exitCode=1
May 25 12:47:42.526378 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:42.526026    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-f32ae1-predictor-546d9cc87-dtmgp" event={"ID":"47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53","Type":"ContainerDied","Data":"f86002d6ed0bedbab209c3b0e26ceade7424110297090416d1ca5cb4d363eaf4"}
May 25 12:47:43.531085 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:43.531057    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-f32ae1-predictor-546d9cc87-dtmgp_47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53/storage-initializer/0.log"
May 25 12:47:43.531460 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:43.531186    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-f32ae1-predictor-546d9cc87-dtmgp" event={"ID":"47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53","Type":"ContainerStarted","Data":"6c3462969edb3e0fb045c19808cc0140a4a4caefb307728a50e9608e09ab7bf9"}
May 25 12:47:47.555991 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:47.555967    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-f32ae1-predictor-546d9cc87-dtmgp_47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53/storage-initializer/1.log"
May 25 12:47:47.556354 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:47.556326    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-f32ae1-predictor-546d9cc87-dtmgp_47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53/storage-initializer/0.log"
May 25 12:47:47.556408 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:47.556357    2555 generic.go:358] "Generic (PLEG): container finished" podID="47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53" containerID="6c3462969edb3e0fb045c19808cc0140a4a4caefb307728a50e9608e09ab7bf9" exitCode=1
May 25 12:47:47.556463 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:47.556430    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-f32ae1-predictor-546d9cc87-dtmgp" event={"ID":"47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53","Type":"ContainerDied","Data":"6c3462969edb3e0fb045c19808cc0140a4a4caefb307728a50e9608e09ab7bf9"}
May 25 12:47:47.556510 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:47.556472    2555 scope.go:117] "RemoveContainer" containerID="f86002d6ed0bedbab209c3b0e26ceade7424110297090416d1ca5cb4d363eaf4"
May 25 12:47:47.556891 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:47.556873    2555 scope.go:117] "RemoveContainer" containerID="f86002d6ed0bedbab209c3b0e26ceade7424110297090416d1ca5cb4d363eaf4"
May 25 12:47:47.566808 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:47:47.566784    2555 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-f32ae1-predictor-546d9cc87-dtmgp_kserve-ci-e2e-test_47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53_0 in pod sandbox 722e712e087e3e8ca51e7543bb25e1b0645ec9a0f01e52676297bb188444dfef from index: no such id: 'f86002d6ed0bedbab209c3b0e26ceade7424110297090416d1ca5cb4d363eaf4'" containerID="f86002d6ed0bedbab209c3b0e26ceade7424110297090416d1ca5cb4d363eaf4"
May 25 12:47:47.566885 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:47:47.566826    2555 kuberuntime_container.go:951] "Unhandled Error" err="failed to remove pod init container \"storage-initializer\": rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-f32ae1-predictor-546d9cc87-dtmgp_kserve-ci-e2e-test_47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53_0 in pod sandbox 722e712e087e3e8ca51e7543bb25e1b0645ec9a0f01e52676297bb188444dfef from index: no such id: 'f86002d6ed0bedbab209c3b0e26ceade7424110297090416d1ca5cb4d363eaf4'; Skipping pod \"isvc-secondary-f32ae1-predictor-546d9cc87-dtmgp_kserve-ci-e2e-test(47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53)\"" logger="UnhandledError"
May 25 12:47:47.568138 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:47:47.568117    2555 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-f32ae1-predictor-546d9cc87-dtmgp_kserve-ci-e2e-test(47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53)\"" pod="kserve-ci-e2e-test/isvc-secondary-f32ae1-predictor-546d9cc87-dtmgp" podUID="47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53"
May 25 12:47:48.561293 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:48.561261    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-f32ae1-predictor-546d9cc87-dtmgp_47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53/storage-initializer/1.log"
May 25 12:47:53.041278 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:53.041246    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-f32ae1-predictor-546d9cc87-dtmgp"]
May 25 12:47:53.093509 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:53.093475    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-f32ae1-predictor-7f478b4d68-nv59n"]
May 25 12:47:53.093979 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:53.093940    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-f32ae1-predictor-7f478b4d68-nv59n" podUID="61ae41ef-8448-4eba-aec7-92c15d8e0a39" containerName="kserve-container" containerID="cri-o://3279db93ade2fadf71a457de74396c15118ecf8c57f9826a8494a096b727c73b" gracePeriod=30
May 25 12:47:53.094135 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:53.093985    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-f32ae1-predictor-7f478b4d68-nv59n" podUID="61ae41ef-8448-4eba-aec7-92c15d8e0a39" containerName="kube-rbac-proxy" containerID="cri-o://38a07528613ebc025d65cd7a1c5cb3e40398ca5a0677925e89c6d2a18b508197" gracePeriod=30
May 25 12:47:53.194079 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:53.194041    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-7f3b77-predictor-54ffd4dd7f-tkljc"]
May 25 12:47:53.198948 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:53.198929    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-7f3b77-predictor-54ffd4dd7f-tkljc"
May 25 12:47:53.200817 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:53.200795    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-7f3b77-dockercfg-mdl7m\""
May 25 12:47:53.200975 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:53.200796    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-7f3b77-kube-rbac-proxy-sar-config\""
May 25 12:47:53.201083 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:53.200816    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-7f3b77-predictor-serving-cert\""
May 25 12:47:53.201083 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:53.200953    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-7f3b77\""
May 25 12:47:53.206993 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:53.206971    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-7f3b77-predictor-54ffd4dd7f-tkljc"]
May 25 12:47:53.216529 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:53.216513    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-f32ae1-predictor-546d9cc87-dtmgp_47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53/storage-initializer/1.log"
May 25 12:47:53.216617 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:53.216568    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-f32ae1-predictor-546d9cc87-dtmgp"
May 25 12:47:53.237494 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:53.237470    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8rqc\" (UniqueName: \"kubernetes.io/projected/47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53-kube-api-access-h8rqc\") pod \"47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53\" (UID: \"47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53\") "
May 25 12:47:53.237598 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:53.237505    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-secondary-f32ae1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53-isvc-secondary-f32ae1-kube-rbac-proxy-sar-config\") pod \"47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53\" (UID: \"47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53\") "
May 25 12:47:53.237676 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:53.237618    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53-proxy-tls\") pod \"47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53\" (UID: \"47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53\") "
May 25 12:47:53.237743 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:53.237692    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53-kserve-provision-location\") pod \"47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53\" (UID: \"47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53\") "
May 25 12:47:53.237743 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:53.237726    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53-cabundle-cert\") pod \"47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53\" (UID: \"47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53\") "
May 25 12:47:53.237915 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:53.237895    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6af932c-dd1a-4e2e-9ac5-8f686b4badea-kserve-provision-location\") pod \"isvc-init-fail-7f3b77-predictor-54ffd4dd7f-tkljc\" (UID: \"b6af932c-dd1a-4e2e-9ac5-8f686b4badea\") " pod="kserve-ci-e2e-test/isvc-init-fail-7f3b77-predictor-54ffd4dd7f-tkljc"
May 25 12:47:53.237993 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:53.237935    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6af932c-dd1a-4e2e-9ac5-8f686b4badea-proxy-tls\") pod \"isvc-init-fail-7f3b77-predictor-54ffd4dd7f-tkljc\" (UID: \"b6af932c-dd1a-4e2e-9ac5-8f686b4badea\") " pod="kserve-ci-e2e-test/isvc-init-fail-7f3b77-predictor-54ffd4dd7f-tkljc"
May 25 12:47:53.237993 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:53.237967    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-init-fail-7f3b77-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6af932c-dd1a-4e2e-9ac5-8f686b4badea-isvc-init-fail-7f3b77-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-7f3b77-predictor-54ffd4dd7f-tkljc\" (UID: \"b6af932c-dd1a-4e2e-9ac5-8f686b4badea\") " pod="kserve-ci-e2e-test/isvc-init-fail-7f3b77-predictor-54ffd4dd7f-tkljc"
May 25 12:47:53.237993 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:53.237951    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53" (UID: "47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:47:53.238128 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:53.237990    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53-isvc-secondary-f32ae1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-secondary-f32ae1-kube-rbac-proxy-sar-config") pod "47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53" (UID: "47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53"). InnerVolumeSpecName "isvc-secondary-f32ae1-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:47:53.238128 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:53.238077    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6af932c-dd1a-4e2e-9ac5-8f686b4badea-cabundle-cert\") pod \"isvc-init-fail-7f3b77-predictor-54ffd4dd7f-tkljc\" (UID: \"b6af932c-dd1a-4e2e-9ac5-8f686b4badea\") " pod="kserve-ci-e2e-test/isvc-init-fail-7f3b77-predictor-54ffd4dd7f-tkljc"
May 25 12:47:53.238128 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:53.238093    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53" (UID: "47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:47:53.238238 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:53.238145    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w4z8\" (UniqueName: \"kubernetes.io/projected/b6af932c-dd1a-4e2e-9ac5-8f686b4badea-kube-api-access-4w4z8\") pod \"isvc-init-fail-7f3b77-predictor-54ffd4dd7f-tkljc\" (UID: \"b6af932c-dd1a-4e2e-9ac5-8f686b4badea\") " pod="kserve-ci-e2e-test/isvc-init-fail-7f3b77-predictor-54ffd4dd7f-tkljc"
May 25 12:47:53.238238 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:53.238190    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53-kserve-provision-location\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:47:53.238238 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:53.238201    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53-cabundle-cert\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:47:53.238238 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:53.238210    2555 reconciler_common.go:299] "Volume detached for volume \"isvc-secondary-f32ae1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53-isvc-secondary-f32ae1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:47:53.239428 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:53.239411    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53-kube-api-access-h8rqc" (OuterVolumeSpecName: "kube-api-access-h8rqc") pod "47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53" (UID: "47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53"). InnerVolumeSpecName "kube-api-access-h8rqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:47:53.239529 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:53.239517    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53" (UID: "47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:47:53.262711 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:53.262683    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f32ae1-predictor-7f478b4d68-nv59n" podUID="61ae41ef-8448-4eba-aec7-92c15d8e0a39" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.48:8643/healthz\": dial tcp 10.134.0.48:8643: connect: connection refused"
May 25 12:47:53.339599 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:53.339533    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6af932c-dd1a-4e2e-9ac5-8f686b4badea-kserve-provision-location\") pod \"isvc-init-fail-7f3b77-predictor-54ffd4dd7f-tkljc\" (UID: \"b6af932c-dd1a-4e2e-9ac5-8f686b4badea\") " pod="kserve-ci-e2e-test/isvc-init-fail-7f3b77-predictor-54ffd4dd7f-tkljc"
May 25 12:47:53.339599 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:53.339575    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6af932c-dd1a-4e2e-9ac5-8f686b4badea-proxy-tls\") pod \"isvc-init-fail-7f3b77-predictor-54ffd4dd7f-tkljc\" (UID: \"b6af932c-dd1a-4e2e-9ac5-8f686b4badea\") " pod="kserve-ci-e2e-test/isvc-init-fail-7f3b77-predictor-54ffd4dd7f-tkljc"
May 25 12:47:53.339599 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:53.339593    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-init-fail-7f3b77-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6af932c-dd1a-4e2e-9ac5-8f686b4badea-isvc-init-fail-7f3b77-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-7f3b77-predictor-54ffd4dd7f-tkljc\" (UID: \"b6af932c-dd1a-4e2e-9ac5-8f686b4badea\") " pod="kserve-ci-e2e-test/isvc-init-fail-7f3b77-predictor-54ffd4dd7f-tkljc"
May 25 12:47:53.339838 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:53.339624    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6af932c-dd1a-4e2e-9ac5-8f686b4badea-cabundle-cert\") pod \"isvc-init-fail-7f3b77-predictor-54ffd4dd7f-tkljc\" (UID: \"b6af932c-dd1a-4e2e-9ac5-8f686b4badea\") " pod="kserve-ci-e2e-test/isvc-init-fail-7f3b77-predictor-54ffd4dd7f-tkljc"
May 25 12:47:53.339838 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:53.339695    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4w4z8\" (UniqueName: \"kubernetes.io/projected/b6af932c-dd1a-4e2e-9ac5-8f686b4badea-kube-api-access-4w4z8\") pod \"isvc-init-fail-7f3b77-predictor-54ffd4dd7f-tkljc\" (UID: \"b6af932c-dd1a-4e2e-9ac5-8f686b4badea\") " pod="kserve-ci-e2e-test/isvc-init-fail-7f3b77-predictor-54ffd4dd7f-tkljc"
May 25 12:47:53.339838 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:47:53.339718    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-7f3b77-predictor-serving-cert: secret "isvc-init-fail-7f3b77-predictor-serving-cert" not found
May 25 12:47:53.339838 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:53.339745    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53-proxy-tls\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:47:53.339838 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:53.339756    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8rqc\" (UniqueName: \"kubernetes.io/projected/47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53-kube-api-access-h8rqc\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:47:53.339838 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:47:53.339791    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6af932c-dd1a-4e2e-9ac5-8f686b4badea-proxy-tls podName:b6af932c-dd1a-4e2e-9ac5-8f686b4badea nodeName:}" failed. No retries permitted until 2026-05-25 12:47:53.83977177 +0000 UTC m=+2086.799822154 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b6af932c-dd1a-4e2e-9ac5-8f686b4badea-proxy-tls") pod "isvc-init-fail-7f3b77-predictor-54ffd4dd7f-tkljc" (UID: "b6af932c-dd1a-4e2e-9ac5-8f686b4badea") : secret "isvc-init-fail-7f3b77-predictor-serving-cert" not found
May 25 12:47:53.340148 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:53.339994    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6af932c-dd1a-4e2e-9ac5-8f686b4badea-kserve-provision-location\") pod \"isvc-init-fail-7f3b77-predictor-54ffd4dd7f-tkljc\" (UID: \"b6af932c-dd1a-4e2e-9ac5-8f686b4badea\") " pod="kserve-ci-e2e-test/isvc-init-fail-7f3b77-predictor-54ffd4dd7f-tkljc"
May 25 12:47:53.340308 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:53.340287    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6af932c-dd1a-4e2e-9ac5-8f686b4badea-cabundle-cert\") pod \"isvc-init-fail-7f3b77-predictor-54ffd4dd7f-tkljc\" (UID: \"b6af932c-dd1a-4e2e-9ac5-8f686b4badea\") " pod="kserve-ci-e2e-test/isvc-init-fail-7f3b77-predictor-54ffd4dd7f-tkljc"
May 25 12:47:53.340401 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:53.340379    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-init-fail-7f3b77-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6af932c-dd1a-4e2e-9ac5-8f686b4badea-isvc-init-fail-7f3b77-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-7f3b77-predictor-54ffd4dd7f-tkljc\" (UID: \"b6af932c-dd1a-4e2e-9ac5-8f686b4badea\") " pod="kserve-ci-e2e-test/isvc-init-fail-7f3b77-predictor-54ffd4dd7f-tkljc"
May 25 12:47:53.348785 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:53.348763    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w4z8\" (UniqueName: \"kubernetes.io/projected/b6af932c-dd1a-4e2e-9ac5-8f686b4badea-kube-api-access-4w4z8\") pod \"isvc-init-fail-7f3b77-predictor-54ffd4dd7f-tkljc\" (UID: \"b6af932c-dd1a-4e2e-9ac5-8f686b4badea\") " pod="kserve-ci-e2e-test/isvc-init-fail-7f3b77-predictor-54ffd4dd7f-tkljc"
May 25 12:47:53.580378 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:53.580348    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-f32ae1-predictor-546d9cc87-dtmgp_47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53/storage-initializer/1.log"
May 25 12:47:53.580517 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:53.580479    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-f32ae1-predictor-546d9cc87-dtmgp"
May 25 12:47:53.580517 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:53.580486    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-f32ae1-predictor-546d9cc87-dtmgp" event={"ID":"47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53","Type":"ContainerDied","Data":"722e712e087e3e8ca51e7543bb25e1b0645ec9a0f01e52676297bb188444dfef"}
May 25 12:47:53.580662 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:53.580528    2555 scope.go:117] "RemoveContainer" containerID="6c3462969edb3e0fb045c19808cc0140a4a4caefb307728a50e9608e09ab7bf9"
May 25 12:47:53.582695 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:53.582670    2555 generic.go:358] "Generic (PLEG): container finished" podID="61ae41ef-8448-4eba-aec7-92c15d8e0a39" containerID="38a07528613ebc025d65cd7a1c5cb3e40398ca5a0677925e89c6d2a18b508197" exitCode=2
May 25 12:47:53.582809 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:53.582705    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-f32ae1-predictor-7f478b4d68-nv59n" event={"ID":"61ae41ef-8448-4eba-aec7-92c15d8e0a39","Type":"ContainerDied","Data":"38a07528613ebc025d65cd7a1c5cb3e40398ca5a0677925e89c6d2a18b508197"}
May 25 12:47:53.612702 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:53.612632    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-f32ae1-predictor-546d9cc87-dtmgp"]
May 25 12:47:53.615109 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:53.615086    2555 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-f32ae1-predictor-546d9cc87-dtmgp"]
May 25 12:47:53.844727 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:53.844689    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6af932c-dd1a-4e2e-9ac5-8f686b4badea-proxy-tls\") pod \"isvc-init-fail-7f3b77-predictor-54ffd4dd7f-tkljc\" (UID: \"b6af932c-dd1a-4e2e-9ac5-8f686b4badea\") " pod="kserve-ci-e2e-test/isvc-init-fail-7f3b77-predictor-54ffd4dd7f-tkljc"
May 25 12:47:53.847004 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:53.846985    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6af932c-dd1a-4e2e-9ac5-8f686b4badea-proxy-tls\") pod \"isvc-init-fail-7f3b77-predictor-54ffd4dd7f-tkljc\" (UID: \"b6af932c-dd1a-4e2e-9ac5-8f686b4badea\") " pod="kserve-ci-e2e-test/isvc-init-fail-7f3b77-predictor-54ffd4dd7f-tkljc"
May 25 12:47:54.124881 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:54.124852    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-7f3b77-predictor-54ffd4dd7f-tkljc"
May 25 12:47:54.246261 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:54.246233    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-7f3b77-predictor-54ffd4dd7f-tkljc"]
May 25 12:47:54.248593 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:47:54.248565    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6af932c_dd1a_4e2e_9ac5_8f686b4badea.slice/crio-50fe940643908e22182c3d72931cb7a8ebde1a9969e2f5c35771710619051837 WatchSource:0}: Error finding container 50fe940643908e22182c3d72931cb7a8ebde1a9969e2f5c35771710619051837: Status 404 returned error can't find the container with id 50fe940643908e22182c3d72931cb7a8ebde1a9969e2f5c35771710619051837
May 25 12:47:54.587285 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:54.587239    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-7f3b77-predictor-54ffd4dd7f-tkljc" event={"ID":"b6af932c-dd1a-4e2e-9ac5-8f686b4badea","Type":"ContainerStarted","Data":"c0ee0e7db8483c95f08ff88cf865b4ba757eb60b911256332a07763a25f1886e"}
May 25 12:47:54.587481 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:54.587295    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-7f3b77-predictor-54ffd4dd7f-tkljc" event={"ID":"b6af932c-dd1a-4e2e-9ac5-8f686b4badea","Type":"ContainerStarted","Data":"50fe940643908e22182c3d72931cb7a8ebde1a9969e2f5c35771710619051837"}
May 25 12:47:55.599236 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:55.599204    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53" path="/var/lib/kubelet/pods/47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53/volumes"
May 25 12:47:56.951833 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:56.951810    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-f32ae1-predictor-7f478b4d68-nv59n"
May 25 12:47:56.969836 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:56.969776    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dnbdm\" (UniqueName: \"kubernetes.io/projected/61ae41ef-8448-4eba-aec7-92c15d8e0a39-kube-api-access-dnbdm\") pod \"61ae41ef-8448-4eba-aec7-92c15d8e0a39\" (UID: \"61ae41ef-8448-4eba-aec7-92c15d8e0a39\") "
May 25 12:47:56.969836 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:56.969827    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61ae41ef-8448-4eba-aec7-92c15d8e0a39-kserve-provision-location\") pod \"61ae41ef-8448-4eba-aec7-92c15d8e0a39\" (UID: \"61ae41ef-8448-4eba-aec7-92c15d8e0a39\") "
May 25 12:47:56.970031 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:56.969916    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-primary-f32ae1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/61ae41ef-8448-4eba-aec7-92c15d8e0a39-isvc-primary-f32ae1-kube-rbac-proxy-sar-config\") pod \"61ae41ef-8448-4eba-aec7-92c15d8e0a39\" (UID: \"61ae41ef-8448-4eba-aec7-92c15d8e0a39\") "
May 25 12:47:56.970031 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:56.969982    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61ae41ef-8448-4eba-aec7-92c15d8e0a39-proxy-tls\") pod \"61ae41ef-8448-4eba-aec7-92c15d8e0a39\" (UID: \"61ae41ef-8448-4eba-aec7-92c15d8e0a39\") "
May 25 12:47:56.970533 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:56.970475    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61ae41ef-8448-4eba-aec7-92c15d8e0a39-isvc-primary-f32ae1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-primary-f32ae1-kube-rbac-proxy-sar-config") pod "61ae41ef-8448-4eba-aec7-92c15d8e0a39" (UID: "61ae41ef-8448-4eba-aec7-92c15d8e0a39"). InnerVolumeSpecName "isvc-primary-f32ae1-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:47:56.970686 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:56.970560    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61ae41ef-8448-4eba-aec7-92c15d8e0a39-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "61ae41ef-8448-4eba-aec7-92c15d8e0a39" (UID: "61ae41ef-8448-4eba-aec7-92c15d8e0a39"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:47:56.972301 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:56.972273    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61ae41ef-8448-4eba-aec7-92c15d8e0a39-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "61ae41ef-8448-4eba-aec7-92c15d8e0a39" (UID: "61ae41ef-8448-4eba-aec7-92c15d8e0a39"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:47:56.972741 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:56.972720    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61ae41ef-8448-4eba-aec7-92c15d8e0a39-kube-api-access-dnbdm" (OuterVolumeSpecName: "kube-api-access-dnbdm") pod "61ae41ef-8448-4eba-aec7-92c15d8e0a39" (UID: "61ae41ef-8448-4eba-aec7-92c15d8e0a39"). InnerVolumeSpecName "kube-api-access-dnbdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:47:57.071335 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:57.071281    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61ae41ef-8448-4eba-aec7-92c15d8e0a39-proxy-tls\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:47:57.071335 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:57.071306    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnbdm\" (UniqueName: \"kubernetes.io/projected/61ae41ef-8448-4eba-aec7-92c15d8e0a39-kube-api-access-dnbdm\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:47:57.071335 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:57.071317    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61ae41ef-8448-4eba-aec7-92c15d8e0a39-kserve-provision-location\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:47:57.071335 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:57.071327    2555 reconciler_common.go:299] "Volume detached for volume \"isvc-primary-f32ae1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/61ae41ef-8448-4eba-aec7-92c15d8e0a39-isvc-primary-f32ae1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:47:57.599373 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:57.599339    2555 generic.go:358] "Generic (PLEG): container finished" podID="61ae41ef-8448-4eba-aec7-92c15d8e0a39" containerID="3279db93ade2fadf71a457de74396c15118ecf8c57f9826a8494a096b727c73b" exitCode=0
May 25 12:47:57.599543 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:57.599423    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-f32ae1-predictor-7f478b4d68-nv59n" event={"ID":"61ae41ef-8448-4eba-aec7-92c15d8e0a39","Type":"ContainerDied","Data":"3279db93ade2fadf71a457de74396c15118ecf8c57f9826a8494a096b727c73b"}
May 25 12:47:57.599543 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:57.599438    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-f32ae1-predictor-7f478b4d68-nv59n"
May 25 12:47:57.599543 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:57.599453    2555 scope.go:117] "RemoveContainer" containerID="38a07528613ebc025d65cd7a1c5cb3e40398ca5a0677925e89c6d2a18b508197"
May 25 12:47:57.599698 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:57.599443    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-f32ae1-predictor-7f478b4d68-nv59n" event={"ID":"61ae41ef-8448-4eba-aec7-92c15d8e0a39","Type":"ContainerDied","Data":"3bd3f131110f726641b11195ccb7bb2a90b9be65dece562d6972911527d1b74f"}
May 25 12:47:57.607821 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:57.607802    2555 scope.go:117] "RemoveContainer" containerID="3279db93ade2fadf71a457de74396c15118ecf8c57f9826a8494a096b727c73b"
May 25 12:47:57.615217 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:57.615199    2555 scope.go:117] "RemoveContainer" containerID="466c7b02dad80b3e8d6f107e36987da8366f1d83c9a825bb7820da6bde43c60c"
May 25 12:47:57.622003 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:57.621977    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-f32ae1-predictor-7f478b4d68-nv59n"]
May 25 12:47:57.622331 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:57.622310    2555 scope.go:117] "RemoveContainer" containerID="38a07528613ebc025d65cd7a1c5cb3e40398ca5a0677925e89c6d2a18b508197"
May 25 12:47:57.622596 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:47:57.622578    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38a07528613ebc025d65cd7a1c5cb3e40398ca5a0677925e89c6d2a18b508197\": container with ID starting with 38a07528613ebc025d65cd7a1c5cb3e40398ca5a0677925e89c6d2a18b508197 not found: ID does not exist" containerID="38a07528613ebc025d65cd7a1c5cb3e40398ca5a0677925e89c6d2a18b508197"
May 25 12:47:57.622675 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:57.622603    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38a07528613ebc025d65cd7a1c5cb3e40398ca5a0677925e89c6d2a18b508197"} err="failed to get container status \"38a07528613ebc025d65cd7a1c5cb3e40398ca5a0677925e89c6d2a18b508197\": rpc error: code = NotFound desc = could not find container \"38a07528613ebc025d65cd7a1c5cb3e40398ca5a0677925e89c6d2a18b508197\": container with ID starting with 38a07528613ebc025d65cd7a1c5cb3e40398ca5a0677925e89c6d2a18b508197 not found: ID does not exist"
May 25 12:47:57.622675 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:57.622621    2555 scope.go:117] "RemoveContainer" containerID="3279db93ade2fadf71a457de74396c15118ecf8c57f9826a8494a096b727c73b"
May 25 12:47:57.623072 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:47:57.623054    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3279db93ade2fadf71a457de74396c15118ecf8c57f9826a8494a096b727c73b\": container with ID starting with 3279db93ade2fadf71a457de74396c15118ecf8c57f9826a8494a096b727c73b not found: ID does not exist" containerID="3279db93ade2fadf71a457de74396c15118ecf8c57f9826a8494a096b727c73b"
May 25 12:47:57.623142 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:57.623076    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3279db93ade2fadf71a457de74396c15118ecf8c57f9826a8494a096b727c73b"} err="failed to get container status \"3279db93ade2fadf71a457de74396c15118ecf8c57f9826a8494a096b727c73b\": rpc error: code = NotFound desc = could not find container \"3279db93ade2fadf71a457de74396c15118ecf8c57f9826a8494a096b727c73b\": container with ID starting with 3279db93ade2fadf71a457de74396c15118ecf8c57f9826a8494a096b727c73b not found: ID does not exist"
May 25 12:47:57.623142 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:57.623092    2555 scope.go:117] "RemoveContainer" containerID="466c7b02dad80b3e8d6f107e36987da8366f1d83c9a825bb7820da6bde43c60c"
May 25 12:47:57.623415 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:47:57.623387    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"466c7b02dad80b3e8d6f107e36987da8366f1d83c9a825bb7820da6bde43c60c\": container with ID starting with 466c7b02dad80b3e8d6f107e36987da8366f1d83c9a825bb7820da6bde43c60c not found: ID does not exist" containerID="466c7b02dad80b3e8d6f107e36987da8366f1d83c9a825bb7820da6bde43c60c"
May 25 12:47:57.623493 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:57.623442    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"466c7b02dad80b3e8d6f107e36987da8366f1d83c9a825bb7820da6bde43c60c"} err="failed to get container status \"466c7b02dad80b3e8d6f107e36987da8366f1d83c9a825bb7820da6bde43c60c\": rpc error: code = NotFound desc = could not find container \"466c7b02dad80b3e8d6f107e36987da8366f1d83c9a825bb7820da6bde43c60c\": container with ID starting with 466c7b02dad80b3e8d6f107e36987da8366f1d83c9a825bb7820da6bde43c60c not found: ID does not exist"
May 25 12:47:57.625172 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:57.625154    2555 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-f32ae1-predictor-7f478b4d68-nv59n"]
May 25 12:47:59.598432 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:59.598388    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61ae41ef-8448-4eba-aec7-92c15d8e0a39" path="/var/lib/kubelet/pods/61ae41ef-8448-4eba-aec7-92c15d8e0a39/volumes"
May 25 12:47:59.607367 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:59.607341    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-7f3b77-predictor-54ffd4dd7f-tkljc_b6af932c-dd1a-4e2e-9ac5-8f686b4badea/storage-initializer/0.log"
May 25 12:47:59.607510 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:59.607382    2555 generic.go:358] "Generic (PLEG): container finished" podID="b6af932c-dd1a-4e2e-9ac5-8f686b4badea" containerID="c0ee0e7db8483c95f08ff88cf865b4ba757eb60b911256332a07763a25f1886e" exitCode=1
May 25 12:47:59.607510 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:59.607409    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-7f3b77-predictor-54ffd4dd7f-tkljc" event={"ID":"b6af932c-dd1a-4e2e-9ac5-8f686b4badea","Type":"ContainerDied","Data":"c0ee0e7db8483c95f08ff88cf865b4ba757eb60b911256332a07763a25f1886e"}
May 25 12:47:59.608551 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:47:59.608535    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 12:48:00.612222 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:48:00.612190    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-7f3b77-predictor-54ffd4dd7f-tkljc_b6af932c-dd1a-4e2e-9ac5-8f686b4badea/storage-initializer/0.log"
May 25 12:48:00.612686 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:48:00.612244    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-7f3b77-predictor-54ffd4dd7f-tkljc" event={"ID":"b6af932c-dd1a-4e2e-9ac5-8f686b4badea","Type":"ContainerStarted","Data":"01b73c2625968822c4d16d65238e3f7a0dec0c8455338b06f3e44024e370e4ab"}
May 25 12:48:03.193737 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:48:03.193704    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-7f3b77-predictor-54ffd4dd7f-tkljc"]
May 25 12:48:03.194179 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:48:03.194036    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-7f3b77-predictor-54ffd4dd7f-tkljc" podUID="b6af932c-dd1a-4e2e-9ac5-8f686b4badea" containerName="storage-initializer" containerID="cri-o://01b73c2625968822c4d16d65238e3f7a0dec0c8455338b06f3e44024e370e4ab" gracePeriod=30
May 25 12:48:04.625586 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:48:04.625561    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-7f3b77-predictor-54ffd4dd7f-tkljc_b6af932c-dd1a-4e2e-9ac5-8f686b4badea/storage-initializer/1.log"
May 25 12:48:04.626006 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:48:04.625984    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-7f3b77-predictor-54ffd4dd7f-tkljc_b6af932c-dd1a-4e2e-9ac5-8f686b4badea/storage-initializer/0.log"
May 25 12:48:04.626064 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:48:04.626014    2555 generic.go:358] "Generic (PLEG): container finished" podID="b6af932c-dd1a-4e2e-9ac5-8f686b4badea" containerID="01b73c2625968822c4d16d65238e3f7a0dec0c8455338b06f3e44024e370e4ab" exitCode=1
May 25 12:48:04.626126 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:48:04.626088    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-7f3b77-predictor-54ffd4dd7f-tkljc" event={"ID":"b6af932c-dd1a-4e2e-9ac5-8f686b4badea","Type":"ContainerDied","Data":"01b73c2625968822c4d16d65238e3f7a0dec0c8455338b06f3e44024e370e4ab"}
May 25 12:48:04.626184 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:48:04.626128    2555 scope.go:117] "RemoveContainer" containerID="c0ee0e7db8483c95f08ff88cf865b4ba757eb60b911256332a07763a25f1886e"
May 25 12:48:04.640697 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:48:04.640682    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-7f3b77-predictor-54ffd4dd7f-tkljc_b6af932c-dd1a-4e2e-9ac5-8f686b4badea/storage-initializer/1.log"
May 25 12:48:04.640994 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:48:04.640975    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0ee0e7db8483c95f08ff88cf865b4ba757eb60b911256332a07763a25f1886e\": container with ID starting with c0ee0e7db8483c95f08ff88cf865b4ba757eb60b911256332a07763a25f1886e not found: ID does not exist" containerID="c0ee0e7db8483c95f08ff88cf865b4ba757eb60b911256332a07763a25f1886e"
May 25 12:48:04.641061 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:48:04.641047    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-7f3b77-predictor-54ffd4dd7f-tkljc"
May 25 12:48:04.730968 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:48:04.730940    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6af932c-dd1a-4e2e-9ac5-8f686b4badea-proxy-tls\") pod \"b6af932c-dd1a-4e2e-9ac5-8f686b4badea\" (UID: \"b6af932c-dd1a-4e2e-9ac5-8f686b4badea\") "
May 25 12:48:04.731124 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:48:04.730984    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-init-fail-7f3b77-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6af932c-dd1a-4e2e-9ac5-8f686b4badea-isvc-init-fail-7f3b77-kube-rbac-proxy-sar-config\") pod \"b6af932c-dd1a-4e2e-9ac5-8f686b4badea\" (UID: \"b6af932c-dd1a-4e2e-9ac5-8f686b4badea\") "
May 25 12:48:04.731124 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:48:04.731033    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6af932c-dd1a-4e2e-9ac5-8f686b4badea-kserve-provision-location\") pod \"b6af932c-dd1a-4e2e-9ac5-8f686b4badea\" (UID: \"b6af932c-dd1a-4e2e-9ac5-8f686b4badea\") "
May 25 12:48:04.731124 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:48:04.731100    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6af932c-dd1a-4e2e-9ac5-8f686b4badea-cabundle-cert\") pod \"b6af932c-dd1a-4e2e-9ac5-8f686b4badea\" (UID: \"b6af932c-dd1a-4e2e-9ac5-8f686b4badea\") "
May 25 12:48:04.731288 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:48:04.731123    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4w4z8\" (UniqueName: \"kubernetes.io/projected/b6af932c-dd1a-4e2e-9ac5-8f686b4badea-kube-api-access-4w4z8\") pod \"b6af932c-dd1a-4e2e-9ac5-8f686b4badea\" (UID: \"b6af932c-dd1a-4e2e-9ac5-8f686b4badea\") "
May 25 12:48:04.731404 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:48:04.731378    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6af932c-dd1a-4e2e-9ac5-8f686b4badea-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b6af932c-dd1a-4e2e-9ac5-8f686b4badea" (UID: "b6af932c-dd1a-4e2e-9ac5-8f686b4badea"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:48:04.731455 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:48:04.731398    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6af932c-dd1a-4e2e-9ac5-8f686b4badea-isvc-init-fail-7f3b77-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-init-fail-7f3b77-kube-rbac-proxy-sar-config") pod "b6af932c-dd1a-4e2e-9ac5-8f686b4badea" (UID: "b6af932c-dd1a-4e2e-9ac5-8f686b4badea"). InnerVolumeSpecName "isvc-init-fail-7f3b77-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:48:04.731455 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:48:04.731444    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6af932c-dd1a-4e2e-9ac5-8f686b4badea-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b6af932c-dd1a-4e2e-9ac5-8f686b4badea" (UID: "b6af932c-dd1a-4e2e-9ac5-8f686b4badea"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:48:04.733066 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:48:04.733045    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6af932c-dd1a-4e2e-9ac5-8f686b4badea-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b6af932c-dd1a-4e2e-9ac5-8f686b4badea" (UID: "b6af932c-dd1a-4e2e-9ac5-8f686b4badea"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:48:04.733066 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:48:04.733046    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6af932c-dd1a-4e2e-9ac5-8f686b4badea-kube-api-access-4w4z8" (OuterVolumeSpecName: "kube-api-access-4w4z8") pod "b6af932c-dd1a-4e2e-9ac5-8f686b4badea" (UID: "b6af932c-dd1a-4e2e-9ac5-8f686b4badea"). InnerVolumeSpecName "kube-api-access-4w4z8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:48:04.831881 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:48:04.831856    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6af932c-dd1a-4e2e-9ac5-8f686b4badea-proxy-tls\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:48:04.831881 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:48:04.831879    2555 reconciler_common.go:299] "Volume detached for volume \"isvc-init-fail-7f3b77-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6af932c-dd1a-4e2e-9ac5-8f686b4badea-isvc-init-fail-7f3b77-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:48:04.832037 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:48:04.831895    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6af932c-dd1a-4e2e-9ac5-8f686b4badea-kserve-provision-location\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:48:04.832037 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:48:04.831908    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6af932c-dd1a-4e2e-9ac5-8f686b4badea-cabundle-cert\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:48:04.832037 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:48:04.831920    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w4z8\" (UniqueName: \"kubernetes.io/projected/b6af932c-dd1a-4e2e-9ac5-8f686b4badea-kube-api-access-4w4z8\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:48:05.632084 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:48:05.632057    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-7f3b77-predictor-54ffd4dd7f-tkljc_b6af932c-dd1a-4e2e-9ac5-8f686b4badea/storage-initializer/1.log"
May 25 12:48:05.632430 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:48:05.632185    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-7f3b77-predictor-54ffd4dd7f-tkljc" event={"ID":"b6af932c-dd1a-4e2e-9ac5-8f686b4badea","Type":"ContainerDied","Data":"50fe940643908e22182c3d72931cb7a8ebde1a9969e2f5c35771710619051837"}
May 25 12:48:05.632430 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:48:05.632209    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-7f3b77-predictor-54ffd4dd7f-tkljc"
May 25 12:48:05.632430 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:48:05.632223    2555 scope.go:117] "RemoveContainer" containerID="01b73c2625968822c4d16d65238e3f7a0dec0c8455338b06f3e44024e370e4ab"
May 25 12:48:05.658737 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:48:05.658715    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-7f3b77-predictor-54ffd4dd7f-tkljc"]
May 25 12:48:05.661953 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:48:05.661932    2555 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-7f3b77-predictor-54ffd4dd7f-tkljc"]
May 25 12:48:07.598611 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:48:07.598574    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6af932c-dd1a-4e2e-9ac5-8f686b4badea" path="/var/lib/kubelet/pods/b6af932c-dd1a-4e2e-9ac5-8f686b4badea/volumes"
May 25 12:48:07.632871 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:48:07.632846    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-2tqtm_00e6f79c-8f05-4330-aa16-1ada2ae9e9cc/console-operator/1.log"
May 25 12:48:07.635341 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:48:07.635315    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-2tqtm_00e6f79c-8f05-4330-aa16-1ada2ae9e9cc/console-operator/1.log"
May 25 12:53:07.655022 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:53:07.654989    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-2tqtm_00e6f79c-8f05-4330-aa16-1ada2ae9e9cc/console-operator/1.log"
May 25 12:53:07.659180 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:53:07.659157    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-2tqtm_00e6f79c-8f05-4330-aa16-1ada2ae9e9cc/console-operator/1.log"
May 25 12:57:46.877411 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:57:46.877376    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pbjnx"]
May 25 12:57:46.877845 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:57:46.877702    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53" containerName="storage-initializer"
May 25 12:57:46.877845 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:57:46.877713    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53" containerName="storage-initializer"
May 25 12:57:46.877845 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:57:46.877724    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61ae41ef-8448-4eba-aec7-92c15d8e0a39" containerName="storage-initializer"
May 25 12:57:46.877845 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:57:46.877729    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="61ae41ef-8448-4eba-aec7-92c15d8e0a39" containerName="storage-initializer"
May 25 12:57:46.877845 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:57:46.877739    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61ae41ef-8448-4eba-aec7-92c15d8e0a39" containerName="kserve-container"
May 25 12:57:46.877845 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:57:46.877744    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="61ae41ef-8448-4eba-aec7-92c15d8e0a39" containerName="kserve-container"
May 25 12:57:46.877845 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:57:46.877752    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6af932c-dd1a-4e2e-9ac5-8f686b4badea" containerName="storage-initializer"
May 25 12:57:46.877845 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:57:46.877757    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6af932c-dd1a-4e2e-9ac5-8f686b4badea" containerName="storage-initializer"
May 25 12:57:46.877845 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:57:46.877771    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61ae41ef-8448-4eba-aec7-92c15d8e0a39" containerName="kube-rbac-proxy"
May 25 12:57:46.877845 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:57:46.877776    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="61ae41ef-8448-4eba-aec7-92c15d8e0a39" containerName="kube-rbac-proxy"
May 25 12:57:46.877845 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:57:46.877782    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6af932c-dd1a-4e2e-9ac5-8f686b4badea" containerName="storage-initializer"
May 25 12:57:46.877845 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:57:46.877787    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6af932c-dd1a-4e2e-9ac5-8f686b4badea" containerName="storage-initializer"
May 25 12:57:46.877845 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:57:46.877834    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="61ae41ef-8448-4eba-aec7-92c15d8e0a39" containerName="kube-rbac-proxy"
May 25 12:57:46.877845 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:57:46.877845    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="61ae41ef-8448-4eba-aec7-92c15d8e0a39" containerName="kserve-container"
May 25 12:57:46.877845 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:57:46.877852    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53" containerName="storage-initializer"
May 25 12:57:46.878358 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:57:46.877858    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="b6af932c-dd1a-4e2e-9ac5-8f686b4badea" containerName="storage-initializer"
May 25 12:57:46.878358 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:57:46.877865    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="b6af932c-dd1a-4e2e-9ac5-8f686b4badea" containerName="storage-initializer"
May 25 12:57:46.878358 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:57:46.877916    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53" containerName="storage-initializer"
May 25 12:57:46.878358 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:57:46.877922    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53" containerName="storage-initializer"
May 25 12:57:46.878358 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:57:46.877975    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="47d40cf7-5ec2-41e1-9ec4-7a961ffc4a53" containerName="storage-initializer"
May 25 12:57:46.880841 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:57:46.880825    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pbjnx"
May 25 12:57:46.882635 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:57:46.882610    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 25 12:57:46.882851 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:57:46.882711    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 25 12:57:46.882851 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:57:46.882794    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
May 25 12:57:46.883413 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:57:46.883394    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-predictor-serving-cert\""
May 25 12:57:46.883513 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:57:46.883413    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6qbkc\""
May 25 12:57:46.890707 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:57:46.890684    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pbjnx"]
May 25 12:57:46.912959 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:57:46.912928    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9bbc\" (UniqueName: \"kubernetes.io/projected/3e1aefc8-03d0-4659-b059-a0f859f29a75-kube-api-access-v9bbc\") pod \"isvc-sklearn-predictor-bc9cf9b57-pbjnx\" (UID: \"3e1aefc8-03d0-4659-b059-a0f859f29a75\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pbjnx"
May 25 12:57:46.913068 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:57:46.912975    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e1aefc8-03d0-4659-b059-a0f859f29a75-kserve-provision-location\") pod \"isvc-sklearn-predictor-bc9cf9b57-pbjnx\" (UID: \"3e1aefc8-03d0-4659-b059-a0f859f29a75\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pbjnx"
May 25 12:57:46.913068 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:57:46.913018    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e1aefc8-03d0-4659-b059-a0f859f29a75-proxy-tls\") pod \"isvc-sklearn-predictor-bc9cf9b57-pbjnx\" (UID: \"3e1aefc8-03d0-4659-b059-a0f859f29a75\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pbjnx"
May 25 12:57:46.913068 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:57:46.913051    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e1aefc8-03d0-4659-b059-a0f859f29a75-isvc-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-bc9cf9b57-pbjnx\" (UID: \"3e1aefc8-03d0-4659-b059-a0f859f29a75\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pbjnx"
May 25 12:57:47.013919 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:57:47.013893    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-v9bbc\" (UniqueName: \"kubernetes.io/projected/3e1aefc8-03d0-4659-b059-a0f859f29a75-kube-api-access-v9bbc\") pod \"isvc-sklearn-predictor-bc9cf9b57-pbjnx\" (UID: \"3e1aefc8-03d0-4659-b059-a0f859f29a75\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pbjnx"
May 25 12:57:47.014058 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:57:47.013935    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e1aefc8-03d0-4659-b059-a0f859f29a75-kserve-provision-location\") pod \"isvc-sklearn-predictor-bc9cf9b57-pbjnx\" (UID: \"3e1aefc8-03d0-4659-b059-a0f859f29a75\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pbjnx"
May 25 12:57:47.014058 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:57:47.013964    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e1aefc8-03d0-4659-b059-a0f859f29a75-proxy-tls\") pod \"isvc-sklearn-predictor-bc9cf9b57-pbjnx\" (UID: \"3e1aefc8-03d0-4659-b059-a0f859f29a75\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pbjnx"
May 25 12:57:47.014058 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:57:47.013990    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e1aefc8-03d0-4659-b059-a0f859f29a75-isvc-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-bc9cf9b57-pbjnx\" (UID: \"3e1aefc8-03d0-4659-b059-a0f859f29a75\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pbjnx"
May 25 12:57:47.014401 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:57:47.014380    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e1aefc8-03d0-4659-b059-a0f859f29a75-kserve-provision-location\") pod \"isvc-sklearn-predictor-bc9cf9b57-pbjnx\" (UID: \"3e1aefc8-03d0-4659-b059-a0f859f29a75\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pbjnx"
May 25 12:57:47.014732 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:57:47.014715    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e1aefc8-03d0-4659-b059-a0f859f29a75-isvc-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-bc9cf9b57-pbjnx\" (UID: \"3e1aefc8-03d0-4659-b059-a0f859f29a75\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pbjnx"
May 25 12:57:47.016439 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:57:47.016424    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e1aefc8-03d0-4659-b059-a0f859f29a75-proxy-tls\") pod \"isvc-sklearn-predictor-bc9cf9b57-pbjnx\" (UID: \"3e1aefc8-03d0-4659-b059-a0f859f29a75\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pbjnx"
May 25 12:57:47.020017 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:57:47.020000    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9bbc\" (UniqueName: \"kubernetes.io/projected/3e1aefc8-03d0-4659-b059-a0f859f29a75-kube-api-access-v9bbc\") pod \"isvc-sklearn-predictor-bc9cf9b57-pbjnx\" (UID: \"3e1aefc8-03d0-4659-b059-a0f859f29a75\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pbjnx"
May 25 12:57:47.192305 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:57:47.192281    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pbjnx"
May 25 12:57:47.319790 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:57:47.319762    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pbjnx"]
May 25 12:57:47.321620 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:57:47.321594    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e1aefc8_03d0_4659_b059_a0f859f29a75.slice/crio-c0a9564dc242443e0b2f8568dc35c6be9d6072cf2958e1c991b1d1376485794c WatchSource:0}: Error finding container c0a9564dc242443e0b2f8568dc35c6be9d6072cf2958e1c991b1d1376485794c: Status 404 returned error can't find the container with id c0a9564dc242443e0b2f8568dc35c6be9d6072cf2958e1c991b1d1376485794c
May 25 12:57:47.323382 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:57:47.323362    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 12:57:47.556463 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:57:47.556377    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pbjnx" event={"ID":"3e1aefc8-03d0-4659-b059-a0f859f29a75","Type":"ContainerStarted","Data":"bdabd09fe5ef8699b308a7eecc583cd2221627c0c4370b79b403afa1bd8ec4b1"}
May 25 12:57:47.556463 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:57:47.556413    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pbjnx" event={"ID":"3e1aefc8-03d0-4659-b059-a0f859f29a75","Type":"ContainerStarted","Data":"c0a9564dc242443e0b2f8568dc35c6be9d6072cf2958e1c991b1d1376485794c"}
May 25 12:57:51.570411 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:57:51.570382    2555 generic.go:358] "Generic (PLEG): container finished" podID="3e1aefc8-03d0-4659-b059-a0f859f29a75" containerID="bdabd09fe5ef8699b308a7eecc583cd2221627c0c4370b79b403afa1bd8ec4b1" exitCode=0
May 25 12:57:51.570916 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:57:51.570463    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pbjnx" event={"ID":"3e1aefc8-03d0-4659-b059-a0f859f29a75","Type":"ContainerDied","Data":"bdabd09fe5ef8699b308a7eecc583cd2221627c0c4370b79b403afa1bd8ec4b1"}
May 25 12:57:52.576521 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:57:52.576492    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pbjnx" event={"ID":"3e1aefc8-03d0-4659-b059-a0f859f29a75","Type":"ContainerStarted","Data":"b9bf3e2cb2daaee362814dd592624d6bd38e031a5869b1bf2c9511ee2844ddb8"}
May 25 12:57:52.576917 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:57:52.576529    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pbjnx" event={"ID":"3e1aefc8-03d0-4659-b059-a0f859f29a75","Type":"ContainerStarted","Data":"c154d99bbf000f46f5c77f482be41898161ec8ec7b05e410e363f3d375ff412f"}
May 25 12:57:52.576917 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:57:52.576806    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pbjnx"
May 25 12:57:52.576917 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:57:52.576835    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pbjnx"
May 25 12:57:52.577831 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:57:52.577803    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pbjnx" podUID="3e1aefc8-03d0-4659-b059-a0f859f29a75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
May 25 12:57:52.593577 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:57:52.593536    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pbjnx" podStartSLOduration=6.593524668 podStartE2EDuration="6.593524668s" podCreationTimestamp="2026-05-25 12:57:46 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:57:52.591734319 +0000 UTC m=+2685.551784729" watchObservedRunningTime="2026-05-25 12:57:52.593524668 +0000 UTC m=+2685.553575074"
May 25 12:57:53.579953 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:57:53.579903    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pbjnx" podUID="3e1aefc8-03d0-4659-b059-a0f859f29a75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
May 25 12:57:58.584846 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:57:58.584764    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pbjnx"
May 25 12:57:58.585457 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:57:58.585424    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pbjnx" podUID="3e1aefc8-03d0-4659-b059-a0f859f29a75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
May 25 12:58:07.685490 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:58:07.685458    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-2tqtm_00e6f79c-8f05-4330-aa16-1ada2ae9e9cc/console-operator/1.log"
May 25 12:58:07.691801 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:58:07.691774    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-2tqtm_00e6f79c-8f05-4330-aa16-1ada2ae9e9cc/console-operator/1.log"
May 25 12:58:08.586310 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:58:08.586271    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pbjnx" podUID="3e1aefc8-03d0-4659-b059-a0f859f29a75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
May 25 12:58:18.585795 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:58:18.585747    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pbjnx" podUID="3e1aefc8-03d0-4659-b059-a0f859f29a75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
May 25 12:58:28.585372 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:58:28.585328    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pbjnx" podUID="3e1aefc8-03d0-4659-b059-a0f859f29a75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
May 25 12:58:38.585355 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:58:38.585309    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pbjnx" podUID="3e1aefc8-03d0-4659-b059-a0f859f29a75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
May 25 12:58:48.586227 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:58:48.586182    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pbjnx" podUID="3e1aefc8-03d0-4659-b059-a0f859f29a75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
May 25 12:58:58.586457 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:58:58.586430    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pbjnx"
May 25 12:59:07.013239 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:07.013196    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pbjnx"]
May 25 12:59:07.013772 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:07.013543    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pbjnx" podUID="3e1aefc8-03d0-4659-b059-a0f859f29a75" containerName="kserve-container" containerID="cri-o://c154d99bbf000f46f5c77f482be41898161ec8ec7b05e410e363f3d375ff412f" gracePeriod=30
May 25 12:59:07.013772 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:07.013598    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pbjnx" podUID="3e1aefc8-03d0-4659-b059-a0f859f29a75" containerName="kube-rbac-proxy" containerID="cri-o://b9bf3e2cb2daaee362814dd592624d6bd38e031a5869b1bf2c9511ee2844ddb8" gracePeriod=30
May 25 12:59:07.096473 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:07.096442    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kl8kb"]
May 25 12:59:07.100370 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:07.100340    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kl8kb"
May 25 12:59:07.102221 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:07.102198    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
May 25 12:59:07.102357 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:07.102339    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
May 25 12:59:07.112265 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:07.112236    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kl8kb"]
May 25 12:59:07.164891 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:07.164855    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72381713-49f2-402b-ac3a-c36acdbe3ee1-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-65d8664766-kl8kb\" (UID: \"72381713-49f2-402b-ac3a-c36acdbe3ee1\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kl8kb"
May 25 12:59:07.165054 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:07.164897    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xxwl\" (UniqueName: \"kubernetes.io/projected/72381713-49f2-402b-ac3a-c36acdbe3ee1-kube-api-access-7xxwl\") pod \"sklearn-v2-mlserver-predictor-65d8664766-kl8kb\" (UID: \"72381713-49f2-402b-ac3a-c36acdbe3ee1\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kl8kb"
May 25 12:59:07.165054 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:07.164918    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72381713-49f2-402b-ac3a-c36acdbe3ee1-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-65d8664766-kl8kb\" (UID: \"72381713-49f2-402b-ac3a-c36acdbe3ee1\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kl8kb"
May 25 12:59:07.165054 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:07.165000    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72381713-49f2-402b-ac3a-c36acdbe3ee1-sklearn-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-65d8664766-kl8kb\" (UID: \"72381713-49f2-402b-ac3a-c36acdbe3ee1\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kl8kb"
May 25 12:59:07.266131 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:07.266010    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72381713-49f2-402b-ac3a-c36acdbe3ee1-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-65d8664766-kl8kb\" (UID: \"72381713-49f2-402b-ac3a-c36acdbe3ee1\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kl8kb"
May 25 12:59:07.266131 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:07.266076    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7xxwl\" (UniqueName: \"kubernetes.io/projected/72381713-49f2-402b-ac3a-c36acdbe3ee1-kube-api-access-7xxwl\") pod \"sklearn-v2-mlserver-predictor-65d8664766-kl8kb\" (UID: \"72381713-49f2-402b-ac3a-c36acdbe3ee1\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kl8kb"
May 25 12:59:07.266131 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:07.266104    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72381713-49f2-402b-ac3a-c36acdbe3ee1-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-65d8664766-kl8kb\" (UID: \"72381713-49f2-402b-ac3a-c36acdbe3ee1\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kl8kb"
May 25 12:59:07.266444 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:07.266163    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72381713-49f2-402b-ac3a-c36acdbe3ee1-sklearn-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-65d8664766-kl8kb\" (UID: \"72381713-49f2-402b-ac3a-c36acdbe3ee1\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kl8kb"
May 25 12:59:07.266708 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:07.266684    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72381713-49f2-402b-ac3a-c36acdbe3ee1-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-65d8664766-kl8kb\" (UID: \"72381713-49f2-402b-ac3a-c36acdbe3ee1\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kl8kb"
May 25 12:59:07.266888 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:07.266868    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72381713-49f2-402b-ac3a-c36acdbe3ee1-sklearn-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-65d8664766-kl8kb\" (UID: \"72381713-49f2-402b-ac3a-c36acdbe3ee1\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kl8kb"
May 25 12:59:07.268693 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:07.268669    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72381713-49f2-402b-ac3a-c36acdbe3ee1-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-65d8664766-kl8kb\" (UID: \"72381713-49f2-402b-ac3a-c36acdbe3ee1\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kl8kb"
May 25 12:59:07.273118 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:07.273093    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xxwl\" (UniqueName: \"kubernetes.io/projected/72381713-49f2-402b-ac3a-c36acdbe3ee1-kube-api-access-7xxwl\") pod \"sklearn-v2-mlserver-predictor-65d8664766-kl8kb\" (UID: \"72381713-49f2-402b-ac3a-c36acdbe3ee1\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kl8kb"
May 25 12:59:07.411623 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:07.411582    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kl8kb"
May 25 12:59:07.541534 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:07.541508    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kl8kb"]
May 25 12:59:07.543888 ip-10-0-134-255 kubenswrapper[2555]: W0525 12:59:07.543860    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72381713_49f2_402b_ac3a_c36acdbe3ee1.slice/crio-0725c984bf434fe42f0f6fbebe407fa0d6c4306d9c8775920c19baa9095ac668 WatchSource:0}: Error finding container 0725c984bf434fe42f0f6fbebe407fa0d6c4306d9c8775920c19baa9095ac668: Status 404 returned error can't find the container with id 0725c984bf434fe42f0f6fbebe407fa0d6c4306d9c8775920c19baa9095ac668
May 25 12:59:07.836294 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:07.836197    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kl8kb" event={"ID":"72381713-49f2-402b-ac3a-c36acdbe3ee1","Type":"ContainerStarted","Data":"b7f882dc27f2cabbdf1b57182ed3475c74f988da412d37690087da2f00db5281"}
May 25 12:59:07.836294 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:07.836239    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kl8kb" event={"ID":"72381713-49f2-402b-ac3a-c36acdbe3ee1","Type":"ContainerStarted","Data":"0725c984bf434fe42f0f6fbebe407fa0d6c4306d9c8775920c19baa9095ac668"}
May 25 12:59:07.838297 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:07.838270    2555 generic.go:358] "Generic (PLEG): container finished" podID="3e1aefc8-03d0-4659-b059-a0f859f29a75" containerID="b9bf3e2cb2daaee362814dd592624d6bd38e031a5869b1bf2c9511ee2844ddb8" exitCode=2
May 25 12:59:07.838410 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:07.838346    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pbjnx" event={"ID":"3e1aefc8-03d0-4659-b059-a0f859f29a75","Type":"ContainerDied","Data":"b9bf3e2cb2daaee362814dd592624d6bd38e031a5869b1bf2c9511ee2844ddb8"}
May 25 12:59:08.580428 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:08.580385    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pbjnx" podUID="3e1aefc8-03d0-4659-b059-a0f859f29a75" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.51:8643/healthz\": dial tcp 10.134.0.51:8643: connect: connection refused"
May 25 12:59:08.586275 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:08.586236    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pbjnx" podUID="3e1aefc8-03d0-4659-b059-a0f859f29a75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
May 25 12:59:11.260983 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:11.260961    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pbjnx"
May 25 12:59:11.403829 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:11.403749    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e1aefc8-03d0-4659-b059-a0f859f29a75-kserve-provision-location\") pod \"3e1aefc8-03d0-4659-b059-a0f859f29a75\" (UID: \"3e1aefc8-03d0-4659-b059-a0f859f29a75\") "
May 25 12:59:11.403829 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:11.403786    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9bbc\" (UniqueName: \"kubernetes.io/projected/3e1aefc8-03d0-4659-b059-a0f859f29a75-kube-api-access-v9bbc\") pod \"3e1aefc8-03d0-4659-b059-a0f859f29a75\" (UID: \"3e1aefc8-03d0-4659-b059-a0f859f29a75\") "
May 25 12:59:11.404057 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:11.403835    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e1aefc8-03d0-4659-b059-a0f859f29a75-proxy-tls\") pod \"3e1aefc8-03d0-4659-b059-a0f859f29a75\" (UID: \"3e1aefc8-03d0-4659-b059-a0f859f29a75\") "
May 25 12:59:11.404057 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:11.403894    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e1aefc8-03d0-4659-b059-a0f859f29a75-isvc-sklearn-kube-rbac-proxy-sar-config\") pod \"3e1aefc8-03d0-4659-b059-a0f859f29a75\" (UID: \"3e1aefc8-03d0-4659-b059-a0f859f29a75\") "
May 25 12:59:11.404173 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:11.404048    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e1aefc8-03d0-4659-b059-a0f859f29a75-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3e1aefc8-03d0-4659-b059-a0f859f29a75" (UID: "3e1aefc8-03d0-4659-b059-a0f859f29a75"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:59:11.404255 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:11.404235    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e1aefc8-03d0-4659-b059-a0f859f29a75-isvc-sklearn-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-kube-rbac-proxy-sar-config") pod "3e1aefc8-03d0-4659-b059-a0f859f29a75" (UID: "3e1aefc8-03d0-4659-b059-a0f859f29a75"). InnerVolumeSpecName "isvc-sklearn-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:59:11.405914 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:11.405889    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e1aefc8-03d0-4659-b059-a0f859f29a75-kube-api-access-v9bbc" (OuterVolumeSpecName: "kube-api-access-v9bbc") pod "3e1aefc8-03d0-4659-b059-a0f859f29a75" (UID: "3e1aefc8-03d0-4659-b059-a0f859f29a75"). InnerVolumeSpecName "kube-api-access-v9bbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:59:11.406027 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:11.405963    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e1aefc8-03d0-4659-b059-a0f859f29a75-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3e1aefc8-03d0-4659-b059-a0f859f29a75" (UID: "3e1aefc8-03d0-4659-b059-a0f859f29a75"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:59:11.504462 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:11.504436    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e1aefc8-03d0-4659-b059-a0f859f29a75-proxy-tls\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:59:11.504462 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:11.504462    2555 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e1aefc8-03d0-4659-b059-a0f859f29a75-isvc-sklearn-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:59:11.504608 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:11.504473    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e1aefc8-03d0-4659-b059-a0f859f29a75-kserve-provision-location\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:59:11.504608 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:11.504486    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9bbc\" (UniqueName: \"kubernetes.io/projected/3e1aefc8-03d0-4659-b059-a0f859f29a75-kube-api-access-v9bbc\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 12:59:11.852999 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:11.852971    2555 generic.go:358] "Generic (PLEG): container finished" podID="72381713-49f2-402b-ac3a-c36acdbe3ee1" containerID="b7f882dc27f2cabbdf1b57182ed3475c74f988da412d37690087da2f00db5281" exitCode=0
May 25 12:59:11.853168 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:11.853051    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kl8kb" event={"ID":"72381713-49f2-402b-ac3a-c36acdbe3ee1","Type":"ContainerDied","Data":"b7f882dc27f2cabbdf1b57182ed3475c74f988da412d37690087da2f00db5281"}
May 25 12:59:11.854816 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:11.854796    2555 generic.go:358] "Generic (PLEG): container finished" podID="3e1aefc8-03d0-4659-b059-a0f859f29a75" containerID="c154d99bbf000f46f5c77f482be41898161ec8ec7b05e410e363f3d375ff412f" exitCode=0
May 25 12:59:11.854912 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:11.854866    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pbjnx"
May 25 12:59:11.854912 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:11.854875    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pbjnx" event={"ID":"3e1aefc8-03d0-4659-b059-a0f859f29a75","Type":"ContainerDied","Data":"c154d99bbf000f46f5c77f482be41898161ec8ec7b05e410e363f3d375ff412f"}
May 25 12:59:11.854989 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:11.854911    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pbjnx" event={"ID":"3e1aefc8-03d0-4659-b059-a0f859f29a75","Type":"ContainerDied","Data":"c0a9564dc242443e0b2f8568dc35c6be9d6072cf2958e1c991b1d1376485794c"}
May 25 12:59:11.854989 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:11.854931    2555 scope.go:117] "RemoveContainer" containerID="b9bf3e2cb2daaee362814dd592624d6bd38e031a5869b1bf2c9511ee2844ddb8"
May 25 12:59:11.865225 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:11.865113    2555 scope.go:117] "RemoveContainer" containerID="c154d99bbf000f46f5c77f482be41898161ec8ec7b05e410e363f3d375ff412f"
May 25 12:59:11.873244 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:11.873229    2555 scope.go:117] "RemoveContainer" containerID="bdabd09fe5ef8699b308a7eecc583cd2221627c0c4370b79b403afa1bd8ec4b1"
May 25 12:59:11.881471 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:11.881456    2555 scope.go:117] "RemoveContainer" containerID="b9bf3e2cb2daaee362814dd592624d6bd38e031a5869b1bf2c9511ee2844ddb8"
May 25 12:59:11.881780 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:59:11.881754    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9bf3e2cb2daaee362814dd592624d6bd38e031a5869b1bf2c9511ee2844ddb8\": container with ID starting with b9bf3e2cb2daaee362814dd592624d6bd38e031a5869b1bf2c9511ee2844ddb8 not found: ID does not exist" containerID="b9bf3e2cb2daaee362814dd592624d6bd38e031a5869b1bf2c9511ee2844ddb8"
May 25 12:59:11.881864 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:11.881788    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9bf3e2cb2daaee362814dd592624d6bd38e031a5869b1bf2c9511ee2844ddb8"} err="failed to get container status \"b9bf3e2cb2daaee362814dd592624d6bd38e031a5869b1bf2c9511ee2844ddb8\": rpc error: code = NotFound desc = could not find container \"b9bf3e2cb2daaee362814dd592624d6bd38e031a5869b1bf2c9511ee2844ddb8\": container with ID starting with b9bf3e2cb2daaee362814dd592624d6bd38e031a5869b1bf2c9511ee2844ddb8 not found: ID does not exist"
May 25 12:59:11.881864 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:11.881810    2555 scope.go:117] "RemoveContainer" containerID="c154d99bbf000f46f5c77f482be41898161ec8ec7b05e410e363f3d375ff412f"
May 25 12:59:11.882126 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:59:11.882099    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c154d99bbf000f46f5c77f482be41898161ec8ec7b05e410e363f3d375ff412f\": container with ID starting with c154d99bbf000f46f5c77f482be41898161ec8ec7b05e410e363f3d375ff412f not found: ID does not exist" containerID="c154d99bbf000f46f5c77f482be41898161ec8ec7b05e410e363f3d375ff412f"
May 25 12:59:11.882195 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:11.882140    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c154d99bbf000f46f5c77f482be41898161ec8ec7b05e410e363f3d375ff412f"} err="failed to get container status \"c154d99bbf000f46f5c77f482be41898161ec8ec7b05e410e363f3d375ff412f\": rpc error: code = NotFound desc = could not find container \"c154d99bbf000f46f5c77f482be41898161ec8ec7b05e410e363f3d375ff412f\": container with ID starting with c154d99bbf000f46f5c77f482be41898161ec8ec7b05e410e363f3d375ff412f not found: ID does not exist"
May 25 12:59:11.882195 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:11.882166    2555 scope.go:117] "RemoveContainer" containerID="bdabd09fe5ef8699b308a7eecc583cd2221627c0c4370b79b403afa1bd8ec4b1"
May 25 12:59:11.882298 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:11.882228    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pbjnx"]
May 25 12:59:11.882481 ip-10-0-134-255 kubenswrapper[2555]: E0525 12:59:11.882457    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdabd09fe5ef8699b308a7eecc583cd2221627c0c4370b79b403afa1bd8ec4b1\": container with ID starting with bdabd09fe5ef8699b308a7eecc583cd2221627c0c4370b79b403afa1bd8ec4b1 not found: ID does not exist" containerID="bdabd09fe5ef8699b308a7eecc583cd2221627c0c4370b79b403afa1bd8ec4b1"
May 25 12:59:11.882525 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:11.882488    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdabd09fe5ef8699b308a7eecc583cd2221627c0c4370b79b403afa1bd8ec4b1"} err="failed to get container status \"bdabd09fe5ef8699b308a7eecc583cd2221627c0c4370b79b403afa1bd8ec4b1\": rpc error: code = NotFound desc = could not find container \"bdabd09fe5ef8699b308a7eecc583cd2221627c0c4370b79b403afa1bd8ec4b1\": container with ID starting with bdabd09fe5ef8699b308a7eecc583cd2221627c0c4370b79b403afa1bd8ec4b1 not found: ID does not exist"
May 25 12:59:11.886289 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:11.886271    2555 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-pbjnx"]
May 25 12:59:12.859859 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:12.859817    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kl8kb" event={"ID":"72381713-49f2-402b-ac3a-c36acdbe3ee1","Type":"ContainerStarted","Data":"dc26b6ad8678a10020ed3a6646755b2c127cc303abb5f3cb28991ebb70052614"}
May 25 12:59:12.859859 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:12.859864    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kl8kb" event={"ID":"72381713-49f2-402b-ac3a-c36acdbe3ee1","Type":"ContainerStarted","Data":"dce8627f413bd9bd16035fae3144f134b108e33a2baddea0282ea3128cd8837e"}
May 25 12:59:12.860340 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:12.860101    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kl8kb"
May 25 12:59:12.860340 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:12.860204    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kl8kb"
May 25 12:59:12.876288 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:12.876240    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kl8kb" podStartSLOduration=5.876229635 podStartE2EDuration="5.876229635s" podCreationTimestamp="2026-05-25 12:59:07 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:59:12.875035971 +0000 UTC m=+2765.835086377" watchObservedRunningTime="2026-05-25 12:59:12.876229635 +0000 UTC m=+2765.836280057"
May 25 12:59:13.600452 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:13.600419    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e1aefc8-03d0-4659-b059-a0f859f29a75" path="/var/lib/kubelet/pods/3e1aefc8-03d0-4659-b059-a0f859f29a75/volumes"
May 25 12:59:18.869576 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:18.869549    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kl8kb"
May 25 12:59:48.880072 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:48.880033    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kl8kb" podUID="72381713-49f2-402b-ac3a-c36acdbe3ee1" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
May 25 12:59:58.872421 ip-10-0-134-255 kubenswrapper[2555]: I0525 12:59:58.872390    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kl8kb"
May 25 13:00:07.201196 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:07.201164    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kl8kb"]
May 25 13:00:07.201616 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:07.201473    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kl8kb" podUID="72381713-49f2-402b-ac3a-c36acdbe3ee1" containerName="kserve-container" containerID="cri-o://dce8627f413bd9bd16035fae3144f134b108e33a2baddea0282ea3128cd8837e" gracePeriod=30
May 25 13:00:07.201616 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:07.201505    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kl8kb" podUID="72381713-49f2-402b-ac3a-c36acdbe3ee1" containerName="kube-rbac-proxy" containerID="cri-o://dc26b6ad8678a10020ed3a6646755b2c127cc303abb5f3cb28991ebb70052614" gracePeriod=30
May 25 13:00:07.299871 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:07.299837    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-99zxq"]
May 25 13:00:07.300224 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:07.300212    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e1aefc8-03d0-4659-b059-a0f859f29a75" containerName="kube-rbac-proxy"
May 25 13:00:07.300271 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:07.300226    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e1aefc8-03d0-4659-b059-a0f859f29a75" containerName="kube-rbac-proxy"
May 25 13:00:07.300271 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:07.300245    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e1aefc8-03d0-4659-b059-a0f859f29a75" containerName="kserve-container"
May 25 13:00:07.300271 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:07.300251    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e1aefc8-03d0-4659-b059-a0f859f29a75" containerName="kserve-container"
May 25 13:00:07.300271 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:07.300260    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e1aefc8-03d0-4659-b059-a0f859f29a75" containerName="storage-initializer"
May 25 13:00:07.300271 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:07.300266    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e1aefc8-03d0-4659-b059-a0f859f29a75" containerName="storage-initializer"
May 25 13:00:07.300464 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:07.300322    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="3e1aefc8-03d0-4659-b059-a0f859f29a75" containerName="kserve-container"
May 25 13:00:07.300464 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:07.300332    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="3e1aefc8-03d0-4659-b059-a0f859f29a75" containerName="kube-rbac-proxy"
May 25 13:00:07.303445 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:07.303430    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-99zxq"
May 25 13:00:07.305241 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:07.305215    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
May 25 13:00:07.305241 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:07.305218    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
May 25 13:00:07.312309 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:07.312289    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-99zxq"]
May 25 13:00:07.372395 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:07.372364    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31f8601f-4893-470d-ae33-3928f68c3419-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-56f6dfd9f9-99zxq\" (UID: \"31f8601f-4893-470d-ae33-3928f68c3419\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-99zxq"
May 25 13:00:07.372572 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:07.372425    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31f8601f-4893-470d-ae33-3928f68c3419-isvc-sklearn-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-56f6dfd9f9-99zxq\" (UID: \"31f8601f-4893-470d-ae33-3928f68c3419\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-99zxq"
May 25 13:00:07.372572 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:07.372515    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31f8601f-4893-470d-ae33-3928f68c3419-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-56f6dfd9f9-99zxq\" (UID: \"31f8601f-4893-470d-ae33-3928f68c3419\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-99zxq"
May 25 13:00:07.372719 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:07.372579    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5g8v\" (UniqueName: \"kubernetes.io/projected/31f8601f-4893-470d-ae33-3928f68c3419-kube-api-access-k5g8v\") pod \"isvc-sklearn-runtime-predictor-56f6dfd9f9-99zxq\" (UID: \"31f8601f-4893-470d-ae33-3928f68c3419\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-99zxq"
May 25 13:00:07.473623 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:07.473516    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31f8601f-4893-470d-ae33-3928f68c3419-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-56f6dfd9f9-99zxq\" (UID: \"31f8601f-4893-470d-ae33-3928f68c3419\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-99zxq"
May 25 13:00:07.473623 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:07.473594    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-k5g8v\" (UniqueName: \"kubernetes.io/projected/31f8601f-4893-470d-ae33-3928f68c3419-kube-api-access-k5g8v\") pod \"isvc-sklearn-runtime-predictor-56f6dfd9f9-99zxq\" (UID: \"31f8601f-4893-470d-ae33-3928f68c3419\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-99zxq"
May 25 13:00:07.473873 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:07.473750    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31f8601f-4893-470d-ae33-3928f68c3419-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-56f6dfd9f9-99zxq\" (UID: \"31f8601f-4893-470d-ae33-3928f68c3419\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-99zxq"
May 25 13:00:07.473873 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:07.473816    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31f8601f-4893-470d-ae33-3928f68c3419-isvc-sklearn-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-56f6dfd9f9-99zxq\" (UID: \"31f8601f-4893-470d-ae33-3928f68c3419\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-99zxq"
May 25 13:00:07.473987 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:00:07.473913    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-serving-cert: secret "isvc-sklearn-runtime-predictor-serving-cert" not found
May 25 13:00:07.473987 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:00:07.473983    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31f8601f-4893-470d-ae33-3928f68c3419-proxy-tls podName:31f8601f-4893-470d-ae33-3928f68c3419 nodeName:}" failed. No retries permitted until 2026-05-25 13:00:07.973961676 +0000 UTC m=+2820.934012078 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/31f8601f-4893-470d-ae33-3928f68c3419-proxy-tls") pod "isvc-sklearn-runtime-predictor-56f6dfd9f9-99zxq" (UID: "31f8601f-4893-470d-ae33-3928f68c3419") : secret "isvc-sklearn-runtime-predictor-serving-cert" not found
May 25 13:00:07.474078 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:07.473913    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31f8601f-4893-470d-ae33-3928f68c3419-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-56f6dfd9f9-99zxq\" (UID: \"31f8601f-4893-470d-ae33-3928f68c3419\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-99zxq"
May 25 13:00:07.474358 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:07.474339    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31f8601f-4893-470d-ae33-3928f68c3419-isvc-sklearn-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-56f6dfd9f9-99zxq\" (UID: \"31f8601f-4893-470d-ae33-3928f68c3419\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-99zxq"
May 25 13:00:07.481771 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:07.481745    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5g8v\" (UniqueName: \"kubernetes.io/projected/31f8601f-4893-470d-ae33-3928f68c3419-kube-api-access-k5g8v\") pod \"isvc-sklearn-runtime-predictor-56f6dfd9f9-99zxq\" (UID: \"31f8601f-4893-470d-ae33-3928f68c3419\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-99zxq"
May 25 13:00:07.978510 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:07.978472    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31f8601f-4893-470d-ae33-3928f68c3419-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-56f6dfd9f9-99zxq\" (UID: \"31f8601f-4893-470d-ae33-3928f68c3419\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-99zxq"
May 25 13:00:07.980926 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:07.980899    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31f8601f-4893-470d-ae33-3928f68c3419-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-56f6dfd9f9-99zxq\" (UID: \"31f8601f-4893-470d-ae33-3928f68c3419\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-99zxq"
May 25 13:00:08.048327 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:08.048244    2555 generic.go:358] "Generic (PLEG): container finished" podID="72381713-49f2-402b-ac3a-c36acdbe3ee1" containerID="dc26b6ad8678a10020ed3a6646755b2c127cc303abb5f3cb28991ebb70052614" exitCode=2
May 25 13:00:08.048456 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:08.048315    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kl8kb" event={"ID":"72381713-49f2-402b-ac3a-c36acdbe3ee1","Type":"ContainerDied","Data":"dc26b6ad8678a10020ed3a6646755b2c127cc303abb5f3cb28991ebb70052614"}
May 25 13:00:08.215780 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:08.215746    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-99zxq"
May 25 13:00:08.336591 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:08.336565    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-99zxq"]
May 25 13:00:08.865360 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:08.865319    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kl8kb" podUID="72381713-49f2-402b-ac3a-c36acdbe3ee1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.52:8643/healthz\": dial tcp 10.134.0.52:8643: connect: connection refused"
May 25 13:00:09.052658 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:09.052610    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-99zxq" event={"ID":"31f8601f-4893-470d-ae33-3928f68c3419","Type":"ContainerStarted","Data":"fb68fdf0124ad6699d38fb1e5c4e2680a7c159cb5d4cca81f4e3214138c3e8a5"}
May 25 13:00:09.052856 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:09.052669    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-99zxq" event={"ID":"31f8601f-4893-470d-ae33-3928f68c3419","Type":"ContainerStarted","Data":"697b1983546425c3610b2926fe0d778094fd12ecdf5f3df85d2aa18ea064bf3c"}
May 25 13:00:09.911808 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:09.911764    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kl8kb" podUID="72381713-49f2-402b-ac3a-c36acdbe3ee1" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.52:8080/v2/models/sklearn-v2-mlserver/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
May 25 13:00:13.865311 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:13.865270    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kl8kb" podUID="72381713-49f2-402b-ac3a-c36acdbe3ee1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.52:8643/healthz\": dial tcp 10.134.0.52:8643: connect: connection refused"
May 25 13:00:14.350456 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:14.350426    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kl8kb"
May 25 13:00:14.430857 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:14.430821    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72381713-49f2-402b-ac3a-c36acdbe3ee1-proxy-tls\") pod \"72381713-49f2-402b-ac3a-c36acdbe3ee1\" (UID: \"72381713-49f2-402b-ac3a-c36acdbe3ee1\") "
May 25 13:00:14.431044 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:14.430888    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72381713-49f2-402b-ac3a-c36acdbe3ee1-kserve-provision-location\") pod \"72381713-49f2-402b-ac3a-c36acdbe3ee1\" (UID: \"72381713-49f2-402b-ac3a-c36acdbe3ee1\") "
May 25 13:00:14.431044 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:14.430968    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7xxwl\" (UniqueName: \"kubernetes.io/projected/72381713-49f2-402b-ac3a-c36acdbe3ee1-kube-api-access-7xxwl\") pod \"72381713-49f2-402b-ac3a-c36acdbe3ee1\" (UID: \"72381713-49f2-402b-ac3a-c36acdbe3ee1\") "
May 25 13:00:14.431044 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:14.431008    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72381713-49f2-402b-ac3a-c36acdbe3ee1-sklearn-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"72381713-49f2-402b-ac3a-c36acdbe3ee1\" (UID: \"72381713-49f2-402b-ac3a-c36acdbe3ee1\") "
May 25 13:00:14.431208 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:14.431178    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72381713-49f2-402b-ac3a-c36acdbe3ee1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "72381713-49f2-402b-ac3a-c36acdbe3ee1" (UID: "72381713-49f2-402b-ac3a-c36acdbe3ee1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:00:14.431337 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:14.431319    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72381713-49f2-402b-ac3a-c36acdbe3ee1-kserve-provision-location\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:00:14.431462 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:14.431439    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72381713-49f2-402b-ac3a-c36acdbe3ee1-sklearn-v2-mlserver-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "sklearn-v2-mlserver-kube-rbac-proxy-sar-config") pod "72381713-49f2-402b-ac3a-c36acdbe3ee1" (UID: "72381713-49f2-402b-ac3a-c36acdbe3ee1"). InnerVolumeSpecName "sklearn-v2-mlserver-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:00:14.433070 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:14.433039    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72381713-49f2-402b-ac3a-c36acdbe3ee1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "72381713-49f2-402b-ac3a-c36acdbe3ee1" (UID: "72381713-49f2-402b-ac3a-c36acdbe3ee1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:00:14.433189 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:14.433129    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72381713-49f2-402b-ac3a-c36acdbe3ee1-kube-api-access-7xxwl" (OuterVolumeSpecName: "kube-api-access-7xxwl") pod "72381713-49f2-402b-ac3a-c36acdbe3ee1" (UID: "72381713-49f2-402b-ac3a-c36acdbe3ee1"). InnerVolumeSpecName "kube-api-access-7xxwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:00:14.532045 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:14.532006    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72381713-49f2-402b-ac3a-c36acdbe3ee1-proxy-tls\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:00:14.532045 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:14.532039    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xxwl\" (UniqueName: \"kubernetes.io/projected/72381713-49f2-402b-ac3a-c36acdbe3ee1-kube-api-access-7xxwl\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:00:14.532045 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:14.532052    2555 reconciler_common.go:299] "Volume detached for volume \"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72381713-49f2-402b-ac3a-c36acdbe3ee1-sklearn-v2-mlserver-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:00:15.074927 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:15.074893    2555 generic.go:358] "Generic (PLEG): container finished" podID="31f8601f-4893-470d-ae33-3928f68c3419" containerID="fb68fdf0124ad6699d38fb1e5c4e2680a7c159cb5d4cca81f4e3214138c3e8a5" exitCode=0
May 25 13:00:15.075338 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:15.074967    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-99zxq" event={"ID":"31f8601f-4893-470d-ae33-3928f68c3419","Type":"ContainerDied","Data":"fb68fdf0124ad6699d38fb1e5c4e2680a7c159cb5d4cca81f4e3214138c3e8a5"}
May 25 13:00:15.076745 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:15.076717    2555 generic.go:358] "Generic (PLEG): container finished" podID="72381713-49f2-402b-ac3a-c36acdbe3ee1" containerID="dce8627f413bd9bd16035fae3144f134b108e33a2baddea0282ea3128cd8837e" exitCode=0
May 25 13:00:15.076872 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:15.076796    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kl8kb" event={"ID":"72381713-49f2-402b-ac3a-c36acdbe3ee1","Type":"ContainerDied","Data":"dce8627f413bd9bd16035fae3144f134b108e33a2baddea0282ea3128cd8837e"}
May 25 13:00:15.076872 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:15.076819    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kl8kb"
May 25 13:00:15.076872 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:15.076834    2555 scope.go:117] "RemoveContainer" containerID="dc26b6ad8678a10020ed3a6646755b2c127cc303abb5f3cb28991ebb70052614"
May 25 13:00:15.077087 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:15.076821    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kl8kb" event={"ID":"72381713-49f2-402b-ac3a-c36acdbe3ee1","Type":"ContainerDied","Data":"0725c984bf434fe42f0f6fbebe407fa0d6c4306d9c8775920c19baa9095ac668"}
May 25 13:00:15.087745 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:15.087723    2555 scope.go:117] "RemoveContainer" containerID="dce8627f413bd9bd16035fae3144f134b108e33a2baddea0282ea3128cd8837e"
May 25 13:00:15.097507 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:15.097485    2555 scope.go:117] "RemoveContainer" containerID="b7f882dc27f2cabbdf1b57182ed3475c74f988da412d37690087da2f00db5281"
May 25 13:00:15.103730 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:15.103703    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kl8kb"]
May 25 13:00:15.106398 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:15.106358    2555 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-kl8kb"]
May 25 13:00:15.110570 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:15.110549    2555 scope.go:117] "RemoveContainer" containerID="dc26b6ad8678a10020ed3a6646755b2c127cc303abb5f3cb28991ebb70052614"
May 25 13:00:15.110936 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:00:15.110917    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc26b6ad8678a10020ed3a6646755b2c127cc303abb5f3cb28991ebb70052614\": container with ID starting with dc26b6ad8678a10020ed3a6646755b2c127cc303abb5f3cb28991ebb70052614 not found: ID does not exist" containerID="dc26b6ad8678a10020ed3a6646755b2c127cc303abb5f3cb28991ebb70052614"
May 25 13:00:15.110997 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:15.110946    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc26b6ad8678a10020ed3a6646755b2c127cc303abb5f3cb28991ebb70052614"} err="failed to get container status \"dc26b6ad8678a10020ed3a6646755b2c127cc303abb5f3cb28991ebb70052614\": rpc error: code = NotFound desc = could not find container \"dc26b6ad8678a10020ed3a6646755b2c127cc303abb5f3cb28991ebb70052614\": container with ID starting with dc26b6ad8678a10020ed3a6646755b2c127cc303abb5f3cb28991ebb70052614 not found: ID does not exist"
May 25 13:00:15.110997 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:15.110965    2555 scope.go:117] "RemoveContainer" containerID="dce8627f413bd9bd16035fae3144f134b108e33a2baddea0282ea3128cd8837e"
May 25 13:00:15.111260 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:00:15.111243    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dce8627f413bd9bd16035fae3144f134b108e33a2baddea0282ea3128cd8837e\": container with ID starting with dce8627f413bd9bd16035fae3144f134b108e33a2baddea0282ea3128cd8837e not found: ID does not exist" containerID="dce8627f413bd9bd16035fae3144f134b108e33a2baddea0282ea3128cd8837e"
May 25 13:00:15.111335 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:15.111270    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dce8627f413bd9bd16035fae3144f134b108e33a2baddea0282ea3128cd8837e"} err="failed to get container status \"dce8627f413bd9bd16035fae3144f134b108e33a2baddea0282ea3128cd8837e\": rpc error: code = NotFound desc = could not find container \"dce8627f413bd9bd16035fae3144f134b108e33a2baddea0282ea3128cd8837e\": container with ID starting with dce8627f413bd9bd16035fae3144f134b108e33a2baddea0282ea3128cd8837e not found: ID does not exist"
May 25 13:00:15.111335 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:15.111295    2555 scope.go:117] "RemoveContainer" containerID="b7f882dc27f2cabbdf1b57182ed3475c74f988da412d37690087da2f00db5281"
May 25 13:00:15.111561 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:00:15.111542    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7f882dc27f2cabbdf1b57182ed3475c74f988da412d37690087da2f00db5281\": container with ID starting with b7f882dc27f2cabbdf1b57182ed3475c74f988da412d37690087da2f00db5281 not found: ID does not exist" containerID="b7f882dc27f2cabbdf1b57182ed3475c74f988da412d37690087da2f00db5281"
May 25 13:00:15.111615 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:15.111568    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7f882dc27f2cabbdf1b57182ed3475c74f988da412d37690087da2f00db5281"} err="failed to get container status \"b7f882dc27f2cabbdf1b57182ed3475c74f988da412d37690087da2f00db5281\": rpc error: code = NotFound desc = could not find container \"b7f882dc27f2cabbdf1b57182ed3475c74f988da412d37690087da2f00db5281\": container with ID starting with b7f882dc27f2cabbdf1b57182ed3475c74f988da412d37690087da2f00db5281 not found: ID does not exist"
May 25 13:00:15.599501 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:15.599466    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72381713-49f2-402b-ac3a-c36acdbe3ee1" path="/var/lib/kubelet/pods/72381713-49f2-402b-ac3a-c36acdbe3ee1/volumes"
May 25 13:00:16.082199 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:16.082164    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-99zxq" event={"ID":"31f8601f-4893-470d-ae33-3928f68c3419","Type":"ContainerStarted","Data":"146f3846a5822c036dde4c4e3a900fdf995d5fa0555be2640ecd762363cdef2d"}
May 25 13:00:16.082658 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:16.082206    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-99zxq" event={"ID":"31f8601f-4893-470d-ae33-3928f68c3419","Type":"ContainerStarted","Data":"593274d55e9bed3d1e32a18440109f8176883f8cdb6b7cdbf2f1ae136084a6ab"}
May 25 13:00:16.082658 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:16.082411    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-99zxq"
May 25 13:00:16.098623 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:16.098577    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-99zxq" podStartSLOduration=9.098557577 podStartE2EDuration="9.098557577s" podCreationTimestamp="2026-05-25 13:00:07 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:00:16.097428013 +0000 UTC m=+2829.057478453" watchObservedRunningTime="2026-05-25 13:00:16.098557577 +0000 UTC m=+2829.058607983"
May 25 13:00:17.087388 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:17.087351    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-99zxq"
May 25 13:00:17.088616 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:17.088588    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-99zxq" podUID="31f8601f-4893-470d-ae33-3928f68c3419" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
May 25 13:00:18.091540 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:18.091503    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-99zxq" podUID="31f8601f-4893-470d-ae33-3928f68c3419" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
May 25 13:00:23.096200 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:23.096168    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-99zxq"
May 25 13:00:23.096708 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:23.096680    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-99zxq" podUID="31f8601f-4893-470d-ae33-3928f68c3419" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
May 25 13:00:33.097783 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:33.097740    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-99zxq"
May 25 13:00:44.308473 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:44.308440    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-56f6dfd9f9-99zxq_31f8601f-4893-470d-ae33-3928f68c3419/kserve-container/0.log"
May 25 13:00:44.433613 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:44.433579    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-99zxq"]
May 25 13:00:44.434005 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:44.433960    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-99zxq" podUID="31f8601f-4893-470d-ae33-3928f68c3419" containerName="kserve-container" containerID="cri-o://593274d55e9bed3d1e32a18440109f8176883f8cdb6b7cdbf2f1ae136084a6ab" gracePeriod=30
May 25 13:00:44.434124 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:44.433998    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-99zxq" podUID="31f8601f-4893-470d-ae33-3928f68c3419" containerName="kube-rbac-proxy" containerID="cri-o://146f3846a5822c036dde4c4e3a900fdf995d5fa0555be2640ecd762363cdef2d" gracePeriod=30
May 25 13:00:44.527549 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:44.527512    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-5pwt4"]
May 25 13:00:44.527949 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:44.527933    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72381713-49f2-402b-ac3a-c36acdbe3ee1" containerName="kube-rbac-proxy"
May 25 13:00:44.528017 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:44.527953    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="72381713-49f2-402b-ac3a-c36acdbe3ee1" containerName="kube-rbac-proxy"
May 25 13:00:44.528017 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:44.527964    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72381713-49f2-402b-ac3a-c36acdbe3ee1" containerName="storage-initializer"
May 25 13:00:44.528017 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:44.527969    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="72381713-49f2-402b-ac3a-c36acdbe3ee1" containerName="storage-initializer"
May 25 13:00:44.528017 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:44.527976    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72381713-49f2-402b-ac3a-c36acdbe3ee1" containerName="kserve-container"
May 25 13:00:44.528017 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:44.527982    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="72381713-49f2-402b-ac3a-c36acdbe3ee1" containerName="kserve-container"
May 25 13:00:44.528184 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:44.528041    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="72381713-49f2-402b-ac3a-c36acdbe3ee1" containerName="kube-rbac-proxy"
May 25 13:00:44.528184 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:44.528052    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="72381713-49f2-402b-ac3a-c36acdbe3ee1" containerName="kserve-container"
May 25 13:00:44.530378 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:44.530360    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-5pwt4"
May 25 13:00:44.532294 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:44.532269    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
May 25 13:00:44.532425 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:44.532369    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
May 25 13:00:44.538591 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:44.538561    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-5pwt4"]
May 25 13:00:44.598616 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:44.598517    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afa89f41-d774-4590-8679-b32702331950-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-5pwt4\" (UID: \"afa89f41-d774-4590-8679-b32702331950\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-5pwt4"
May 25 13:00:44.598616 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:44.598588    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afa89f41-d774-4590-8679-b32702331950-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-5pwt4\" (UID: \"afa89f41-d774-4590-8679-b32702331950\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-5pwt4"
May 25 13:00:44.598847 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:44.598710    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afa89f41-d774-4590-8679-b32702331950-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-5pwt4\" (UID: \"afa89f41-d774-4590-8679-b32702331950\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-5pwt4"
May 25 13:00:44.598847 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:44.598740    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcsfj\" (UniqueName: \"kubernetes.io/projected/afa89f41-d774-4590-8679-b32702331950-kube-api-access-zcsfj\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-5pwt4\" (UID: \"afa89f41-d774-4590-8679-b32702331950\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-5pwt4"
May 25 13:00:44.699904 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:44.699859    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afa89f41-d774-4590-8679-b32702331950-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-5pwt4\" (UID: \"afa89f41-d774-4590-8679-b32702331950\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-5pwt4"
May 25 13:00:44.699904 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:44.699905    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zcsfj\" (UniqueName: \"kubernetes.io/projected/afa89f41-d774-4590-8679-b32702331950-kube-api-access-zcsfj\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-5pwt4\" (UID: \"afa89f41-d774-4590-8679-b32702331950\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-5pwt4"
May 25 13:00:44.700173 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:44.699979    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afa89f41-d774-4590-8679-b32702331950-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-5pwt4\" (UID: \"afa89f41-d774-4590-8679-b32702331950\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-5pwt4"
May 25 13:00:44.700173 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:44.700028    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afa89f41-d774-4590-8679-b32702331950-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-5pwt4\" (UID: \"afa89f41-d774-4590-8679-b32702331950\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-5pwt4"
May 25 13:00:44.700496 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:44.700465    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afa89f41-d774-4590-8679-b32702331950-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-5pwt4\" (UID: \"afa89f41-d774-4590-8679-b32702331950\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-5pwt4"
May 25 13:00:44.700749 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:44.700726    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afa89f41-d774-4590-8679-b32702331950-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-5pwt4\" (UID: \"afa89f41-d774-4590-8679-b32702331950\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-5pwt4"
May 25 13:00:44.702566 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:44.702544    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afa89f41-d774-4590-8679-b32702331950-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-5pwt4\" (UID: \"afa89f41-d774-4590-8679-b32702331950\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-5pwt4"
May 25 13:00:44.706806 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:44.706781    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcsfj\" (UniqueName: \"kubernetes.io/projected/afa89f41-d774-4590-8679-b32702331950-kube-api-access-zcsfj\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-5pwt4\" (UID: \"afa89f41-d774-4590-8679-b32702331950\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-5pwt4"
May 25 13:00:44.843108 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:44.843064    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-5pwt4"
May 25 13:00:44.982942 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:44.982909    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-5pwt4"]
May 25 13:00:44.986760 ip-10-0-134-255 kubenswrapper[2555]: W0525 13:00:44.986729    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafa89f41_d774_4590_8679_b32702331950.slice/crio-6b5f4c6ff517377a7601713d3c9a65b209281c7ba6b9e30b48c0d13223fa8843 WatchSource:0}: Error finding container 6b5f4c6ff517377a7601713d3c9a65b209281c7ba6b9e30b48c0d13223fa8843: Status 404 returned error can't find the container with id 6b5f4c6ff517377a7601713d3c9a65b209281c7ba6b9e30b48c0d13223fa8843
May 25 13:00:45.194281 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:45.194242    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-5pwt4" event={"ID":"afa89f41-d774-4590-8679-b32702331950","Type":"ContainerStarted","Data":"40fc1e101fa0d976b03274b80970c9698eac0e89390ea336f5d45ccd627b5418"}
May 25 13:00:45.194281 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:45.194285    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-5pwt4" event={"ID":"afa89f41-d774-4590-8679-b32702331950","Type":"ContainerStarted","Data":"6b5f4c6ff517377a7601713d3c9a65b209281c7ba6b9e30b48c0d13223fa8843"}
May 25 13:00:45.203201 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:45.203169    2555 generic.go:358] "Generic (PLEG): container finished" podID="31f8601f-4893-470d-ae33-3928f68c3419" containerID="146f3846a5822c036dde4c4e3a900fdf995d5fa0555be2640ecd762363cdef2d" exitCode=2
May 25 13:00:45.203353 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:45.203223    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-99zxq" event={"ID":"31f8601f-4893-470d-ae33-3928f68c3419","Type":"ContainerDied","Data":"146f3846a5822c036dde4c4e3a900fdf995d5fa0555be2640ecd762363cdef2d"}
May 25 13:00:45.368070 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:45.368044    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-99zxq"
May 25 13:00:45.406630 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:45.406564    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31f8601f-4893-470d-ae33-3928f68c3419-proxy-tls\") pod \"31f8601f-4893-470d-ae33-3928f68c3419\" (UID: \"31f8601f-4893-470d-ae33-3928f68c3419\") "
May 25 13:00:45.406630 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:45.406603    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k5g8v\" (UniqueName: \"kubernetes.io/projected/31f8601f-4893-470d-ae33-3928f68c3419-kube-api-access-k5g8v\") pod \"31f8601f-4893-470d-ae33-3928f68c3419\" (UID: \"31f8601f-4893-470d-ae33-3928f68c3419\") "
May 25 13:00:45.406933 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:45.406667    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31f8601f-4893-470d-ae33-3928f68c3419-isvc-sklearn-runtime-kube-rbac-proxy-sar-config\") pod \"31f8601f-4893-470d-ae33-3928f68c3419\" (UID: \"31f8601f-4893-470d-ae33-3928f68c3419\") "
May 25 13:00:45.406933 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:45.406707    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31f8601f-4893-470d-ae33-3928f68c3419-kserve-provision-location\") pod \"31f8601f-4893-470d-ae33-3928f68c3419\" (UID: \"31f8601f-4893-470d-ae33-3928f68c3419\") "
May 25 13:00:45.407101 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:45.407072    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31f8601f-4893-470d-ae33-3928f68c3419-isvc-sklearn-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-runtime-kube-rbac-proxy-sar-config") pod "31f8601f-4893-470d-ae33-3928f68c3419" (UID: "31f8601f-4893-470d-ae33-3928f68c3419"). InnerVolumeSpecName "isvc-sklearn-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:00:45.408808 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:45.408782    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31f8601f-4893-470d-ae33-3928f68c3419-kube-api-access-k5g8v" (OuterVolumeSpecName: "kube-api-access-k5g8v") pod "31f8601f-4893-470d-ae33-3928f68c3419" (UID: "31f8601f-4893-470d-ae33-3928f68c3419"). InnerVolumeSpecName "kube-api-access-k5g8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:00:45.409058 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:45.409030    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31f8601f-4893-470d-ae33-3928f68c3419-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "31f8601f-4893-470d-ae33-3928f68c3419" (UID: "31f8601f-4893-470d-ae33-3928f68c3419"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:00:45.433051 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:45.433021    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31f8601f-4893-470d-ae33-3928f68c3419-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "31f8601f-4893-470d-ae33-3928f68c3419" (UID: "31f8601f-4893-470d-ae33-3928f68c3419"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:00:45.508158 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:45.508124    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31f8601f-4893-470d-ae33-3928f68c3419-proxy-tls\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:00:45.508158 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:45.508153    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5g8v\" (UniqueName: \"kubernetes.io/projected/31f8601f-4893-470d-ae33-3928f68c3419-kube-api-access-k5g8v\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:00:45.508158 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:45.508165    2555 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31f8601f-4893-470d-ae33-3928f68c3419-isvc-sklearn-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:00:45.508384 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:45.508177    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31f8601f-4893-470d-ae33-3928f68c3419-kserve-provision-location\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:00:46.208210 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:46.208171    2555 generic.go:358] "Generic (PLEG): container finished" podID="31f8601f-4893-470d-ae33-3928f68c3419" containerID="593274d55e9bed3d1e32a18440109f8176883f8cdb6b7cdbf2f1ae136084a6ab" exitCode=0
May 25 13:00:46.208344 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:46.208262    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-99zxq"
May 25 13:00:46.208344 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:46.208260    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-99zxq" event={"ID":"31f8601f-4893-470d-ae33-3928f68c3419","Type":"ContainerDied","Data":"593274d55e9bed3d1e32a18440109f8176883f8cdb6b7cdbf2f1ae136084a6ab"}
May 25 13:00:46.208344 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:46.208309    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-99zxq" event={"ID":"31f8601f-4893-470d-ae33-3928f68c3419","Type":"ContainerDied","Data":"697b1983546425c3610b2926fe0d778094fd12ecdf5f3df85d2aa18ea064bf3c"}
May 25 13:00:46.208344 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:46.208329    2555 scope.go:117] "RemoveContainer" containerID="146f3846a5822c036dde4c4e3a900fdf995d5fa0555be2640ecd762363cdef2d"
May 25 13:00:46.217198 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:46.217178    2555 scope.go:117] "RemoveContainer" containerID="593274d55e9bed3d1e32a18440109f8176883f8cdb6b7cdbf2f1ae136084a6ab"
May 25 13:00:46.224293 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:46.224268    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-99zxq"]
May 25 13:00:46.225149 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:46.225128    2555 scope.go:117] "RemoveContainer" containerID="fb68fdf0124ad6699d38fb1e5c4e2680a7c159cb5d4cca81f4e3214138c3e8a5"
May 25 13:00:46.226164 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:46.226145    2555 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-99zxq"]
May 25 13:00:46.232545 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:46.232529    2555 scope.go:117] "RemoveContainer" containerID="146f3846a5822c036dde4c4e3a900fdf995d5fa0555be2640ecd762363cdef2d"
May 25 13:00:46.232868 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:00:46.232850    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"146f3846a5822c036dde4c4e3a900fdf995d5fa0555be2640ecd762363cdef2d\": container with ID starting with 146f3846a5822c036dde4c4e3a900fdf995d5fa0555be2640ecd762363cdef2d not found: ID does not exist" containerID="146f3846a5822c036dde4c4e3a900fdf995d5fa0555be2640ecd762363cdef2d"
May 25 13:00:46.232918 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:46.232877    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"146f3846a5822c036dde4c4e3a900fdf995d5fa0555be2640ecd762363cdef2d"} err="failed to get container status \"146f3846a5822c036dde4c4e3a900fdf995d5fa0555be2640ecd762363cdef2d\": rpc error: code = NotFound desc = could not find container \"146f3846a5822c036dde4c4e3a900fdf995d5fa0555be2640ecd762363cdef2d\": container with ID starting with 146f3846a5822c036dde4c4e3a900fdf995d5fa0555be2640ecd762363cdef2d not found: ID does not exist"
May 25 13:00:46.232918 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:46.232897    2555 scope.go:117] "RemoveContainer" containerID="593274d55e9bed3d1e32a18440109f8176883f8cdb6b7cdbf2f1ae136084a6ab"
May 25 13:00:46.233143 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:00:46.233125    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"593274d55e9bed3d1e32a18440109f8176883f8cdb6b7cdbf2f1ae136084a6ab\": container with ID starting with 593274d55e9bed3d1e32a18440109f8176883f8cdb6b7cdbf2f1ae136084a6ab not found: ID does not exist" containerID="593274d55e9bed3d1e32a18440109f8176883f8cdb6b7cdbf2f1ae136084a6ab"
May 25 13:00:46.233209 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:46.233153    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"593274d55e9bed3d1e32a18440109f8176883f8cdb6b7cdbf2f1ae136084a6ab"} err="failed to get container status \"593274d55e9bed3d1e32a18440109f8176883f8cdb6b7cdbf2f1ae136084a6ab\": rpc error: code = NotFound desc = could not find container \"593274d55e9bed3d1e32a18440109f8176883f8cdb6b7cdbf2f1ae136084a6ab\": container with ID starting with 593274d55e9bed3d1e32a18440109f8176883f8cdb6b7cdbf2f1ae136084a6ab not found: ID does not exist"
May 25 13:00:46.233209 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:46.233176    2555 scope.go:117] "RemoveContainer" containerID="fb68fdf0124ad6699d38fb1e5c4e2680a7c159cb5d4cca81f4e3214138c3e8a5"
May 25 13:00:46.233434 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:00:46.233416    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb68fdf0124ad6699d38fb1e5c4e2680a7c159cb5d4cca81f4e3214138c3e8a5\": container with ID starting with fb68fdf0124ad6699d38fb1e5c4e2680a7c159cb5d4cca81f4e3214138c3e8a5 not found: ID does not exist" containerID="fb68fdf0124ad6699d38fb1e5c4e2680a7c159cb5d4cca81f4e3214138c3e8a5"
May 25 13:00:46.233477 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:46.233442    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb68fdf0124ad6699d38fb1e5c4e2680a7c159cb5d4cca81f4e3214138c3e8a5"} err="failed to get container status \"fb68fdf0124ad6699d38fb1e5c4e2680a7c159cb5d4cca81f4e3214138c3e8a5\": rpc error: code = NotFound desc = could not find container \"fb68fdf0124ad6699d38fb1e5c4e2680a7c159cb5d4cca81f4e3214138c3e8a5\": container with ID starting with fb68fdf0124ad6699d38fb1e5c4e2680a7c159cb5d4cca81f4e3214138c3e8a5 not found: ID does not exist"
May 25 13:00:47.600080 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:47.600046    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31f8601f-4893-470d-ae33-3928f68c3419" path="/var/lib/kubelet/pods/31f8601f-4893-470d-ae33-3928f68c3419/volumes"
May 25 13:00:49.220423 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:49.220391    2555 generic.go:358] "Generic (PLEG): container finished" podID="afa89f41-d774-4590-8679-b32702331950" containerID="40fc1e101fa0d976b03274b80970c9698eac0e89390ea336f5d45ccd627b5418" exitCode=0
May 25 13:00:49.220872 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:49.220467    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-5pwt4" event={"ID":"afa89f41-d774-4590-8679-b32702331950","Type":"ContainerDied","Data":"40fc1e101fa0d976b03274b80970c9698eac0e89390ea336f5d45ccd627b5418"}
May 25 13:00:50.225214 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:50.225178    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-5pwt4" event={"ID":"afa89f41-d774-4590-8679-b32702331950","Type":"ContainerStarted","Data":"284134e40d18e3730f386e6b128c3e2c0d17336ceda64ab8084852a0e3acf88a"}
May 25 13:00:50.225214 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:50.225213    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-5pwt4" event={"ID":"afa89f41-d774-4590-8679-b32702331950","Type":"ContainerStarted","Data":"24bedb235a86a42209e9572ce3a346574146a076c243103b994ad5084d1cbadb"}
May 25 13:00:50.225721 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:50.225440    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-5pwt4"
May 25 13:00:50.225721 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:50.225468    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-5pwt4"
May 25 13:00:50.243930 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:50.243883    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-5pwt4" podStartSLOduration=6.243868738 podStartE2EDuration="6.243868738s" podCreationTimestamp="2026-05-25 13:00:44 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:00:50.241821594 +0000 UTC m=+2863.201872000" watchObservedRunningTime="2026-05-25 13:00:50.243868738 +0000 UTC m=+2863.203919144"
May 25 13:00:56.234165 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:00:56.234137    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-5pwt4"
May 25 13:01:26.280393 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:26.280335    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-5pwt4" podUID="afa89f41-d774-4590-8679-b32702331950" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
May 25 13:01:36.237336 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:36.237304    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-5pwt4"
May 25 13:01:44.607879 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:44.607849    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-5pwt4"]
May 25 13:01:44.608280 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:44.608123    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-5pwt4" podUID="afa89f41-d774-4590-8679-b32702331950" containerName="kserve-container" containerID="cri-o://24bedb235a86a42209e9572ce3a346574146a076c243103b994ad5084d1cbadb" gracePeriod=30
May 25 13:01:44.608280 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:44.608180    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-5pwt4" podUID="afa89f41-d774-4590-8679-b32702331950" containerName="kube-rbac-proxy" containerID="cri-o://284134e40d18e3730f386e6b128c3e2c0d17336ceda64ab8084852a0e3acf88a" gracePeriod=30
May 25 13:01:44.696225 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:44.696187    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-s2fzk"]
May 25 13:01:44.697066 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:44.697039    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31f8601f-4893-470d-ae33-3928f68c3419" containerName="kube-rbac-proxy"
May 25 13:01:44.697066 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:44.697066    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="31f8601f-4893-470d-ae33-3928f68c3419" containerName="kube-rbac-proxy"
May 25 13:01:44.697234 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:44.697125    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31f8601f-4893-470d-ae33-3928f68c3419" containerName="kserve-container"
May 25 13:01:44.697234 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:44.697135    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="31f8601f-4893-470d-ae33-3928f68c3419" containerName="kserve-container"
May 25 13:01:44.697234 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:44.697166    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31f8601f-4893-470d-ae33-3928f68c3419" containerName="storage-initializer"
May 25 13:01:44.697234 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:44.697174    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="31f8601f-4893-470d-ae33-3928f68c3419" containerName="storage-initializer"
May 25 13:01:44.697424 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:44.697335    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="31f8601f-4893-470d-ae33-3928f68c3419" containerName="kserve-container"
May 25 13:01:44.697424 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:44.697360    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="31f8601f-4893-470d-ae33-3928f68c3419" containerName="kube-rbac-proxy"
May 25 13:01:44.702297 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:44.702276    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-s2fzk"
May 25 13:01:44.704183 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:44.704151    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
May 25 13:01:44.704183 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:44.704286    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-predictor-serving-cert\""
May 25 13:01:44.706812 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:44.706788    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-s2fzk"]
May 25 13:01:44.796097 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:44.796073    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d13f3b53-90a6-483f-bde3-976c62e4c10d-proxy-tls\") pod \"isvc-sklearn-v2-predictor-9df9c96c7-s2fzk\" (UID: \"d13f3b53-90a6-483f-bde3-976c62e4c10d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-s2fzk"
May 25 13:01:44.796229 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:44.796115    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhtdg\" (UniqueName: \"kubernetes.io/projected/d13f3b53-90a6-483f-bde3-976c62e4c10d-kube-api-access-dhtdg\") pod \"isvc-sklearn-v2-predictor-9df9c96c7-s2fzk\" (UID: \"d13f3b53-90a6-483f-bde3-976c62e4c10d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-s2fzk"
May 25 13:01:44.796229 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:44.796136    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d13f3b53-90a6-483f-bde3-976c62e4c10d-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-9df9c96c7-s2fzk\" (UID: \"d13f3b53-90a6-483f-bde3-976c62e4c10d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-s2fzk"
May 25 13:01:44.796229 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:44.796156    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d13f3b53-90a6-483f-bde3-976c62e4c10d-isvc-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-9df9c96c7-s2fzk\" (UID: \"d13f3b53-90a6-483f-bde3-976c62e4c10d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-s2fzk"
May 25 13:01:44.897168 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:44.897096    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d13f3b53-90a6-483f-bde3-976c62e4c10d-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-9df9c96c7-s2fzk\" (UID: \"d13f3b53-90a6-483f-bde3-976c62e4c10d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-s2fzk"
May 25 13:01:44.897168 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:44.897127    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d13f3b53-90a6-483f-bde3-976c62e4c10d-isvc-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-9df9c96c7-s2fzk\" (UID: \"d13f3b53-90a6-483f-bde3-976c62e4c10d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-s2fzk"
May 25 13:01:44.897374 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:44.897189    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d13f3b53-90a6-483f-bde3-976c62e4c10d-proxy-tls\") pod \"isvc-sklearn-v2-predictor-9df9c96c7-s2fzk\" (UID: \"d13f3b53-90a6-483f-bde3-976c62e4c10d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-s2fzk"
May 25 13:01:44.897374 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:44.897220    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dhtdg\" (UniqueName: \"kubernetes.io/projected/d13f3b53-90a6-483f-bde3-976c62e4c10d-kube-api-access-dhtdg\") pod \"isvc-sklearn-v2-predictor-9df9c96c7-s2fzk\" (UID: \"d13f3b53-90a6-483f-bde3-976c62e4c10d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-s2fzk"
May 25 13:01:44.897564 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:44.897539    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d13f3b53-90a6-483f-bde3-976c62e4c10d-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-9df9c96c7-s2fzk\" (UID: \"d13f3b53-90a6-483f-bde3-976c62e4c10d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-s2fzk"
May 25 13:01:44.897877 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:44.897856    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d13f3b53-90a6-483f-bde3-976c62e4c10d-isvc-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-9df9c96c7-s2fzk\" (UID: \"d13f3b53-90a6-483f-bde3-976c62e4c10d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-s2fzk"
May 25 13:01:44.899744 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:44.899720    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d13f3b53-90a6-483f-bde3-976c62e4c10d-proxy-tls\") pod \"isvc-sklearn-v2-predictor-9df9c96c7-s2fzk\" (UID: \"d13f3b53-90a6-483f-bde3-976c62e4c10d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-s2fzk"
May 25 13:01:44.904014 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:44.903987    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhtdg\" (UniqueName: \"kubernetes.io/projected/d13f3b53-90a6-483f-bde3-976c62e4c10d-kube-api-access-dhtdg\") pod \"isvc-sklearn-v2-predictor-9df9c96c7-s2fzk\" (UID: \"d13f3b53-90a6-483f-bde3-976c62e4c10d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-s2fzk"
May 25 13:01:45.013879 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:45.013850    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-s2fzk"
May 25 13:01:45.136830 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:45.136800    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-s2fzk"]
May 25 13:01:45.138751 ip-10-0-134-255 kubenswrapper[2555]: W0525 13:01:45.138720    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd13f3b53_90a6_483f_bde3_976c62e4c10d.slice/crio-1070533fe209f6f412bd5c6b08b5a3831dfb1f752438b696a12e86bc4be611ec WatchSource:0}: Error finding container 1070533fe209f6f412bd5c6b08b5a3831dfb1f752438b696a12e86bc4be611ec: Status 404 returned error can't find the container with id 1070533fe209f6f412bd5c6b08b5a3831dfb1f752438b696a12e86bc4be611ec
May 25 13:01:45.416114 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:45.416036    2555 generic.go:358] "Generic (PLEG): container finished" podID="afa89f41-d774-4590-8679-b32702331950" containerID="284134e40d18e3730f386e6b128c3e2c0d17336ceda64ab8084852a0e3acf88a" exitCode=2
May 25 13:01:45.416325 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:45.416119    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-5pwt4" event={"ID":"afa89f41-d774-4590-8679-b32702331950","Type":"ContainerDied","Data":"284134e40d18e3730f386e6b128c3e2c0d17336ceda64ab8084852a0e3acf88a"}
May 25 13:01:45.417439 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:45.417417    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-s2fzk" event={"ID":"d13f3b53-90a6-483f-bde3-976c62e4c10d","Type":"ContainerStarted","Data":"d100f7678f5389b2a595f3d3be07d9f4857ad5e5520bc4f408276e9ea0ed0d4b"}
May 25 13:01:45.417559 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:45.417443    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-s2fzk" event={"ID":"d13f3b53-90a6-483f-bde3-976c62e4c10d","Type":"ContainerStarted","Data":"1070533fe209f6f412bd5c6b08b5a3831dfb1f752438b696a12e86bc4be611ec"}
May 25 13:01:46.229325 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:46.229286    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-5pwt4" podUID="afa89f41-d774-4590-8679-b32702331950" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.54:8643/healthz\": dial tcp 10.134.0.54:8643: connect: connection refused"
May 25 13:01:47.275786 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:47.275741    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-5pwt4" podUID="afa89f41-d774-4590-8679-b32702331950" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.54:8080/v2/models/isvc-sklearn-v2-runtime/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
May 25 13:01:49.432733 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:49.432698    2555 generic.go:358] "Generic (PLEG): container finished" podID="d13f3b53-90a6-483f-bde3-976c62e4c10d" containerID="d100f7678f5389b2a595f3d3be07d9f4857ad5e5520bc4f408276e9ea0ed0d4b" exitCode=0
May 25 13:01:49.433126 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:49.432771    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-s2fzk" event={"ID":"d13f3b53-90a6-483f-bde3-976c62e4c10d","Type":"ContainerDied","Data":"d100f7678f5389b2a595f3d3be07d9f4857ad5e5520bc4f408276e9ea0ed0d4b"}
May 25 13:01:50.438289 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:50.438248    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-s2fzk" event={"ID":"d13f3b53-90a6-483f-bde3-976c62e4c10d","Type":"ContainerStarted","Data":"1d651d8aa3730e3cbf5589a8bc270424a19f6e2fbbc01f9e2efaed364b65b38f"}
May 25 13:01:50.438679 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:50.438298    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-s2fzk" event={"ID":"d13f3b53-90a6-483f-bde3-976c62e4c10d","Type":"ContainerStarted","Data":"97f2c1a21d114e1c3f2c79c0db4f14c9e9ac7e0b17ac3dbb78cc425b41a3ae8a"}
May 25 13:01:50.438679 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:50.438526    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-s2fzk"
May 25 13:01:50.456240 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:50.456194    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-s2fzk" podStartSLOduration=6.456182148 podStartE2EDuration="6.456182148s" podCreationTimestamp="2026-05-25 13:01:44 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:01:50.455305562 +0000 UTC m=+2923.415355968" watchObservedRunningTime="2026-05-25 13:01:50.456182148 +0000 UTC m=+2923.416232556"
May 25 13:01:51.045851 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:51.045829    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-5pwt4"
May 25 13:01:51.145879 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:51.145805    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zcsfj\" (UniqueName: \"kubernetes.io/projected/afa89f41-d774-4590-8679-b32702331950-kube-api-access-zcsfj\") pod \"afa89f41-d774-4590-8679-b32702331950\" (UID: \"afa89f41-d774-4590-8679-b32702331950\") "
May 25 13:01:51.145879 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:51.145843    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afa89f41-d774-4590-8679-b32702331950-kserve-provision-location\") pod \"afa89f41-d774-4590-8679-b32702331950\" (UID: \"afa89f41-d774-4590-8679-b32702331950\") "
May 25 13:01:51.146094 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:51.145884    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afa89f41-d774-4590-8679-b32702331950-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\") pod \"afa89f41-d774-4590-8679-b32702331950\" (UID: \"afa89f41-d774-4590-8679-b32702331950\") "
May 25 13:01:51.146094 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:51.145920    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afa89f41-d774-4590-8679-b32702331950-proxy-tls\") pod \"afa89f41-d774-4590-8679-b32702331950\" (UID: \"afa89f41-d774-4590-8679-b32702331950\") "
May 25 13:01:51.146208 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:51.146182    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afa89f41-d774-4590-8679-b32702331950-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "afa89f41-d774-4590-8679-b32702331950" (UID: "afa89f41-d774-4590-8679-b32702331950"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:01:51.146253 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:51.146213    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afa89f41-d774-4590-8679-b32702331950-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config") pod "afa89f41-d774-4590-8679-b32702331950" (UID: "afa89f41-d774-4590-8679-b32702331950"). InnerVolumeSpecName "isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:01:51.147923 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:51.147896    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afa89f41-d774-4590-8679-b32702331950-kube-api-access-zcsfj" (OuterVolumeSpecName: "kube-api-access-zcsfj") pod "afa89f41-d774-4590-8679-b32702331950" (UID: "afa89f41-d774-4590-8679-b32702331950"). InnerVolumeSpecName "kube-api-access-zcsfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:01:51.148039 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:51.147945    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afa89f41-d774-4590-8679-b32702331950-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "afa89f41-d774-4590-8679-b32702331950" (UID: "afa89f41-d774-4590-8679-b32702331950"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:01:51.246989 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:51.246963    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcsfj\" (UniqueName: \"kubernetes.io/projected/afa89f41-d774-4590-8679-b32702331950-kube-api-access-zcsfj\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:01:51.246989 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:51.246985    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afa89f41-d774-4590-8679-b32702331950-kserve-provision-location\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:01:51.247136 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:51.246996    2555 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afa89f41-d774-4590-8679-b32702331950-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:01:51.247136 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:51.247006    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afa89f41-d774-4590-8679-b32702331950-proxy-tls\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:01:51.443905 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:51.443876    2555 generic.go:358] "Generic (PLEG): container finished" podID="afa89f41-d774-4590-8679-b32702331950" containerID="24bedb235a86a42209e9572ce3a346574146a076c243103b994ad5084d1cbadb" exitCode=0
May 25 13:01:51.444318 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:51.443970    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-5pwt4"
May 25 13:01:51.444318 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:51.443968    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-5pwt4" event={"ID":"afa89f41-d774-4590-8679-b32702331950","Type":"ContainerDied","Data":"24bedb235a86a42209e9572ce3a346574146a076c243103b994ad5084d1cbadb"}
May 25 13:01:51.444318 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:51.444082    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-5pwt4" event={"ID":"afa89f41-d774-4590-8679-b32702331950","Type":"ContainerDied","Data":"6b5f4c6ff517377a7601713d3c9a65b209281c7ba6b9e30b48c0d13223fa8843"}
May 25 13:01:51.444318 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:51.444098    2555 scope.go:117] "RemoveContainer" containerID="284134e40d18e3730f386e6b128c3e2c0d17336ceda64ab8084852a0e3acf88a"
May 25 13:01:51.444575 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:51.444559    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-s2fzk"
May 25 13:01:51.446042 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:51.446017    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-s2fzk" podUID="d13f3b53-90a6-483f-bde3-976c62e4c10d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
May 25 13:01:51.452737 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:51.452718    2555 scope.go:117] "RemoveContainer" containerID="24bedb235a86a42209e9572ce3a346574146a076c243103b994ad5084d1cbadb"
May 25 13:01:51.459871 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:51.459850    2555 scope.go:117] "RemoveContainer" containerID="40fc1e101fa0d976b03274b80970c9698eac0e89390ea336f5d45ccd627b5418"
May 25 13:01:51.465723 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:51.465702    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-5pwt4"]
May 25 13:01:51.468399 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:51.468377    2555 scope.go:117] "RemoveContainer" containerID="284134e40d18e3730f386e6b128c3e2c0d17336ceda64ab8084852a0e3acf88a"
May 25 13:01:51.468659 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:01:51.468627    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"284134e40d18e3730f386e6b128c3e2c0d17336ceda64ab8084852a0e3acf88a\": container with ID starting with 284134e40d18e3730f386e6b128c3e2c0d17336ceda64ab8084852a0e3acf88a not found: ID does not exist" containerID="284134e40d18e3730f386e6b128c3e2c0d17336ceda64ab8084852a0e3acf88a"
May 25 13:01:51.468727 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:51.468668    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"284134e40d18e3730f386e6b128c3e2c0d17336ceda64ab8084852a0e3acf88a"} err="failed to get container status \"284134e40d18e3730f386e6b128c3e2c0d17336ceda64ab8084852a0e3acf88a\": rpc error: code = NotFound desc = could not find container \"284134e40d18e3730f386e6b128c3e2c0d17336ceda64ab8084852a0e3acf88a\": container with ID starting with 284134e40d18e3730f386e6b128c3e2c0d17336ceda64ab8084852a0e3acf88a not found: ID does not exist"
May 25 13:01:51.468727 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:51.468686    2555 scope.go:117] "RemoveContainer" containerID="24bedb235a86a42209e9572ce3a346574146a076c243103b994ad5084d1cbadb"
May 25 13:01:51.468953 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:01:51.468929    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24bedb235a86a42209e9572ce3a346574146a076c243103b994ad5084d1cbadb\": container with ID starting with 24bedb235a86a42209e9572ce3a346574146a076c243103b994ad5084d1cbadb not found: ID does not exist" containerID="24bedb235a86a42209e9572ce3a346574146a076c243103b994ad5084d1cbadb"
May 25 13:01:51.468998 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:51.468963    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24bedb235a86a42209e9572ce3a346574146a076c243103b994ad5084d1cbadb"} err="failed to get container status \"24bedb235a86a42209e9572ce3a346574146a076c243103b994ad5084d1cbadb\": rpc error: code = NotFound desc = could not find container \"24bedb235a86a42209e9572ce3a346574146a076c243103b994ad5084d1cbadb\": container with ID starting with 24bedb235a86a42209e9572ce3a346574146a076c243103b994ad5084d1cbadb not found: ID does not exist"
May 25 13:01:51.468998 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:51.468979    2555 scope.go:117] "RemoveContainer" containerID="40fc1e101fa0d976b03274b80970c9698eac0e89390ea336f5d45ccd627b5418"
May 25 13:01:51.469077 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:51.468936    2555 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-5pwt4"]
May 25 13:01:51.469199 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:01:51.469183    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40fc1e101fa0d976b03274b80970c9698eac0e89390ea336f5d45ccd627b5418\": container with ID starting with 40fc1e101fa0d976b03274b80970c9698eac0e89390ea336f5d45ccd627b5418 not found: ID does not exist" containerID="40fc1e101fa0d976b03274b80970c9698eac0e89390ea336f5d45ccd627b5418"
May 25 13:01:51.469239 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:51.469203    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40fc1e101fa0d976b03274b80970c9698eac0e89390ea336f5d45ccd627b5418"} err="failed to get container status \"40fc1e101fa0d976b03274b80970c9698eac0e89390ea336f5d45ccd627b5418\": rpc error: code = NotFound desc = could not find container \"40fc1e101fa0d976b03274b80970c9698eac0e89390ea336f5d45ccd627b5418\": container with ID starting with 40fc1e101fa0d976b03274b80970c9698eac0e89390ea336f5d45ccd627b5418 not found: ID does not exist"
May 25 13:01:51.599092 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:51.599062    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afa89f41-d774-4590-8679-b32702331950" path="/var/lib/kubelet/pods/afa89f41-d774-4590-8679-b32702331950/volumes"
May 25 13:01:52.449388 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:52.449347    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-s2fzk" podUID="d13f3b53-90a6-483f-bde3-976c62e4c10d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
May 25 13:01:57.453686 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:57.453632    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-s2fzk"
May 25 13:01:57.454202 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:01:57.454176    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-s2fzk" podUID="d13f3b53-90a6-483f-bde3-976c62e4c10d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
May 25 13:02:07.454863 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:02:07.454820    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-s2fzk" podUID="d13f3b53-90a6-483f-bde3-976c62e4c10d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
May 25 13:02:17.454175 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:02:17.454133    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-s2fzk" podUID="d13f3b53-90a6-483f-bde3-976c62e4c10d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
May 25 13:02:27.454800 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:02:27.454698    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-s2fzk" podUID="d13f3b53-90a6-483f-bde3-976c62e4c10d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
May 25 13:02:37.454756 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:02:37.454717    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-s2fzk" podUID="d13f3b53-90a6-483f-bde3-976c62e4c10d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
May 25 13:02:47.454873 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:02:47.454834    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-s2fzk" podUID="d13f3b53-90a6-483f-bde3-976c62e4c10d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
May 25 13:02:57.454849 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:02:57.454811    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-s2fzk"
May 25 13:03:04.905839 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:04.905801    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-s2fzk"]
May 25 13:03:04.906254 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:04.906195    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-s2fzk" podUID="d13f3b53-90a6-483f-bde3-976c62e4c10d" containerName="kube-rbac-proxy" containerID="cri-o://1d651d8aa3730e3cbf5589a8bc270424a19f6e2fbbc01f9e2efaed364b65b38f" gracePeriod=30
May 25 13:03:04.906322 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:04.906174    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-s2fzk" podUID="d13f3b53-90a6-483f-bde3-976c62e4c10d" containerName="kserve-container" containerID="cri-o://97f2c1a21d114e1c3f2c79c0db4f14c9e9ac7e0b17ac3dbb78cc425b41a3ae8a" gracePeriod=30
May 25 13:03:04.986479 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:04.986437    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6"]
May 25 13:03:04.986830 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:04.986815    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afa89f41-d774-4590-8679-b32702331950" containerName="kserve-container"
May 25 13:03:04.986890 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:04.986832    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="afa89f41-d774-4590-8679-b32702331950" containerName="kserve-container"
May 25 13:03:04.986890 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:04.986843    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afa89f41-d774-4590-8679-b32702331950" containerName="storage-initializer"
May 25 13:03:04.986890 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:04.986849    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="afa89f41-d774-4590-8679-b32702331950" containerName="storage-initializer"
May 25 13:03:04.986890 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:04.986862    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afa89f41-d774-4590-8679-b32702331950" containerName="kube-rbac-proxy"
May 25 13:03:04.986890 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:04.986867    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="afa89f41-d774-4590-8679-b32702331950" containerName="kube-rbac-proxy"
May 25 13:03:04.987064 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:04.986928    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="afa89f41-d774-4590-8679-b32702331950" containerName="kserve-container"
May 25 13:03:04.987064 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:04.986938    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="afa89f41-d774-4590-8679-b32702331950" containerName="kube-rbac-proxy"
May 25 13:03:04.990392 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:04.990363    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6"
May 25 13:03:04.992461 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:04.992436    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
May 25 13:03:04.992591 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:04.992443    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
May 25 13:03:04.999389 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:04.999361    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6"]
May 25 13:03:05.059049 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:05.059009    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9329233d-b66e-4781-b0e8-deab57371355-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6\" (UID: \"9329233d-b66e-4781-b0e8-deab57371355\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6"
May 25 13:03:05.059243 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:05.059066    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qckn9\" (UniqueName: \"kubernetes.io/projected/9329233d-b66e-4781-b0e8-deab57371355-kube-api-access-qckn9\") pod \"isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6\" (UID: \"9329233d-b66e-4781-b0e8-deab57371355\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6"
May 25 13:03:05.059243 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:05.059130    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9329233d-b66e-4781-b0e8-deab57371355-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6\" (UID: \"9329233d-b66e-4781-b0e8-deab57371355\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6"
May 25 13:03:05.059243 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:05.059172    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9329233d-b66e-4781-b0e8-deab57371355-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6\" (UID: \"9329233d-b66e-4781-b0e8-deab57371355\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6"
May 25 13:03:05.159869 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:05.159773    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9329233d-b66e-4781-b0e8-deab57371355-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6\" (UID: \"9329233d-b66e-4781-b0e8-deab57371355\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6"
May 25 13:03:05.159869 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:05.159826    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9329233d-b66e-4781-b0e8-deab57371355-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6\" (UID: \"9329233d-b66e-4781-b0e8-deab57371355\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6"
May 25 13:03:05.159869 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:05.159865    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qckn9\" (UniqueName: \"kubernetes.io/projected/9329233d-b66e-4781-b0e8-deab57371355-kube-api-access-qckn9\") pod \"isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6\" (UID: \"9329233d-b66e-4781-b0e8-deab57371355\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6"
May 25 13:03:05.160163 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:05.159901    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9329233d-b66e-4781-b0e8-deab57371355-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6\" (UID: \"9329233d-b66e-4781-b0e8-deab57371355\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6"
May 25 13:03:05.160163 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:03:05.160007    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-serving-cert: secret "isvc-sklearn-v2-mixed-predictor-serving-cert" not found
May 25 13:03:05.160163 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:03:05.160081    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9329233d-b66e-4781-b0e8-deab57371355-proxy-tls podName:9329233d-b66e-4781-b0e8-deab57371355 nodeName:}" failed. No retries permitted until 2026-05-25 13:03:05.660061178 +0000 UTC m=+2998.620111564 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9329233d-b66e-4781-b0e8-deab57371355-proxy-tls") pod "isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6" (UID: "9329233d-b66e-4781-b0e8-deab57371355") : secret "isvc-sklearn-v2-mixed-predictor-serving-cert" not found
May 25 13:03:05.160327 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:05.160240    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9329233d-b66e-4781-b0e8-deab57371355-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6\" (UID: \"9329233d-b66e-4781-b0e8-deab57371355\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6"
May 25 13:03:05.160540 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:05.160520    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9329233d-b66e-4781-b0e8-deab57371355-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6\" (UID: \"9329233d-b66e-4781-b0e8-deab57371355\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6"
May 25 13:03:05.170519 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:05.170483    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qckn9\" (UniqueName: \"kubernetes.io/projected/9329233d-b66e-4781-b0e8-deab57371355-kube-api-access-qckn9\") pod \"isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6\" (UID: \"9329233d-b66e-4781-b0e8-deab57371355\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6"
May 25 13:03:05.664686 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:05.664604    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9329233d-b66e-4781-b0e8-deab57371355-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6\" (UID: \"9329233d-b66e-4781-b0e8-deab57371355\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6"
May 25 13:03:05.667049 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:05.667028    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9329233d-b66e-4781-b0e8-deab57371355-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6\" (UID: \"9329233d-b66e-4781-b0e8-deab57371355\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6"
May 25 13:03:05.700155 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:05.700121    2555 generic.go:358] "Generic (PLEG): container finished" podID="d13f3b53-90a6-483f-bde3-976c62e4c10d" containerID="1d651d8aa3730e3cbf5589a8bc270424a19f6e2fbbc01f9e2efaed364b65b38f" exitCode=2
May 25 13:03:05.700337 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:05.700200    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-s2fzk" event={"ID":"d13f3b53-90a6-483f-bde3-976c62e4c10d","Type":"ContainerDied","Data":"1d651d8aa3730e3cbf5589a8bc270424a19f6e2fbbc01f9e2efaed364b65b38f"}
May 25 13:03:05.901933 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:05.901890    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6"
May 25 13:03:06.031682 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:06.031634    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6"]
May 25 13:03:06.033767 ip-10-0-134-255 kubenswrapper[2555]: W0525 13:03:06.033729    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9329233d_b66e_4781_b0e8_deab57371355.slice/crio-028d8e482f3d9948f8e9703f165bae42f20f07cc5876891ea492e648c458ed94 WatchSource:0}: Error finding container 028d8e482f3d9948f8e9703f165bae42f20f07cc5876891ea492e648c458ed94: Status 404 returned error can't find the container with id 028d8e482f3d9948f8e9703f165bae42f20f07cc5876891ea492e648c458ed94
May 25 13:03:06.035746 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:06.035727    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 13:03:06.705531 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:06.705495    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6" event={"ID":"9329233d-b66e-4781-b0e8-deab57371355","Type":"ContainerStarted","Data":"e4e9becb188506297e9b69b5a7c8a89651e6cfeea9e94666cfbd41a19f864916"}
May 25 13:03:06.705531 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:06.705537    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6" event={"ID":"9329233d-b66e-4781-b0e8-deab57371355","Type":"ContainerStarted","Data":"028d8e482f3d9948f8e9703f165bae42f20f07cc5876891ea492e648c458ed94"}
May 25 13:03:07.450494 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:07.450451    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-s2fzk" podUID="d13f3b53-90a6-483f-bde3-976c62e4c10d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.55:8643/healthz\": dial tcp 10.134.0.55:8643: connect: connection refused"
May 25 13:03:07.455012 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:07.454970    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-s2fzk" podUID="d13f3b53-90a6-483f-bde3-976c62e4c10d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
May 25 13:03:07.712863 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:07.712774    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-2tqtm_00e6f79c-8f05-4330-aa16-1ada2ae9e9cc/console-operator/1.log"
May 25 13:03:07.719880 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:07.719859    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-2tqtm_00e6f79c-8f05-4330-aa16-1ada2ae9e9cc/console-operator/1.log"
May 25 13:03:09.564193 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:09.564154    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-s2fzk"
May 25 13:03:09.703297 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:09.703244    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d13f3b53-90a6-483f-bde3-976c62e4c10d-kserve-provision-location\") pod \"d13f3b53-90a6-483f-bde3-976c62e4c10d\" (UID: \"d13f3b53-90a6-483f-bde3-976c62e4c10d\") "
May 25 13:03:09.703529 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:09.703316    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d13f3b53-90a6-483f-bde3-976c62e4c10d-proxy-tls\") pod \"d13f3b53-90a6-483f-bde3-976c62e4c10d\" (UID: \"d13f3b53-90a6-483f-bde3-976c62e4c10d\") "
May 25 13:03:09.703529 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:09.703337    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d13f3b53-90a6-483f-bde3-976c62e4c10d-isvc-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"d13f3b53-90a6-483f-bde3-976c62e4c10d\" (UID: \"d13f3b53-90a6-483f-bde3-976c62e4c10d\") "
May 25 13:03:09.703529 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:09.703371    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhtdg\" (UniqueName: \"kubernetes.io/projected/d13f3b53-90a6-483f-bde3-976c62e4c10d-kube-api-access-dhtdg\") pod \"d13f3b53-90a6-483f-bde3-976c62e4c10d\" (UID: \"d13f3b53-90a6-483f-bde3-976c62e4c10d\") "
May 25 13:03:09.703750 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:09.703627    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d13f3b53-90a6-483f-bde3-976c62e4c10d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d13f3b53-90a6-483f-bde3-976c62e4c10d" (UID: "d13f3b53-90a6-483f-bde3-976c62e4c10d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:03:09.703750 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:09.703720    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d13f3b53-90a6-483f-bde3-976c62e4c10d-isvc-sklearn-v2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-v2-kube-rbac-proxy-sar-config") pod "d13f3b53-90a6-483f-bde3-976c62e4c10d" (UID: "d13f3b53-90a6-483f-bde3-976c62e4c10d"). InnerVolumeSpecName "isvc-sklearn-v2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:03:09.705536 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:09.705512    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d13f3b53-90a6-483f-bde3-976c62e4c10d-kube-api-access-dhtdg" (OuterVolumeSpecName: "kube-api-access-dhtdg") pod "d13f3b53-90a6-483f-bde3-976c62e4c10d" (UID: "d13f3b53-90a6-483f-bde3-976c62e4c10d"). InnerVolumeSpecName "kube-api-access-dhtdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:03:09.705620 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:09.705574    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d13f3b53-90a6-483f-bde3-976c62e4c10d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d13f3b53-90a6-483f-bde3-976c62e4c10d" (UID: "d13f3b53-90a6-483f-bde3-976c62e4c10d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:03:09.716222 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:09.716196    2555 generic.go:358] "Generic (PLEG): container finished" podID="d13f3b53-90a6-483f-bde3-976c62e4c10d" containerID="97f2c1a21d114e1c3f2c79c0db4f14c9e9ac7e0b17ac3dbb78cc425b41a3ae8a" exitCode=0
May 25 13:03:09.716325 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:09.716288    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-s2fzk"
May 25 13:03:09.716378 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:09.716281    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-s2fzk" event={"ID":"d13f3b53-90a6-483f-bde3-976c62e4c10d","Type":"ContainerDied","Data":"97f2c1a21d114e1c3f2c79c0db4f14c9e9ac7e0b17ac3dbb78cc425b41a3ae8a"}
May 25 13:03:09.716420 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:09.716397    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-s2fzk" event={"ID":"d13f3b53-90a6-483f-bde3-976c62e4c10d","Type":"ContainerDied","Data":"1070533fe209f6f412bd5c6b08b5a3831dfb1f752438b696a12e86bc4be611ec"}
May 25 13:03:09.716454 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:09.716418    2555 scope.go:117] "RemoveContainer" containerID="1d651d8aa3730e3cbf5589a8bc270424a19f6e2fbbc01f9e2efaed364b65b38f"
May 25 13:03:09.724798 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:09.724772    2555 scope.go:117] "RemoveContainer" containerID="97f2c1a21d114e1c3f2c79c0db4f14c9e9ac7e0b17ac3dbb78cc425b41a3ae8a"
May 25 13:03:09.732844 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:09.732822    2555 scope.go:117] "RemoveContainer" containerID="d100f7678f5389b2a595f3d3be07d9f4857ad5e5520bc4f408276e9ea0ed0d4b"
May 25 13:03:09.738207 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:09.738183    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-s2fzk"]
May 25 13:03:09.741830 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:09.741811    2555 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-s2fzk"]
May 25 13:03:09.741902 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:09.741885    2555 scope.go:117] "RemoveContainer" containerID="1d651d8aa3730e3cbf5589a8bc270424a19f6e2fbbc01f9e2efaed364b65b38f"
May 25 13:03:09.742167 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:03:09.742148    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d651d8aa3730e3cbf5589a8bc270424a19f6e2fbbc01f9e2efaed364b65b38f\": container with ID starting with 1d651d8aa3730e3cbf5589a8bc270424a19f6e2fbbc01f9e2efaed364b65b38f not found: ID does not exist" containerID="1d651d8aa3730e3cbf5589a8bc270424a19f6e2fbbc01f9e2efaed364b65b38f"
May 25 13:03:09.742233 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:09.742180    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d651d8aa3730e3cbf5589a8bc270424a19f6e2fbbc01f9e2efaed364b65b38f"} err="failed to get container status \"1d651d8aa3730e3cbf5589a8bc270424a19f6e2fbbc01f9e2efaed364b65b38f\": rpc error: code = NotFound desc = could not find container \"1d651d8aa3730e3cbf5589a8bc270424a19f6e2fbbc01f9e2efaed364b65b38f\": container with ID starting with 1d651d8aa3730e3cbf5589a8bc270424a19f6e2fbbc01f9e2efaed364b65b38f not found: ID does not exist"
May 25 13:03:09.742233 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:09.742209    2555 scope.go:117] "RemoveContainer" containerID="97f2c1a21d114e1c3f2c79c0db4f14c9e9ac7e0b17ac3dbb78cc425b41a3ae8a"
May 25 13:03:09.742481 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:03:09.742460    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97f2c1a21d114e1c3f2c79c0db4f14c9e9ac7e0b17ac3dbb78cc425b41a3ae8a\": container with ID starting with 97f2c1a21d114e1c3f2c79c0db4f14c9e9ac7e0b17ac3dbb78cc425b41a3ae8a not found: ID does not exist" containerID="97f2c1a21d114e1c3f2c79c0db4f14c9e9ac7e0b17ac3dbb78cc425b41a3ae8a"
May 25 13:03:09.742521 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:09.742487    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97f2c1a21d114e1c3f2c79c0db4f14c9e9ac7e0b17ac3dbb78cc425b41a3ae8a"} err="failed to get container status \"97f2c1a21d114e1c3f2c79c0db4f14c9e9ac7e0b17ac3dbb78cc425b41a3ae8a\": rpc error: code = NotFound desc = could not find container \"97f2c1a21d114e1c3f2c79c0db4f14c9e9ac7e0b17ac3dbb78cc425b41a3ae8a\": container with ID starting with 97f2c1a21d114e1c3f2c79c0db4f14c9e9ac7e0b17ac3dbb78cc425b41a3ae8a not found: ID does not exist"
May 25 13:03:09.742521 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:09.742503    2555 scope.go:117] "RemoveContainer" containerID="d100f7678f5389b2a595f3d3be07d9f4857ad5e5520bc4f408276e9ea0ed0d4b"
May 25 13:03:09.742750 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:03:09.742732    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d100f7678f5389b2a595f3d3be07d9f4857ad5e5520bc4f408276e9ea0ed0d4b\": container with ID starting with d100f7678f5389b2a595f3d3be07d9f4857ad5e5520bc4f408276e9ea0ed0d4b not found: ID does not exist" containerID="d100f7678f5389b2a595f3d3be07d9f4857ad5e5520bc4f408276e9ea0ed0d4b"
May 25 13:03:09.742816 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:09.742759    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d100f7678f5389b2a595f3d3be07d9f4857ad5e5520bc4f408276e9ea0ed0d4b"} err="failed to get container status \"d100f7678f5389b2a595f3d3be07d9f4857ad5e5520bc4f408276e9ea0ed0d4b\": rpc error: code = NotFound desc = could not find container \"d100f7678f5389b2a595f3d3be07d9f4857ad5e5520bc4f408276e9ea0ed0d4b\": container with ID starting with d100f7678f5389b2a595f3d3be07d9f4857ad5e5520bc4f408276e9ea0ed0d4b not found: ID does not exist"
May 25 13:03:09.804559 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:09.804521    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d13f3b53-90a6-483f-bde3-976c62e4c10d-kserve-provision-location\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:03:09.804559 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:09.804553    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d13f3b53-90a6-483f-bde3-976c62e4c10d-proxy-tls\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:03:09.804559 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:09.804567    2555 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d13f3b53-90a6-483f-bde3-976c62e4c10d-isvc-sklearn-v2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:03:09.804826 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:09.804581    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhtdg\" (UniqueName: \"kubernetes.io/projected/d13f3b53-90a6-483f-bde3-976c62e4c10d-kube-api-access-dhtdg\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:03:10.720914 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:10.720880    2555 generic.go:358] "Generic (PLEG): container finished" podID="9329233d-b66e-4781-b0e8-deab57371355" containerID="e4e9becb188506297e9b69b5a7c8a89651e6cfeea9e94666cfbd41a19f864916" exitCode=0
May 25 13:03:10.721355 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:10.720962    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6" event={"ID":"9329233d-b66e-4781-b0e8-deab57371355","Type":"ContainerDied","Data":"e4e9becb188506297e9b69b5a7c8a89651e6cfeea9e94666cfbd41a19f864916"}
May 25 13:03:11.599093 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:11.599061    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d13f3b53-90a6-483f-bde3-976c62e4c10d" path="/var/lib/kubelet/pods/d13f3b53-90a6-483f-bde3-976c62e4c10d/volumes"
May 25 13:03:11.726735 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:11.726700    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6" event={"ID":"9329233d-b66e-4781-b0e8-deab57371355","Type":"ContainerStarted","Data":"bb3941059e9f9513eb10710ec2952a6c99fe064cc9ad92f610555095f04421fd"}
May 25 13:03:11.726735 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:11.726742    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6" event={"ID":"9329233d-b66e-4781-b0e8-deab57371355","Type":"ContainerStarted","Data":"f3ef59187d7f1e9f5922c0a30d865779de76985fc0614fbd1cfff6e533a74f53"}
May 25 13:03:11.727157 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:11.726946    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6"
May 25 13:03:11.743955 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:11.743908    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6" podStartSLOduration=7.743895727 podStartE2EDuration="7.743895727s" podCreationTimestamp="2026-05-25 13:03:04 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:03:11.742159875 +0000 UTC m=+3004.702210279" watchObservedRunningTime="2026-05-25 13:03:11.743895727 +0000 UTC m=+3004.703946201"
May 25 13:03:12.730341 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:12.730305    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6"
May 25 13:03:12.731515 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:12.731486    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6" podUID="9329233d-b66e-4781-b0e8-deab57371355" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
May 25 13:03:13.733668 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:13.733612    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6" podUID="9329233d-b66e-4781-b0e8-deab57371355" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
May 25 13:03:18.738610 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:18.738574    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6"
May 25 13:03:18.739276 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:18.739244    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6" podUID="9329233d-b66e-4781-b0e8-deab57371355" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
May 25 13:03:28.739309 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:28.739265    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6" podUID="9329233d-b66e-4781-b0e8-deab57371355" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
May 25 13:03:38.739390 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:38.739352    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6" podUID="9329233d-b66e-4781-b0e8-deab57371355" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
May 25 13:03:48.739757 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:48.739716    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6" podUID="9329233d-b66e-4781-b0e8-deab57371355" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
May 25 13:03:58.739981 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:03:58.739896    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6" podUID="9329233d-b66e-4781-b0e8-deab57371355" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
May 25 13:04:08.739759 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:04:08.739719    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6" podUID="9329233d-b66e-4781-b0e8-deab57371355" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
May 25 13:04:18.740373 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:04:18.740342    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6"
May 25 13:04:25.065577 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:04:25.065539    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6"]
May 25 13:04:25.066134 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:04:25.065889    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6" podUID="9329233d-b66e-4781-b0e8-deab57371355" containerName="kserve-container" containerID="cri-o://f3ef59187d7f1e9f5922c0a30d865779de76985fc0614fbd1cfff6e533a74f53" gracePeriod=30
May 25 13:04:25.066134 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:04:25.066081    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6" podUID="9329233d-b66e-4781-b0e8-deab57371355" containerName="kube-rbac-proxy" containerID="cri-o://bb3941059e9f9513eb10710ec2952a6c99fe064cc9ad92f610555095f04421fd" gracePeriod=30
May 25 13:04:25.981349 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:04:25.981310    2555 generic.go:358] "Generic (PLEG): container finished" podID="9329233d-b66e-4781-b0e8-deab57371355" containerID="bb3941059e9f9513eb10710ec2952a6c99fe064cc9ad92f610555095f04421fd" exitCode=2
May 25 13:04:25.981527 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:04:25.981380    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6" event={"ID":"9329233d-b66e-4781-b0e8-deab57371355","Type":"ContainerDied","Data":"bb3941059e9f9513eb10710ec2952a6c99fe064cc9ad92f610555095f04421fd"}
May 25 13:04:28.734619 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:04:28.734585    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6" podUID="9329233d-b66e-4781-b0e8-deab57371355" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.56:8643/healthz\": dial tcp 10.134.0.56:8643: connect: connection refused"
May 25 13:04:28.739965 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:04:28.739942    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6" podUID="9329233d-b66e-4781-b0e8-deab57371355" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
May 25 13:04:28.908358 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:04:28.908333    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6"
May 25 13:04:28.992826 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:04:28.992799    2555 generic.go:358] "Generic (PLEG): container finished" podID="9329233d-b66e-4781-b0e8-deab57371355" containerID="f3ef59187d7f1e9f5922c0a30d865779de76985fc0614fbd1cfff6e533a74f53" exitCode=0
May 25 13:04:28.992967 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:04:28.992844    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6" event={"ID":"9329233d-b66e-4781-b0e8-deab57371355","Type":"ContainerDied","Data":"f3ef59187d7f1e9f5922c0a30d865779de76985fc0614fbd1cfff6e533a74f53"}
May 25 13:04:28.992967 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:04:28.992866    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6" event={"ID":"9329233d-b66e-4781-b0e8-deab57371355","Type":"ContainerDied","Data":"028d8e482f3d9948f8e9703f165bae42f20f07cc5876891ea492e648c458ed94"}
May 25 13:04:28.992967 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:04:28.992874    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6"
May 25 13:04:28.993077 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:04:28.992879    2555 scope.go:117] "RemoveContainer" containerID="bb3941059e9f9513eb10710ec2952a6c99fe064cc9ad92f610555095f04421fd"
May 25 13:04:29.001040 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:04:29.001016    2555 scope.go:117] "RemoveContainer" containerID="f3ef59187d7f1e9f5922c0a30d865779de76985fc0614fbd1cfff6e533a74f53"
May 25 13:04:29.008401 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:04:29.008384    2555 scope.go:117] "RemoveContainer" containerID="e4e9becb188506297e9b69b5a7c8a89651e6cfeea9e94666cfbd41a19f864916"
May 25 13:04:29.014879 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:04:29.014863    2555 scope.go:117] "RemoveContainer" containerID="bb3941059e9f9513eb10710ec2952a6c99fe064cc9ad92f610555095f04421fd"
May 25 13:04:29.015124 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:04:29.015105    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb3941059e9f9513eb10710ec2952a6c99fe064cc9ad92f610555095f04421fd\": container with ID starting with bb3941059e9f9513eb10710ec2952a6c99fe064cc9ad92f610555095f04421fd not found: ID does not exist" containerID="bb3941059e9f9513eb10710ec2952a6c99fe064cc9ad92f610555095f04421fd"
May 25 13:04:29.015169 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:04:29.015133    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb3941059e9f9513eb10710ec2952a6c99fe064cc9ad92f610555095f04421fd"} err="failed to get container status \"bb3941059e9f9513eb10710ec2952a6c99fe064cc9ad92f610555095f04421fd\": rpc error: code = NotFound desc = could not find container \"bb3941059e9f9513eb10710ec2952a6c99fe064cc9ad92f610555095f04421fd\": container with ID starting with bb3941059e9f9513eb10710ec2952a6c99fe064cc9ad92f610555095f04421fd not found: ID does not exist"
May 25 13:04:29.015169 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:04:29.015150    2555 scope.go:117] "RemoveContainer" containerID="f3ef59187d7f1e9f5922c0a30d865779de76985fc0614fbd1cfff6e533a74f53"
May 25 13:04:29.015391 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:04:29.015371    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3ef59187d7f1e9f5922c0a30d865779de76985fc0614fbd1cfff6e533a74f53\": container with ID starting with f3ef59187d7f1e9f5922c0a30d865779de76985fc0614fbd1cfff6e533a74f53 not found: ID does not exist" containerID="f3ef59187d7f1e9f5922c0a30d865779de76985fc0614fbd1cfff6e533a74f53"
May 25 13:04:29.015428 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:04:29.015404    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3ef59187d7f1e9f5922c0a30d865779de76985fc0614fbd1cfff6e533a74f53"} err="failed to get container status \"f3ef59187d7f1e9f5922c0a30d865779de76985fc0614fbd1cfff6e533a74f53\": rpc error: code = NotFound desc = could not find container \"f3ef59187d7f1e9f5922c0a30d865779de76985fc0614fbd1cfff6e533a74f53\": container with ID starting with f3ef59187d7f1e9f5922c0a30d865779de76985fc0614fbd1cfff6e533a74f53 not found: ID does not exist"
May 25 13:04:29.015428 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:04:29.015421    2555 scope.go:117] "RemoveContainer" containerID="e4e9becb188506297e9b69b5a7c8a89651e6cfeea9e94666cfbd41a19f864916"
May 25 13:04:29.015654 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:04:29.015617    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4e9becb188506297e9b69b5a7c8a89651e6cfeea9e94666cfbd41a19f864916\": container with ID starting with e4e9becb188506297e9b69b5a7c8a89651e6cfeea9e94666cfbd41a19f864916 not found: ID does not exist" containerID="e4e9becb188506297e9b69b5a7c8a89651e6cfeea9e94666cfbd41a19f864916"
May 25 13:04:29.015748 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:04:29.015663    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4e9becb188506297e9b69b5a7c8a89651e6cfeea9e94666cfbd41a19f864916"} err="failed to get container status \"e4e9becb188506297e9b69b5a7c8a89651e6cfeea9e94666cfbd41a19f864916\": rpc error: code = NotFound desc = could not find container \"e4e9becb188506297e9b69b5a7c8a89651e6cfeea9e94666cfbd41a19f864916\": container with ID starting with e4e9becb188506297e9b69b5a7c8a89651e6cfeea9e94666cfbd41a19f864916 not found: ID does not exist"
May 25 13:04:29.074136 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:04:29.074114    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qckn9\" (UniqueName: \"kubernetes.io/projected/9329233d-b66e-4781-b0e8-deab57371355-kube-api-access-qckn9\") pod \"9329233d-b66e-4781-b0e8-deab57371355\" (UID: \"9329233d-b66e-4781-b0e8-deab57371355\") "
May 25 13:04:29.074223 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:04:29.074165    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9329233d-b66e-4781-b0e8-deab57371355-kserve-provision-location\") pod \"9329233d-b66e-4781-b0e8-deab57371355\" (UID: \"9329233d-b66e-4781-b0e8-deab57371355\") "
May 25 13:04:29.074223 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:04:29.074196    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9329233d-b66e-4781-b0e8-deab57371355-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\") pod \"9329233d-b66e-4781-b0e8-deab57371355\" (UID: \"9329233d-b66e-4781-b0e8-deab57371355\") "
May 25 13:04:29.074329 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:04:29.074233    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9329233d-b66e-4781-b0e8-deab57371355-proxy-tls\") pod \"9329233d-b66e-4781-b0e8-deab57371355\" (UID: \"9329233d-b66e-4781-b0e8-deab57371355\") "
May 25 13:04:29.074512 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:04:29.074487    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9329233d-b66e-4781-b0e8-deab57371355-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9329233d-b66e-4781-b0e8-deab57371355" (UID: "9329233d-b66e-4781-b0e8-deab57371355"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:04:29.074609 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:04:29.074583    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9329233d-b66e-4781-b0e8-deab57371355-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config") pod "9329233d-b66e-4781-b0e8-deab57371355" (UID: "9329233d-b66e-4781-b0e8-deab57371355"). InnerVolumeSpecName "isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:04:29.076052 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:04:29.076029    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9329233d-b66e-4781-b0e8-deab57371355-kube-api-access-qckn9" (OuterVolumeSpecName: "kube-api-access-qckn9") pod "9329233d-b66e-4781-b0e8-deab57371355" (UID: "9329233d-b66e-4781-b0e8-deab57371355"). InnerVolumeSpecName "kube-api-access-qckn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:04:29.076178 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:04:29.076163    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9329233d-b66e-4781-b0e8-deab57371355-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9329233d-b66e-4781-b0e8-deab57371355" (UID: "9329233d-b66e-4781-b0e8-deab57371355"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:04:29.175782 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:04:29.175761    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9329233d-b66e-4781-b0e8-deab57371355-kserve-provision-location\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:04:29.175895 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:04:29.175789    2555 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9329233d-b66e-4781-b0e8-deab57371355-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:04:29.175895 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:04:29.175805    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9329233d-b66e-4781-b0e8-deab57371355-proxy-tls\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:04:29.175895 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:04:29.175819    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qckn9\" (UniqueName: \"kubernetes.io/projected/9329233d-b66e-4781-b0e8-deab57371355-kube-api-access-qckn9\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:04:29.313493 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:04:29.313465    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6"]
May 25 13:04:29.316876 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:04:29.316854    2555 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-ntbc6"]
May 25 13:04:29.598691 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:04:29.598610    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9329233d-b66e-4781-b0e8-deab57371355" path="/var/lib/kubelet/pods/9329233d-b66e-4781-b0e8-deab57371355/volumes"
May 25 13:05:06.233185 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:06.233149    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m"]
May 25 13:05:06.233704 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:06.233491    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9329233d-b66e-4781-b0e8-deab57371355" containerName="kube-rbac-proxy"
May 25 13:05:06.233704 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:06.233502    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="9329233d-b66e-4781-b0e8-deab57371355" containerName="kube-rbac-proxy"
May 25 13:05:06.233704 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:06.233513    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9329233d-b66e-4781-b0e8-deab57371355" containerName="storage-initializer"
May 25 13:05:06.233704 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:06.233518    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="9329233d-b66e-4781-b0e8-deab57371355" containerName="storage-initializer"
May 25 13:05:06.233704 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:06.233525    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d13f3b53-90a6-483f-bde3-976c62e4c10d" containerName="kserve-container"
May 25 13:05:06.233704 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:06.233531    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d13f3b53-90a6-483f-bde3-976c62e4c10d" containerName="kserve-container"
May 25 13:05:06.233704 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:06.233543    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d13f3b53-90a6-483f-bde3-976c62e4c10d" containerName="kube-rbac-proxy"
May 25 13:05:06.233704 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:06.233549    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d13f3b53-90a6-483f-bde3-976c62e4c10d" containerName="kube-rbac-proxy"
May 25 13:05:06.233704 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:06.233566    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d13f3b53-90a6-483f-bde3-976c62e4c10d" containerName="storage-initializer"
May 25 13:05:06.233704 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:06.233572    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d13f3b53-90a6-483f-bde3-976c62e4c10d" containerName="storage-initializer"
May 25 13:05:06.233704 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:06.233578    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9329233d-b66e-4781-b0e8-deab57371355" containerName="kserve-container"
May 25 13:05:06.233704 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:06.233584    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="9329233d-b66e-4781-b0e8-deab57371355" containerName="kserve-container"
May 25 13:05:06.233704 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:06.233631    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="d13f3b53-90a6-483f-bde3-976c62e4c10d" containerName="kube-rbac-proxy"
May 25 13:05:06.233704 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:06.233656    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="9329233d-b66e-4781-b0e8-deab57371355" containerName="kserve-container"
May 25 13:05:06.233704 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:06.233664    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="9329233d-b66e-4781-b0e8-deab57371355" containerName="kube-rbac-proxy"
May 25 13:05:06.233704 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:06.233670    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="d13f3b53-90a6-483f-bde3-976c62e4c10d" containerName="kserve-container"
May 25 13:05:06.236932 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:06.236899    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m"
May 25 13:05:06.238934 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:06.238906    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6qbkc\""
May 25 13:05:06.238934 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:06.238917    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 25 13:05:06.239112 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:06.238957    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-predictor-serving-cert\""
May 25 13:05:06.239112 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:06.238992    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\""
May 25 13:05:06.239112 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:06.238917    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 25 13:05:06.246313 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:06.246284    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m"]
May 25 13:05:06.263039 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:06.263008    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt9ks\" (UniqueName: \"kubernetes.io/projected/56c1b207-9a74-4f0f-a503-ce36347b8515-kube-api-access-wt9ks\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m\" (UID: \"56c1b207-9a74-4f0f-a503-ce36347b8515\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m"
May 25 13:05:06.263039 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:06.263040    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56c1b207-9a74-4f0f-a503-ce36347b8515-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m\" (UID: \"56c1b207-9a74-4f0f-a503-ce36347b8515\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m"
May 25 13:05:06.263243 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:06.263071    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56c1b207-9a74-4f0f-a503-ce36347b8515-isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m\" (UID: \"56c1b207-9a74-4f0f-a503-ce36347b8515\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m"
May 25 13:05:06.263243 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:06.263167    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56c1b207-9a74-4f0f-a503-ce36347b8515-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m\" (UID: \"56c1b207-9a74-4f0f-a503-ce36347b8515\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m"
May 25 13:05:06.364050 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:06.364013    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56c1b207-9a74-4f0f-a503-ce36347b8515-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m\" (UID: \"56c1b207-9a74-4f0f-a503-ce36347b8515\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m"
May 25 13:05:06.364248 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:06.364126    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wt9ks\" (UniqueName: \"kubernetes.io/projected/56c1b207-9a74-4f0f-a503-ce36347b8515-kube-api-access-wt9ks\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m\" (UID: \"56c1b207-9a74-4f0f-a503-ce36347b8515\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m"
May 25 13:05:06.364248 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:06.364155    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56c1b207-9a74-4f0f-a503-ce36347b8515-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m\" (UID: \"56c1b207-9a74-4f0f-a503-ce36347b8515\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m"
May 25 13:05:06.364248 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:05:06.364172    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-serving-cert: secret "isvc-tensorflow-runtime-predictor-serving-cert" not found
May 25 13:05:06.364248 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:06.364195    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56c1b207-9a74-4f0f-a503-ce36347b8515-isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m\" (UID: \"56c1b207-9a74-4f0f-a503-ce36347b8515\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m"
May 25 13:05:06.364248 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:05:06.364235    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56c1b207-9a74-4f0f-a503-ce36347b8515-proxy-tls podName:56c1b207-9a74-4f0f-a503-ce36347b8515 nodeName:}" failed. No retries permitted until 2026-05-25 13:05:06.864219338 +0000 UTC m=+3119.824269727 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/56c1b207-9a74-4f0f-a503-ce36347b8515-proxy-tls") pod "isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m" (UID: "56c1b207-9a74-4f0f-a503-ce36347b8515") : secret "isvc-tensorflow-runtime-predictor-serving-cert" not found
May 25 13:05:06.364563 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:06.364544    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56c1b207-9a74-4f0f-a503-ce36347b8515-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m\" (UID: \"56c1b207-9a74-4f0f-a503-ce36347b8515\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m"
May 25 13:05:06.364907 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:06.364889    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56c1b207-9a74-4f0f-a503-ce36347b8515-isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m\" (UID: \"56c1b207-9a74-4f0f-a503-ce36347b8515\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m"
May 25 13:05:06.373183 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:06.373162    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt9ks\" (UniqueName: \"kubernetes.io/projected/56c1b207-9a74-4f0f-a503-ce36347b8515-kube-api-access-wt9ks\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m\" (UID: \"56c1b207-9a74-4f0f-a503-ce36347b8515\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m"
May 25 13:05:06.868812 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:06.868771    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56c1b207-9a74-4f0f-a503-ce36347b8515-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m\" (UID: \"56c1b207-9a74-4f0f-a503-ce36347b8515\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m"
May 25 13:05:06.871165 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:06.871144    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56c1b207-9a74-4f0f-a503-ce36347b8515-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m\" (UID: \"56c1b207-9a74-4f0f-a503-ce36347b8515\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m"
May 25 13:05:07.150902 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:07.150815    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m"
May 25 13:05:07.279709 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:07.279682    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m"]
May 25 13:05:07.282213 ip-10-0-134-255 kubenswrapper[2555]: W0525 13:05:07.282181    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56c1b207_9a74_4f0f_a503_ce36347b8515.slice/crio-2c07bcee8765d5197a56f7b2461724245bcc1fb3d8908780552178008fa8606c WatchSource:0}: Error finding container 2c07bcee8765d5197a56f7b2461724245bcc1fb3d8908780552178008fa8606c: Status 404 returned error can't find the container with id 2c07bcee8765d5197a56f7b2461724245bcc1fb3d8908780552178008fa8606c
May 25 13:05:08.127909 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:08.127818    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m" event={"ID":"56c1b207-9a74-4f0f-a503-ce36347b8515","Type":"ContainerStarted","Data":"18ac68b5afff40c3c0f1573ddac11df45f2d5f4653eaaf8b6a0f3887574e9b1e"}
May 25 13:05:08.128138 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:08.128111    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m" event={"ID":"56c1b207-9a74-4f0f-a503-ce36347b8515","Type":"ContainerStarted","Data":"2c07bcee8765d5197a56f7b2461724245bcc1fb3d8908780552178008fa8606c"}
May 25 13:05:13.148864 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:13.148828    2555 generic.go:358] "Generic (PLEG): container finished" podID="56c1b207-9a74-4f0f-a503-ce36347b8515" containerID="18ac68b5afff40c3c0f1573ddac11df45f2d5f4653eaaf8b6a0f3887574e9b1e" exitCode=0
May 25 13:05:13.149348 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:13.148903    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m" event={"ID":"56c1b207-9a74-4f0f-a503-ce36347b8515","Type":"ContainerDied","Data":"18ac68b5afff40c3c0f1573ddac11df45f2d5f4653eaaf8b6a0f3887574e9b1e"}
May 25 13:05:17.168831 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:17.168746    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m" event={"ID":"56c1b207-9a74-4f0f-a503-ce36347b8515","Type":"ContainerStarted","Data":"b843570466d51916e78a9da3ba50c643082cc2659b8c1ecafd6b14da34646c90"}
May 25 13:05:17.168831 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:17.168788    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m" event={"ID":"56c1b207-9a74-4f0f-a503-ce36347b8515","Type":"ContainerStarted","Data":"a2f7cda28625ca7410b96d4480ad6cedc03292ed74aa69923d54bd5c71712a14"}
May 25 13:05:17.169221 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:17.168982    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m"
May 25 13:05:17.184331 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:17.184280    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m" podStartSLOduration=7.614461678 podStartE2EDuration="11.18426575s" podCreationTimestamp="2026-05-25 13:05:06 +0000 UTC" firstStartedPulling="2026-05-25 13:05:13.150122895 +0000 UTC m=+3126.110173279" lastFinishedPulling="2026-05-25 13:05:16.719926962 +0000 UTC m=+3129.679977351" observedRunningTime="2026-05-25 13:05:17.183745034 +0000 UTC m=+3130.143795439" watchObservedRunningTime="2026-05-25 13:05:17.18426575 +0000 UTC m=+3130.144316154"
May 25 13:05:18.172481 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:18.172450    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m"
May 25 13:05:18.173769 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:18.173742    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m" podUID="56c1b207-9a74-4f0f-a503-ce36347b8515" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
May 25 13:05:19.176107 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:19.176062    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m" podUID="56c1b207-9a74-4f0f-a503-ce36347b8515" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
May 25 13:05:24.181341 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:24.181301    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m"
May 25 13:05:24.181915 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:24.181886    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m" podUID="56c1b207-9a74-4f0f-a503-ce36347b8515" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
May 25 13:05:34.182611 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:34.182528    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m"
May 25 13:05:47.329415 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:47.329379    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m"]
May 25 13:05:47.329883 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:47.329761    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m" podUID="56c1b207-9a74-4f0f-a503-ce36347b8515" containerName="kserve-container" containerID="cri-o://a2f7cda28625ca7410b96d4480ad6cedc03292ed74aa69923d54bd5c71712a14" gracePeriod=30
May 25 13:05:47.329883 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:47.329788    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m" podUID="56c1b207-9a74-4f0f-a503-ce36347b8515" containerName="kube-rbac-proxy" containerID="cri-o://b843570466d51916e78a9da3ba50c643082cc2659b8c1ecafd6b14da34646c90" gracePeriod=30
May 25 13:05:48.276039 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:48.276000    2555 generic.go:358] "Generic (PLEG): container finished" podID="56c1b207-9a74-4f0f-a503-ce36347b8515" containerID="b843570466d51916e78a9da3ba50c643082cc2659b8c1ecafd6b14da34646c90" exitCode=2
May 25 13:05:48.276214 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:48.276049    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m" event={"ID":"56c1b207-9a74-4f0f-a503-ce36347b8515","Type":"ContainerDied","Data":"b843570466d51916e78a9da3ba50c643082cc2659b8c1ecafd6b14da34646c90"}
May 25 13:05:49.177013 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:49.176969    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m" podUID="56c1b207-9a74-4f0f-a503-ce36347b8515" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.57:8643/healthz\": dial tcp 10.134.0.57:8643: connect: connection refused"
May 25 13:05:54.176431 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:54.176384    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m" podUID="56c1b207-9a74-4f0f-a503-ce36347b8515" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.57:8643/healthz\": dial tcp 10.134.0.57:8643: connect: connection refused"
May 25 13:05:59.176468 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:59.176422    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m" podUID="56c1b207-9a74-4f0f-a503-ce36347b8515" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.57:8643/healthz\": dial tcp 10.134.0.57:8643: connect: connection refused"
May 25 13:05:59.177001 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:05:59.176589    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m"
May 25 13:06:04.176415 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:06:04.176369    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m" podUID="56c1b207-9a74-4f0f-a503-ce36347b8515" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.57:8643/healthz\": dial tcp 10.134.0.57:8643: connect: connection refused"
May 25 13:06:09.176532 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:06:09.176491    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m" podUID="56c1b207-9a74-4f0f-a503-ce36347b8515" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.57:8643/healthz\": dial tcp 10.134.0.57:8643: connect: connection refused"
May 25 13:06:14.176536 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:06:14.176494    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m" podUID="56c1b207-9a74-4f0f-a503-ce36347b8515" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.57:8643/healthz\": dial tcp 10.134.0.57:8643: connect: connection refused"
May 25 13:06:17.979262 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:06:17.979237    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m"
May 25 13:06:18.085805 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:06:18.085763    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56c1b207-9a74-4f0f-a503-ce36347b8515-kserve-provision-location\") pod \"56c1b207-9a74-4f0f-a503-ce36347b8515\" (UID: \"56c1b207-9a74-4f0f-a503-ce36347b8515\") "
May 25 13:06:18.085965 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:06:18.085821    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wt9ks\" (UniqueName: \"kubernetes.io/projected/56c1b207-9a74-4f0f-a503-ce36347b8515-kube-api-access-wt9ks\") pod \"56c1b207-9a74-4f0f-a503-ce36347b8515\" (UID: \"56c1b207-9a74-4f0f-a503-ce36347b8515\") "
May 25 13:06:18.085965 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:06:18.085892    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56c1b207-9a74-4f0f-a503-ce36347b8515-proxy-tls\") pod \"56c1b207-9a74-4f0f-a503-ce36347b8515\" (UID: \"56c1b207-9a74-4f0f-a503-ce36347b8515\") "
May 25 13:06:18.085965 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:06:18.085932    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56c1b207-9a74-4f0f-a503-ce36347b8515-isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\") pod \"56c1b207-9a74-4f0f-a503-ce36347b8515\" (UID: \"56c1b207-9a74-4f0f-a503-ce36347b8515\") "
May 25 13:06:18.086336 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:06:18.086307    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56c1b207-9a74-4f0f-a503-ce36347b8515-isvc-tensorflow-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-tensorflow-runtime-kube-rbac-proxy-sar-config") pod "56c1b207-9a74-4f0f-a503-ce36347b8515" (UID: "56c1b207-9a74-4f0f-a503-ce36347b8515"). InnerVolumeSpecName "isvc-tensorflow-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:06:18.088157 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:06:18.088126    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56c1b207-9a74-4f0f-a503-ce36347b8515-kube-api-access-wt9ks" (OuterVolumeSpecName: "kube-api-access-wt9ks") pod "56c1b207-9a74-4f0f-a503-ce36347b8515" (UID: "56c1b207-9a74-4f0f-a503-ce36347b8515"). InnerVolumeSpecName "kube-api-access-wt9ks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:06:18.088300 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:06:18.088202    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56c1b207-9a74-4f0f-a503-ce36347b8515-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "56c1b207-9a74-4f0f-a503-ce36347b8515" (UID: "56c1b207-9a74-4f0f-a503-ce36347b8515"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:06:18.097375 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:06:18.097316    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56c1b207-9a74-4f0f-a503-ce36347b8515-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "56c1b207-9a74-4f0f-a503-ce36347b8515" (UID: "56c1b207-9a74-4f0f-a503-ce36347b8515"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:06:18.186552 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:06:18.186517    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56c1b207-9a74-4f0f-a503-ce36347b8515-proxy-tls\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:06:18.186552 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:06:18.186549    2555 reconciler_common.go:299] "Volume detached for volume \"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56c1b207-9a74-4f0f-a503-ce36347b8515-isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:06:18.186552 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:06:18.186562    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56c1b207-9a74-4f0f-a503-ce36347b8515-kserve-provision-location\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:06:18.186846 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:06:18.186571    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wt9ks\" (UniqueName: \"kubernetes.io/projected/56c1b207-9a74-4f0f-a503-ce36347b8515-kube-api-access-wt9ks\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:06:18.376290 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:06:18.376181    2555 generic.go:358] "Generic (PLEG): container finished" podID="56c1b207-9a74-4f0f-a503-ce36347b8515" containerID="a2f7cda28625ca7410b96d4480ad6cedc03292ed74aa69923d54bd5c71712a14" exitCode=137
May 25 13:06:18.376290 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:06:18.376249    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m" event={"ID":"56c1b207-9a74-4f0f-a503-ce36347b8515","Type":"ContainerDied","Data":"a2f7cda28625ca7410b96d4480ad6cedc03292ed74aa69923d54bd5c71712a14"}
May 25 13:06:18.376290 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:06:18.376262    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m"
May 25 13:06:18.376290 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:06:18.376285    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m" event={"ID":"56c1b207-9a74-4f0f-a503-ce36347b8515","Type":"ContainerDied","Data":"2c07bcee8765d5197a56f7b2461724245bcc1fb3d8908780552178008fa8606c"}
May 25 13:06:18.376620 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:06:18.376306    2555 scope.go:117] "RemoveContainer" containerID="b843570466d51916e78a9da3ba50c643082cc2659b8c1ecafd6b14da34646c90"
May 25 13:06:18.385562 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:06:18.385537    2555 scope.go:117] "RemoveContainer" containerID="a2f7cda28625ca7410b96d4480ad6cedc03292ed74aa69923d54bd5c71712a14"
May 25 13:06:18.393461 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:06:18.393443    2555 scope.go:117] "RemoveContainer" containerID="18ac68b5afff40c3c0f1573ddac11df45f2d5f4653eaaf8b6a0f3887574e9b1e"
May 25 13:06:18.396693 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:06:18.396667    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m"]
May 25 13:06:18.400495 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:06:18.400470    2555 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-mvq7m"]
May 25 13:06:18.402224 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:06:18.402203    2555 scope.go:117] "RemoveContainer" containerID="b843570466d51916e78a9da3ba50c643082cc2659b8c1ecafd6b14da34646c90"
May 25 13:06:18.402496 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:06:18.402477    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b843570466d51916e78a9da3ba50c643082cc2659b8c1ecafd6b14da34646c90\": container with ID starting with b843570466d51916e78a9da3ba50c643082cc2659b8c1ecafd6b14da34646c90 not found: ID does not exist" containerID="b843570466d51916e78a9da3ba50c643082cc2659b8c1ecafd6b14da34646c90"
May 25 13:06:18.402541 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:06:18.402506    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b843570466d51916e78a9da3ba50c643082cc2659b8c1ecafd6b14da34646c90"} err="failed to get container status \"b843570466d51916e78a9da3ba50c643082cc2659b8c1ecafd6b14da34646c90\": rpc error: code = NotFound desc = could not find container \"b843570466d51916e78a9da3ba50c643082cc2659b8c1ecafd6b14da34646c90\": container with ID starting with b843570466d51916e78a9da3ba50c643082cc2659b8c1ecafd6b14da34646c90 not found: ID does not exist"
May 25 13:06:18.402541 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:06:18.402528    2555 scope.go:117] "RemoveContainer" containerID="a2f7cda28625ca7410b96d4480ad6cedc03292ed74aa69923d54bd5c71712a14"
May 25 13:06:18.402808 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:06:18.402788    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2f7cda28625ca7410b96d4480ad6cedc03292ed74aa69923d54bd5c71712a14\": container with ID starting with a2f7cda28625ca7410b96d4480ad6cedc03292ed74aa69923d54bd5c71712a14 not found: ID does not exist" containerID="a2f7cda28625ca7410b96d4480ad6cedc03292ed74aa69923d54bd5c71712a14"
May 25 13:06:18.402875 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:06:18.402813    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2f7cda28625ca7410b96d4480ad6cedc03292ed74aa69923d54bd5c71712a14"} err="failed to get container status \"a2f7cda28625ca7410b96d4480ad6cedc03292ed74aa69923d54bd5c71712a14\": rpc error: code = NotFound desc = could not find container \"a2f7cda28625ca7410b96d4480ad6cedc03292ed74aa69923d54bd5c71712a14\": container with ID starting with a2f7cda28625ca7410b96d4480ad6cedc03292ed74aa69923d54bd5c71712a14 not found: ID does not exist"
May 25 13:06:18.402875 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:06:18.402829    2555 scope.go:117] "RemoveContainer" containerID="18ac68b5afff40c3c0f1573ddac11df45f2d5f4653eaaf8b6a0f3887574e9b1e"
May 25 13:06:18.403022 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:06:18.403007    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18ac68b5afff40c3c0f1573ddac11df45f2d5f4653eaaf8b6a0f3887574e9b1e\": container with ID starting with 18ac68b5afff40c3c0f1573ddac11df45f2d5f4653eaaf8b6a0f3887574e9b1e not found: ID does not exist" containerID="18ac68b5afff40c3c0f1573ddac11df45f2d5f4653eaaf8b6a0f3887574e9b1e"
May 25 13:06:18.403061 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:06:18.403025    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18ac68b5afff40c3c0f1573ddac11df45f2d5f4653eaaf8b6a0f3887574e9b1e"} err="failed to get container status \"18ac68b5afff40c3c0f1573ddac11df45f2d5f4653eaaf8b6a0f3887574e9b1e\": rpc error: code = NotFound desc = could not find container \"18ac68b5afff40c3c0f1573ddac11df45f2d5f4653eaaf8b6a0f3887574e9b1e\": container with ID starting with 18ac68b5afff40c3c0f1573ddac11df45f2d5f4653eaaf8b6a0f3887574e9b1e not found: ID does not exist"
May 25 13:06:19.599115 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:06:19.599078    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56c1b207-9a74-4f0f-a503-ce36347b8515" path="/var/lib/kubelet/pods/56c1b207-9a74-4f0f-a503-ce36347b8515/volumes"
May 25 13:07:59.096919 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:07:59.096883    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-56gqk"]
May 25 13:07:59.099393 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:07:59.097256    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56c1b207-9a74-4f0f-a503-ce36347b8515" containerName="storage-initializer"
May 25 13:07:59.099393 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:07:59.097267    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="56c1b207-9a74-4f0f-a503-ce36347b8515" containerName="storage-initializer"
May 25 13:07:59.099393 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:07:59.097275    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56c1b207-9a74-4f0f-a503-ce36347b8515" containerName="kserve-container"
May 25 13:07:59.099393 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:07:59.097282    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="56c1b207-9a74-4f0f-a503-ce36347b8515" containerName="kserve-container"
May 25 13:07:59.099393 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:07:59.097295    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56c1b207-9a74-4f0f-a503-ce36347b8515" containerName="kube-rbac-proxy"
May 25 13:07:59.099393 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:07:59.097300    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="56c1b207-9a74-4f0f-a503-ce36347b8515" containerName="kube-rbac-proxy"
May 25 13:07:59.099393 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:07:59.097352    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="56c1b207-9a74-4f0f-a503-ce36347b8515" containerName="kube-rbac-proxy"
May 25 13:07:59.099393 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:07:59.097361    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="56c1b207-9a74-4f0f-a503-ce36347b8515" containerName="kserve-container"
May 25 13:07:59.101950 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:07:59.101917    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-56gqk"
May 25 13:07:59.104630 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:07:59.104597    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6qbkc\""
May 25 13:07:59.104630 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:07:59.104617    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\""
May 25 13:07:59.104860 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:07:59.104617    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-predictor-serving-cert\""
May 25 13:07:59.105094 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:07:59.105072    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 25 13:07:59.105168 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:07:59.105156    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 25 13:07:59.109436 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:07:59.109414    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-56gqk"]
May 25 13:07:59.202472 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:07:59.202432    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f199f36-2d68-464a-84fa-3a9fe148ceb9-proxy-tls\") pod \"isvc-xgboost-predictor-8689c4cfcc-56gqk\" (UID: \"4f199f36-2d68-464a-84fa-3a9fe148ceb9\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-56gqk"
May 25 13:07:59.202702 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:07:59.202483    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg6pz\" (UniqueName: \"kubernetes.io/projected/4f199f36-2d68-464a-84fa-3a9fe148ceb9-kube-api-access-hg6pz\") pod \"isvc-xgboost-predictor-8689c4cfcc-56gqk\" (UID: \"4f199f36-2d68-464a-84fa-3a9fe148ceb9\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-56gqk"
May 25 13:07:59.202702 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:07:59.202511    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f199f36-2d68-464a-84fa-3a9fe148ceb9-isvc-xgboost-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-8689c4cfcc-56gqk\" (UID: \"4f199f36-2d68-464a-84fa-3a9fe148ceb9\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-56gqk"
May 25 13:07:59.202702 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:07:59.202636    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f199f36-2d68-464a-84fa-3a9fe148ceb9-kserve-provision-location\") pod \"isvc-xgboost-predictor-8689c4cfcc-56gqk\" (UID: \"4f199f36-2d68-464a-84fa-3a9fe148ceb9\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-56gqk"
May 25 13:07:59.303238 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:07:59.303198    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f199f36-2d68-464a-84fa-3a9fe148ceb9-kserve-provision-location\") pod \"isvc-xgboost-predictor-8689c4cfcc-56gqk\" (UID: \"4f199f36-2d68-464a-84fa-3a9fe148ceb9\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-56gqk"
May 25 13:07:59.303432 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:07:59.303283    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f199f36-2d68-464a-84fa-3a9fe148ceb9-proxy-tls\") pod \"isvc-xgboost-predictor-8689c4cfcc-56gqk\" (UID: \"4f199f36-2d68-464a-84fa-3a9fe148ceb9\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-56gqk"
May 25 13:07:59.303432 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:07:59.303333    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hg6pz\" (UniqueName: \"kubernetes.io/projected/4f199f36-2d68-464a-84fa-3a9fe148ceb9-kube-api-access-hg6pz\") pod \"isvc-xgboost-predictor-8689c4cfcc-56gqk\" (UID: \"4f199f36-2d68-464a-84fa-3a9fe148ceb9\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-56gqk"
May 25 13:07:59.303432 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:07:59.303366    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f199f36-2d68-464a-84fa-3a9fe148ceb9-isvc-xgboost-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-8689c4cfcc-56gqk\" (UID: \"4f199f36-2d68-464a-84fa-3a9fe148ceb9\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-56gqk"
May 25 13:07:59.303612 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:07:59.303431    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-predictor-serving-cert: secret "isvc-xgboost-predictor-serving-cert" not found
May 25 13:07:59.303612 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:07:59.303510    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f199f36-2d68-464a-84fa-3a9fe148ceb9-proxy-tls podName:4f199f36-2d68-464a-84fa-3a9fe148ceb9 nodeName:}" failed. No retries permitted until 2026-05-25 13:07:59.80348746 +0000 UTC m=+3292.763537844 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4f199f36-2d68-464a-84fa-3a9fe148ceb9-proxy-tls") pod "isvc-xgboost-predictor-8689c4cfcc-56gqk" (UID: "4f199f36-2d68-464a-84fa-3a9fe148ceb9") : secret "isvc-xgboost-predictor-serving-cert" not found
May 25 13:07:59.303783 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:07:59.303621    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f199f36-2d68-464a-84fa-3a9fe148ceb9-kserve-provision-location\") pod \"isvc-xgboost-predictor-8689c4cfcc-56gqk\" (UID: \"4f199f36-2d68-464a-84fa-3a9fe148ceb9\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-56gqk"
May 25 13:07:59.304070 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:07:59.304048    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f199f36-2d68-464a-84fa-3a9fe148ceb9-isvc-xgboost-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-8689c4cfcc-56gqk\" (UID: \"4f199f36-2d68-464a-84fa-3a9fe148ceb9\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-56gqk"
May 25 13:07:59.310636 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:07:59.310615    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg6pz\" (UniqueName: \"kubernetes.io/projected/4f199f36-2d68-464a-84fa-3a9fe148ceb9-kube-api-access-hg6pz\") pod \"isvc-xgboost-predictor-8689c4cfcc-56gqk\" (UID: \"4f199f36-2d68-464a-84fa-3a9fe148ceb9\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-56gqk"
May 25 13:07:59.808256 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:07:59.808218    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f199f36-2d68-464a-84fa-3a9fe148ceb9-proxy-tls\") pod \"isvc-xgboost-predictor-8689c4cfcc-56gqk\" (UID: \"4f199f36-2d68-464a-84fa-3a9fe148ceb9\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-56gqk"
May 25 13:07:59.810777 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:07:59.810752    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f199f36-2d68-464a-84fa-3a9fe148ceb9-proxy-tls\") pod \"isvc-xgboost-predictor-8689c4cfcc-56gqk\" (UID: \"4f199f36-2d68-464a-84fa-3a9fe148ceb9\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-56gqk"
May 25 13:08:00.015782 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:08:00.015743    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-56gqk"
May 25 13:08:00.140492 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:08:00.140464    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-56gqk"]
May 25 13:08:00.142798 ip-10-0-134-255 kubenswrapper[2555]: W0525 13:08:00.142773    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f199f36_2d68_464a_84fa_3a9fe148ceb9.slice/crio-07ea1e2e57f5b357cc684ae48504c4f19301fbb7119385d81347c38587b313c1 WatchSource:0}: Error finding container 07ea1e2e57f5b357cc684ae48504c4f19301fbb7119385d81347c38587b313c1: Status 404 returned error can't find the container with id 07ea1e2e57f5b357cc684ae48504c4f19301fbb7119385d81347c38587b313c1
May 25 13:08:00.708682 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:08:00.708633    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-56gqk" event={"ID":"4f199f36-2d68-464a-84fa-3a9fe148ceb9","Type":"ContainerStarted","Data":"ae7f1968b25e21e28a38f47277f3ee8802d50c63f5ed31f5153bbed9b373d029"}
May 25 13:08:00.708682 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:08:00.708686    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-56gqk" event={"ID":"4f199f36-2d68-464a-84fa-3a9fe148ceb9","Type":"ContainerStarted","Data":"07ea1e2e57f5b357cc684ae48504c4f19301fbb7119385d81347c38587b313c1"}
May 25 13:08:04.722180 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:08:04.722146    2555 generic.go:358] "Generic (PLEG): container finished" podID="4f199f36-2d68-464a-84fa-3a9fe148ceb9" containerID="ae7f1968b25e21e28a38f47277f3ee8802d50c63f5ed31f5153bbed9b373d029" exitCode=0
May 25 13:08:04.722565 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:08:04.722218    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-56gqk" event={"ID":"4f199f36-2d68-464a-84fa-3a9fe148ceb9","Type":"ContainerDied","Data":"ae7f1968b25e21e28a38f47277f3ee8802d50c63f5ed31f5153bbed9b373d029"}
May 25 13:08:07.736438 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:08:07.736404    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-2tqtm_00e6f79c-8f05-4330-aa16-1ada2ae9e9cc/console-operator/1.log"
May 25 13:08:07.744982 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:08:07.744957    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-2tqtm_00e6f79c-8f05-4330-aa16-1ada2ae9e9cc/console-operator/1.log"
May 25 13:08:23.273286 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:08:23.273266    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 13:08:23.799613 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:08:23.799578    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-56gqk" event={"ID":"4f199f36-2d68-464a-84fa-3a9fe148ceb9","Type":"ContainerStarted","Data":"8d5a18cf19741097e79747d68a5811071ea53759449061f7ce56f39f2fe5a97e"}
May 25 13:08:23.799613 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:08:23.799615    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-56gqk" event={"ID":"4f199f36-2d68-464a-84fa-3a9fe148ceb9","Type":"ContainerStarted","Data":"f2e2d31d3a480784c0b71d28f9ec0801be1cbb23902b49ba117ac03977539a73"}
May 25 13:08:23.799851 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:08:23.799824    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-56gqk"
May 25 13:08:23.815423 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:08:23.815371    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-56gqk" podStartSLOduration=6.332406352 podStartE2EDuration="24.815354143s" podCreationTimestamp="2026-05-25 13:07:59 +0000 UTC" firstStartedPulling="2026-05-25 13:08:04.723444097 +0000 UTC m=+3297.683494484" lastFinishedPulling="2026-05-25 13:08:23.206391891 +0000 UTC m=+3316.166442275" observedRunningTime="2026-05-25 13:08:23.814499867 +0000 UTC m=+3316.774550274" watchObservedRunningTime="2026-05-25 13:08:23.815354143 +0000 UTC m=+3316.775404550"
May 25 13:08:24.803392 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:08:24.803352    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-56gqk"
May 25 13:08:24.804513 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:08:24.804486    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-56gqk" podUID="4f199f36-2d68-464a-84fa-3a9fe148ceb9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
May 25 13:08:25.806602 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:08:25.806561    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-56gqk" podUID="4f199f36-2d68-464a-84fa-3a9fe148ceb9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
May 25 13:08:30.812340 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:08:30.812255    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-56gqk"
May 25 13:08:30.812963 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:08:30.812852    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-56gqk" podUID="4f199f36-2d68-464a-84fa-3a9fe148ceb9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
May 25 13:08:40.813488 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:08:40.813445    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-56gqk" podUID="4f199f36-2d68-464a-84fa-3a9fe148ceb9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
May 25 13:08:50.812889 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:08:50.812849    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-56gqk" podUID="4f199f36-2d68-464a-84fa-3a9fe148ceb9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
May 25 13:09:00.812811 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:00.812767    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-56gqk" podUID="4f199f36-2d68-464a-84fa-3a9fe148ceb9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
May 25 13:09:10.813240 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:10.813195    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-56gqk" podUID="4f199f36-2d68-464a-84fa-3a9fe148ceb9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
May 25 13:09:20.813706 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:20.813661    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-56gqk" podUID="4f199f36-2d68-464a-84fa-3a9fe148ceb9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
May 25 13:09:30.814391 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:30.814357    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-56gqk"
May 25 13:09:39.208458 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:39.208403    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-56gqk"]
May 25 13:09:39.208943 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:39.208809    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-56gqk" podUID="4f199f36-2d68-464a-84fa-3a9fe148ceb9" containerName="kserve-container" containerID="cri-o://f2e2d31d3a480784c0b71d28f9ec0801be1cbb23902b49ba117ac03977539a73" gracePeriod=30
May 25 13:09:39.208943 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:39.208891    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-56gqk" podUID="4f199f36-2d68-464a-84fa-3a9fe148ceb9" containerName="kube-rbac-proxy" containerID="cri-o://8d5a18cf19741097e79747d68a5811071ea53759449061f7ce56f39f2fe5a97e" gracePeriod=30
May 25 13:09:39.317827 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:39.317796    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7t2dt"]
May 25 13:09:39.320498 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:39.320473    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7t2dt"
May 25 13:09:39.322180 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:39.322149    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
May 25 13:09:39.322295 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:39.322157    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
May 25 13:09:39.330895 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:39.330857    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7t2dt"]
May 25 13:09:39.362688 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:39.362634    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7t2dt\" (UID: \"7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7t2dt"
May 25 13:09:39.362688 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:39.362691    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7t2dt\" (UID: \"7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7t2dt"
May 25 13:09:39.362886 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:39.362736    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgrg2\" (UniqueName: \"kubernetes.io/projected/7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114-kube-api-access-bgrg2\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7t2dt\" (UID: \"7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7t2dt"
May 25 13:09:39.362886 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:39.362756    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7t2dt\" (UID: \"7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7t2dt"
May 25 13:09:39.469391 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:39.463891    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7t2dt\" (UID: \"7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7t2dt"
May 25 13:09:39.469391 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:39.463953    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7t2dt\" (UID: \"7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7t2dt"
May 25 13:09:39.469391 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:39.464019    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bgrg2\" (UniqueName: \"kubernetes.io/projected/7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114-kube-api-access-bgrg2\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7t2dt\" (UID: \"7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7t2dt"
May 25 13:09:39.469391 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:39.464060    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7t2dt\" (UID: \"7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7t2dt"
May 25 13:09:39.469391 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:39.465000    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7t2dt\" (UID: \"7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7t2dt"
May 25 13:09:39.469391 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:09:39.465154    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-serving-cert: secret "isvc-xgboost-v2-mlserver-predictor-serving-cert" not found
May 25 13:09:39.469391 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:09:39.465222    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114-proxy-tls podName:7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114 nodeName:}" failed. No retries permitted until 2026-05-25 13:09:39.965195757 +0000 UTC m=+3392.925246151 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114-proxy-tls") pod "isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7t2dt" (UID: "7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114") : secret "isvc-xgboost-v2-mlserver-predictor-serving-cert" not found
May 25 13:09:39.469391 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:39.465763    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7t2dt\" (UID: \"7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7t2dt"
May 25 13:09:39.475383 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:39.475361    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgrg2\" (UniqueName: \"kubernetes.io/projected/7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114-kube-api-access-bgrg2\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7t2dt\" (UID: \"7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7t2dt"
May 25 13:09:39.969597 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:39.969547    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7t2dt\" (UID: \"7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7t2dt"
May 25 13:09:39.972162 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:39.972134    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7t2dt\" (UID: \"7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7t2dt"
May 25 13:09:40.061890 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:40.061852    2555 generic.go:358] "Generic (PLEG): container finished" podID="4f199f36-2d68-464a-84fa-3a9fe148ceb9" containerID="8d5a18cf19741097e79747d68a5811071ea53759449061f7ce56f39f2fe5a97e" exitCode=2
May 25 13:09:40.062066 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:40.061927    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-56gqk" event={"ID":"4f199f36-2d68-464a-84fa-3a9fe148ceb9","Type":"ContainerDied","Data":"8d5a18cf19741097e79747d68a5811071ea53759449061f7ce56f39f2fe5a97e"}
May 25 13:09:40.232756 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:40.232662    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7t2dt"
May 25 13:09:40.366507 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:40.366472    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7t2dt"]
May 25 13:09:40.370005 ip-10-0-134-255 kubenswrapper[2555]: W0525 13:09:40.369972    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c20f3fc_7f16_4a7d_afc2_6cf38cbc7114.slice/crio-13cf8c0c9b50cab88d1b49554f512d74a1a07a1b50171bfb3f690da8f5df8642 WatchSource:0}: Error finding container 13cf8c0c9b50cab88d1b49554f512d74a1a07a1b50171bfb3f690da8f5df8642: Status 404 returned error can't find the container with id 13cf8c0c9b50cab88d1b49554f512d74a1a07a1b50171bfb3f690da8f5df8642
May 25 13:09:40.807145 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:40.807097    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-56gqk" podUID="4f199f36-2d68-464a-84fa-3a9fe148ceb9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.58:8643/healthz\": dial tcp 10.134.0.58:8643: connect: connection refused"
May 25 13:09:40.813759 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:40.813718    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-56gqk" podUID="4f199f36-2d68-464a-84fa-3a9fe148ceb9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
May 25 13:09:41.068414 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:41.068318    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7t2dt" event={"ID":"7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114","Type":"ContainerStarted","Data":"e998947ec2b73577cda44a984bc1a4b225024997484e6bfadb3bdfef701b2bc4"}
May 25 13:09:41.068414 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:41.068367    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7t2dt" event={"ID":"7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114","Type":"ContainerStarted","Data":"13cf8c0c9b50cab88d1b49554f512d74a1a07a1b50171bfb3f690da8f5df8642"}
May 25 13:09:42.956233 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:42.956210    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-56gqk"
May 25 13:09:42.996134 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:42.996106    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f199f36-2d68-464a-84fa-3a9fe148ceb9-isvc-xgboost-kube-rbac-proxy-sar-config\") pod \"4f199f36-2d68-464a-84fa-3a9fe148ceb9\" (UID: \"4f199f36-2d68-464a-84fa-3a9fe148ceb9\") "
May 25 13:09:42.996264 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:42.996148    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f199f36-2d68-464a-84fa-3a9fe148ceb9-kserve-provision-location\") pod \"4f199f36-2d68-464a-84fa-3a9fe148ceb9\" (UID: \"4f199f36-2d68-464a-84fa-3a9fe148ceb9\") "
May 25 13:09:42.996264 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:42.996183    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f199f36-2d68-464a-84fa-3a9fe148ceb9-proxy-tls\") pod \"4f199f36-2d68-464a-84fa-3a9fe148ceb9\" (UID: \"4f199f36-2d68-464a-84fa-3a9fe148ceb9\") "
May 25 13:09:42.996264 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:42.996207    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hg6pz\" (UniqueName: \"kubernetes.io/projected/4f199f36-2d68-464a-84fa-3a9fe148ceb9-kube-api-access-hg6pz\") pod \"4f199f36-2d68-464a-84fa-3a9fe148ceb9\" (UID: \"4f199f36-2d68-464a-84fa-3a9fe148ceb9\") "
May 25 13:09:42.996520 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:42.996498    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f199f36-2d68-464a-84fa-3a9fe148ceb9-isvc-xgboost-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-kube-rbac-proxy-sar-config") pod "4f199f36-2d68-464a-84fa-3a9fe148ceb9" (UID: "4f199f36-2d68-464a-84fa-3a9fe148ceb9"). InnerVolumeSpecName "isvc-xgboost-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:09:42.996600 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:42.996495    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f199f36-2d68-464a-84fa-3a9fe148ceb9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4f199f36-2d68-464a-84fa-3a9fe148ceb9" (UID: "4f199f36-2d68-464a-84fa-3a9fe148ceb9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:09:42.998321 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:42.998300    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f199f36-2d68-464a-84fa-3a9fe148ceb9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4f199f36-2d68-464a-84fa-3a9fe148ceb9" (UID: "4f199f36-2d68-464a-84fa-3a9fe148ceb9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:09:42.998321 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:42.998308    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f199f36-2d68-464a-84fa-3a9fe148ceb9-kube-api-access-hg6pz" (OuterVolumeSpecName: "kube-api-access-hg6pz") pod "4f199f36-2d68-464a-84fa-3a9fe148ceb9" (UID: "4f199f36-2d68-464a-84fa-3a9fe148ceb9"). InnerVolumeSpecName "kube-api-access-hg6pz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:09:43.077097 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:43.077036    2555 generic.go:358] "Generic (PLEG): container finished" podID="4f199f36-2d68-464a-84fa-3a9fe148ceb9" containerID="f2e2d31d3a480784c0b71d28f9ec0801be1cbb23902b49ba117ac03977539a73" exitCode=0
May 25 13:09:43.077097 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:43.077087    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-56gqk" event={"ID":"4f199f36-2d68-464a-84fa-3a9fe148ceb9","Type":"ContainerDied","Data":"f2e2d31d3a480784c0b71d28f9ec0801be1cbb23902b49ba117ac03977539a73"}
May 25 13:09:43.077231 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:43.077115    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-56gqk" event={"ID":"4f199f36-2d68-464a-84fa-3a9fe148ceb9","Type":"ContainerDied","Data":"07ea1e2e57f5b357cc684ae48504c4f19301fbb7119385d81347c38587b313c1"}
May 25 13:09:43.077231 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:43.077125    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-56gqk"
May 25 13:09:43.077297 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:43.077130    2555 scope.go:117] "RemoveContainer" containerID="8d5a18cf19741097e79747d68a5811071ea53759449061f7ce56f39f2fe5a97e"
May 25 13:09:43.086073 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:43.086050    2555 scope.go:117] "RemoveContainer" containerID="f2e2d31d3a480784c0b71d28f9ec0801be1cbb23902b49ba117ac03977539a73"
May 25 13:09:43.093154 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:43.093135    2555 scope.go:117] "RemoveContainer" containerID="ae7f1968b25e21e28a38f47277f3ee8802d50c63f5ed31f5153bbed9b373d029"
May 25 13:09:43.097317 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:43.097237    2555 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f199f36-2d68-464a-84fa-3a9fe148ceb9-isvc-xgboost-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:09:43.097317 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:43.097265    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f199f36-2d68-464a-84fa-3a9fe148ceb9-kserve-provision-location\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:09:43.097317 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:43.097281    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f199f36-2d68-464a-84fa-3a9fe148ceb9-proxy-tls\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:09:43.097317 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:43.097295    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg6pz\" (UniqueName: \"kubernetes.io/projected/4f199f36-2d68-464a-84fa-3a9fe148ceb9-kube-api-access-hg6pz\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:09:43.100185 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:43.100163    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-56gqk"]
May 25 13:09:43.101894 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:43.101871    2555 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-56gqk"]
May 25 13:09:43.102937 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:43.102922    2555 scope.go:117] "RemoveContainer" containerID="8d5a18cf19741097e79747d68a5811071ea53759449061f7ce56f39f2fe5a97e"
May 25 13:09:43.103161 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:09:43.103144    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d5a18cf19741097e79747d68a5811071ea53759449061f7ce56f39f2fe5a97e\": container with ID starting with 8d5a18cf19741097e79747d68a5811071ea53759449061f7ce56f39f2fe5a97e not found: ID does not exist" containerID="8d5a18cf19741097e79747d68a5811071ea53759449061f7ce56f39f2fe5a97e"
May 25 13:09:43.103209 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:43.103168    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d5a18cf19741097e79747d68a5811071ea53759449061f7ce56f39f2fe5a97e"} err="failed to get container status \"8d5a18cf19741097e79747d68a5811071ea53759449061f7ce56f39f2fe5a97e\": rpc error: code = NotFound desc = could not find container \"8d5a18cf19741097e79747d68a5811071ea53759449061f7ce56f39f2fe5a97e\": container with ID starting with 8d5a18cf19741097e79747d68a5811071ea53759449061f7ce56f39f2fe5a97e not found: ID does not exist"
May 25 13:09:43.103209 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:43.103184    2555 scope.go:117] "RemoveContainer" containerID="f2e2d31d3a480784c0b71d28f9ec0801be1cbb23902b49ba117ac03977539a73"
May 25 13:09:43.103374 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:09:43.103359    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2e2d31d3a480784c0b71d28f9ec0801be1cbb23902b49ba117ac03977539a73\": container with ID starting with f2e2d31d3a480784c0b71d28f9ec0801be1cbb23902b49ba117ac03977539a73 not found: ID does not exist" containerID="f2e2d31d3a480784c0b71d28f9ec0801be1cbb23902b49ba117ac03977539a73"
May 25 13:09:43.103414 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:43.103378    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2e2d31d3a480784c0b71d28f9ec0801be1cbb23902b49ba117ac03977539a73"} err="failed to get container status \"f2e2d31d3a480784c0b71d28f9ec0801be1cbb23902b49ba117ac03977539a73\": rpc error: code = NotFound desc = could not find container \"f2e2d31d3a480784c0b71d28f9ec0801be1cbb23902b49ba117ac03977539a73\": container with ID starting with f2e2d31d3a480784c0b71d28f9ec0801be1cbb23902b49ba117ac03977539a73 not found: ID does not exist"
May 25 13:09:43.103414 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:43.103389    2555 scope.go:117] "RemoveContainer" containerID="ae7f1968b25e21e28a38f47277f3ee8802d50c63f5ed31f5153bbed9b373d029"
May 25 13:09:43.103620 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:09:43.103602    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae7f1968b25e21e28a38f47277f3ee8802d50c63f5ed31f5153bbed9b373d029\": container with ID starting with ae7f1968b25e21e28a38f47277f3ee8802d50c63f5ed31f5153bbed9b373d029 not found: ID does not exist" containerID="ae7f1968b25e21e28a38f47277f3ee8802d50c63f5ed31f5153bbed9b373d029"
May 25 13:09:43.103690 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:43.103627    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae7f1968b25e21e28a38f47277f3ee8802d50c63f5ed31f5153bbed9b373d029"} err="failed to get container status \"ae7f1968b25e21e28a38f47277f3ee8802d50c63f5ed31f5153bbed9b373d029\": rpc error: code = NotFound desc = could not find container \"ae7f1968b25e21e28a38f47277f3ee8802d50c63f5ed31f5153bbed9b373d029\": container with ID starting with ae7f1968b25e21e28a38f47277f3ee8802d50c63f5ed31f5153bbed9b373d029 not found: ID does not exist"
May 25 13:09:43.599962 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:43.599928    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f199f36-2d68-464a-84fa-3a9fe148ceb9" path="/var/lib/kubelet/pods/4f199f36-2d68-464a-84fa-3a9fe148ceb9/volumes"
May 25 13:09:45.086322 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:45.086287    2555 generic.go:358] "Generic (PLEG): container finished" podID="7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114" containerID="e998947ec2b73577cda44a984bc1a4b225024997484e6bfadb3bdfef701b2bc4" exitCode=0
May 25 13:09:45.086745 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:45.086362    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7t2dt" event={"ID":"7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114","Type":"ContainerDied","Data":"e998947ec2b73577cda44a984bc1a4b225024997484e6bfadb3bdfef701b2bc4"}
May 25 13:09:46.091532 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:46.091495    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7t2dt" event={"ID":"7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114","Type":"ContainerStarted","Data":"ca2a9d54ff7588156e0f72c7533f365360f5b128903481eae6536b0b12c84f84"}
May 25 13:09:46.091532 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:46.091537    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7t2dt" event={"ID":"7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114","Type":"ContainerStarted","Data":"9dc769cdc3753a8deb634f7936511f4991a8c5e32522eef7c6bede4ec0d9480c"}
May 25 13:09:46.092001 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:46.091806    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7t2dt"
May 25 13:09:46.108886 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:46.108836    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7t2dt" podStartSLOduration=7.108821943 podStartE2EDuration="7.108821943s" podCreationTimestamp="2026-05-25 13:09:39 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:09:46.10749304 +0000 UTC m=+3399.067543446" watchObservedRunningTime="2026-05-25 13:09:46.108821943 +0000 UTC m=+3399.068872348"
May 25 13:09:47.094445 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:47.094411    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7t2dt"
May 25 13:09:53.102920 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:09:53.102891    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7t2dt"
May 25 13:10:23.106570 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:23.106538    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7t2dt"
May 25 13:10:29.382693 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:29.382658    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7t2dt"]
May 25 13:10:29.383247 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:29.382971    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7t2dt" podUID="7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114" containerName="kserve-container" containerID="cri-o://9dc769cdc3753a8deb634f7936511f4991a8c5e32522eef7c6bede4ec0d9480c" gracePeriod=30
May 25 13:10:29.383247 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:29.383011    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7t2dt" podUID="7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114" containerName="kube-rbac-proxy" containerID="cri-o://ca2a9d54ff7588156e0f72c7533f365360f5b128903481eae6536b0b12c84f84" gracePeriod=30
May 25 13:10:29.475785 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:29.475756    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-s5ssz"]
May 25 13:10:29.476096 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:29.476085    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f199f36-2d68-464a-84fa-3a9fe148ceb9" containerName="kube-rbac-proxy"
May 25 13:10:29.476140 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:29.476098    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f199f36-2d68-464a-84fa-3a9fe148ceb9" containerName="kube-rbac-proxy"
May 25 13:10:29.476140 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:29.476121    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f199f36-2d68-464a-84fa-3a9fe148ceb9" containerName="storage-initializer"
May 25 13:10:29.476140 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:29.476127    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f199f36-2d68-464a-84fa-3a9fe148ceb9" containerName="storage-initializer"
May 25 13:10:29.476140 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:29.476136    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f199f36-2d68-464a-84fa-3a9fe148ceb9" containerName="kserve-container"
May 25 13:10:29.476140 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:29.476141    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f199f36-2d68-464a-84fa-3a9fe148ceb9" containerName="kserve-container"
May 25 13:10:29.476299 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:29.476188    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="4f199f36-2d68-464a-84fa-3a9fe148ceb9" containerName="kube-rbac-proxy"
May 25 13:10:29.476299 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:29.476198    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="4f199f36-2d68-464a-84fa-3a9fe148ceb9" containerName="kserve-container"
May 25 13:10:29.478293 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:29.478277    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-s5ssz"
May 25 13:10:29.479972 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:29.479949    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
May 25 13:10:29.479972 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:29.479964    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
May 25 13:10:29.488977 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:29.488951    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-s5ssz"]
May 25 13:10:29.492832 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:29.492809    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b7c6b019-57cc-43f3-a475-7755083971f3-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-s5ssz\" (UID: \"b7c6b019-57cc-43f3-a475-7755083971f3\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-s5ssz"
May 25 13:10:29.492963 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:29.492872    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b7c6b019-57cc-43f3-a475-7755083971f3-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-s5ssz\" (UID: \"b7c6b019-57cc-43f3-a475-7755083971f3\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-s5ssz"
May 25 13:10:29.493034 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:29.492963    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c7c6\" (UniqueName: \"kubernetes.io/projected/b7c6b019-57cc-43f3-a475-7755083971f3-kube-api-access-5c7c6\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-s5ssz\" (UID: \"b7c6b019-57cc-43f3-a475-7755083971f3\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-s5ssz"
May 25 13:10:29.493034 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:29.492993    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7c6b019-57cc-43f3-a475-7755083971f3-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-s5ssz\" (UID: \"b7c6b019-57cc-43f3-a475-7755083971f3\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-s5ssz"
May 25 13:10:29.593986 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:29.593952    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b7c6b019-57cc-43f3-a475-7755083971f3-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-s5ssz\" (UID: \"b7c6b019-57cc-43f3-a475-7755083971f3\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-s5ssz"
May 25 13:10:29.594138 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:29.594008    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b7c6b019-57cc-43f3-a475-7755083971f3-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-s5ssz\" (UID: \"b7c6b019-57cc-43f3-a475-7755083971f3\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-s5ssz"
May 25 13:10:29.594138 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:29.594065    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5c7c6\" (UniqueName: \"kubernetes.io/projected/b7c6b019-57cc-43f3-a475-7755083971f3-kube-api-access-5c7c6\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-s5ssz\" (UID: \"b7c6b019-57cc-43f3-a475-7755083971f3\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-s5ssz"
May 25 13:10:29.594138 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:29.594088    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7c6b019-57cc-43f3-a475-7755083971f3-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-s5ssz\" (UID: \"b7c6b019-57cc-43f3-a475-7755083971f3\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-s5ssz"
May 25 13:10:29.594892 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:10:29.594777    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-serving-cert: secret "xgboost-v2-mlserver-predictor-serving-cert" not found
May 25 13:10:29.594892 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:29.594835    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b7c6b019-57cc-43f3-a475-7755083971f3-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-s5ssz\" (UID: \"b7c6b019-57cc-43f3-a475-7755083971f3\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-s5ssz"
May 25 13:10:29.594892 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:10:29.594869    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7c6b019-57cc-43f3-a475-7755083971f3-proxy-tls podName:b7c6b019-57cc-43f3-a475-7755083971f3 nodeName:}" failed. No retries permitted until 2026-05-25 13:10:30.094846532 +0000 UTC m=+3443.054896916 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b7c6b019-57cc-43f3-a475-7755083971f3-proxy-tls") pod "xgboost-v2-mlserver-predictor-7799869d6f-s5ssz" (UID: "b7c6b019-57cc-43f3-a475-7755083971f3") : secret "xgboost-v2-mlserver-predictor-serving-cert" not found
May 25 13:10:29.598669 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:29.595182    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b7c6b019-57cc-43f3-a475-7755083971f3-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-s5ssz\" (UID: \"b7c6b019-57cc-43f3-a475-7755083971f3\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-s5ssz"
May 25 13:10:29.604218 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:29.604189    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c7c6\" (UniqueName: \"kubernetes.io/projected/b7c6b019-57cc-43f3-a475-7755083971f3-kube-api-access-5c7c6\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-s5ssz\" (UID: \"b7c6b019-57cc-43f3-a475-7755083971f3\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-s5ssz"
May 25 13:10:30.096730 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:30.096692    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7c6b019-57cc-43f3-a475-7755083971f3-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-s5ssz\" (UID: \"b7c6b019-57cc-43f3-a475-7755083971f3\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-s5ssz"
May 25 13:10:30.099149 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:30.099115    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7c6b019-57cc-43f3-a475-7755083971f3-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-s5ssz\" (UID: \"b7c6b019-57cc-43f3-a475-7755083971f3\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-s5ssz"
May 25 13:10:30.235500 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:30.235464    2555 generic.go:358] "Generic (PLEG): container finished" podID="7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114" containerID="ca2a9d54ff7588156e0f72c7533f365360f5b128903481eae6536b0b12c84f84" exitCode=2
May 25 13:10:30.235662 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:30.235513    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7t2dt" event={"ID":"7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114","Type":"ContainerDied","Data":"ca2a9d54ff7588156e0f72c7533f365360f5b128903481eae6536b0b12c84f84"}
May 25 13:10:30.392186 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:30.392098    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-s5ssz"
May 25 13:10:30.519013 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:30.518967    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-s5ssz"]
May 25 13:10:30.522134 ip-10-0-134-255 kubenswrapper[2555]: W0525 13:10:30.522103    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7c6b019_57cc_43f3_a475_7755083971f3.slice/crio-2f754dd9fd2c09b3312db3fa1076a19460af44bb2735e431d1b187ed73991411 WatchSource:0}: Error finding container 2f754dd9fd2c09b3312db3fa1076a19460af44bb2735e431d1b187ed73991411: Status 404 returned error can't find the container with id 2f754dd9fd2c09b3312db3fa1076a19460af44bb2735e431d1b187ed73991411
May 25 13:10:31.241009 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:31.240972    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-s5ssz" event={"ID":"b7c6b019-57cc-43f3-a475-7755083971f3","Type":"ContainerStarted","Data":"043436619e2c1785bb0ea5bbb16efe0e6b813891df247d788715c2ff24ecc941"}
May 25 13:10:31.241009 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:31.241015    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-s5ssz" event={"ID":"b7c6b019-57cc-43f3-a475-7755083971f3","Type":"ContainerStarted","Data":"2f754dd9fd2c09b3312db3fa1076a19460af44bb2735e431d1b187ed73991411"}
May 25 13:10:33.098446 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:33.098404    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7t2dt" podUID="7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.59:8643/healthz\": dial tcp 10.134.0.59:8643: connect: connection refused"
May 25 13:10:33.103916 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:33.103890    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7t2dt" podUID="7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.59:8080/v2/models/isvc-xgboost-v2-mlserver/ready\": dial tcp 10.134.0.59:8080: connect: connection refused"
May 25 13:10:34.251890 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:34.251862    2555 generic.go:358] "Generic (PLEG): container finished" podID="b7c6b019-57cc-43f3-a475-7755083971f3" containerID="043436619e2c1785bb0ea5bbb16efe0e6b813891df247d788715c2ff24ecc941" exitCode=0
May 25 13:10:34.252165 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:34.251930    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-s5ssz" event={"ID":"b7c6b019-57cc-43f3-a475-7755083971f3","Type":"ContainerDied","Data":"043436619e2c1785bb0ea5bbb16efe0e6b813891df247d788715c2ff24ecc941"}
May 25 13:10:35.257281 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:35.257243    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-s5ssz" event={"ID":"b7c6b019-57cc-43f3-a475-7755083971f3","Type":"ContainerStarted","Data":"d70baa681f62370ac182ea8f0c979948970c973647fdb528ff0b78d9d7c0c429"}
May 25 13:10:35.257281 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:35.257286    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-s5ssz" event={"ID":"b7c6b019-57cc-43f3-a475-7755083971f3","Type":"ContainerStarted","Data":"63d94457eb8d817715578d463d081fc8712a2d58186899762dbf8cbe7140dd6c"}
May 25 13:10:35.257763 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:35.257480    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-s5ssz"
May 25 13:10:35.274116 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:35.274074    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-s5ssz" podStartSLOduration=6.274062235 podStartE2EDuration="6.274062235s" podCreationTimestamp="2026-05-25 13:10:29 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:10:35.272730402 +0000 UTC m=+3448.232780808" watchObservedRunningTime="2026-05-25 13:10:35.274062235 +0000 UTC m=+3448.234112641"
May 25 13:10:35.736616 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:35.736592    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7t2dt"
May 25 13:10:35.739216 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:35.739196    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114-kserve-provision-location\") pod \"7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114\" (UID: \"7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114\") "
May 25 13:10:35.739301 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:35.739247    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114-proxy-tls\") pod \"7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114\" (UID: \"7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114\") "
May 25 13:10:35.739301 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:35.739282    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114\" (UID: \"7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114\") "
May 25 13:10:35.739407 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:35.739307    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bgrg2\" (UniqueName: \"kubernetes.io/projected/7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114-kube-api-access-bgrg2\") pod \"7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114\" (UID: \"7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114\") "
May 25 13:10:35.739598 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:35.739571    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114" (UID: "7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:10:35.739705 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:35.739683    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config") pod "7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114" (UID: "7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114"). InnerVolumeSpecName "isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:10:35.741287 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:35.741263    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114-kube-api-access-bgrg2" (OuterVolumeSpecName: "kube-api-access-bgrg2") pod "7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114" (UID: "7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114"). InnerVolumeSpecName "kube-api-access-bgrg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:10:35.741369 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:35.741273    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114" (UID: "7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:10:35.840664 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:35.840585    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114-kserve-provision-location\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:10:35.840664 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:35.840608    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114-proxy-tls\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:10:35.840664 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:35.840619    2555 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:10:35.840664 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:35.840628    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgrg2\" (UniqueName: \"kubernetes.io/projected/7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114-kube-api-access-bgrg2\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:10:36.262013 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:36.261972    2555 generic.go:358] "Generic (PLEG): container finished" podID="7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114" containerID="9dc769cdc3753a8deb634f7936511f4991a8c5e32522eef7c6bede4ec0d9480c" exitCode=0
May 25 13:10:36.262013 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:36.262006    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7t2dt" event={"ID":"7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114","Type":"ContainerDied","Data":"9dc769cdc3753a8deb634f7936511f4991a8c5e32522eef7c6bede4ec0d9480c"}
May 25 13:10:36.262537 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:36.262058    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7t2dt" event={"ID":"7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114","Type":"ContainerDied","Data":"13cf8c0c9b50cab88d1b49554f512d74a1a07a1b50171bfb3f690da8f5df8642"}
May 25 13:10:36.262537 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:36.262076    2555 scope.go:117] "RemoveContainer" containerID="ca2a9d54ff7588156e0f72c7533f365360f5b128903481eae6536b0b12c84f84"
May 25 13:10:36.262537 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:36.262087    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7t2dt"
May 25 13:10:36.262741 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:36.262666    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-s5ssz"
May 25 13:10:36.273356 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:36.273340    2555 scope.go:117] "RemoveContainer" containerID="9dc769cdc3753a8deb634f7936511f4991a8c5e32522eef7c6bede4ec0d9480c"
May 25 13:10:36.282215 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:36.282195    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7t2dt"]
May 25 13:10:36.282854 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:36.282832    2555 scope.go:117] "RemoveContainer" containerID="e998947ec2b73577cda44a984bc1a4b225024997484e6bfadb3bdfef701b2bc4"
May 25 13:10:36.285431 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:36.285412    2555 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-7t2dt"]
May 25 13:10:36.290456 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:36.290438    2555 scope.go:117] "RemoveContainer" containerID="ca2a9d54ff7588156e0f72c7533f365360f5b128903481eae6536b0b12c84f84"
May 25 13:10:36.290727 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:10:36.290707    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca2a9d54ff7588156e0f72c7533f365360f5b128903481eae6536b0b12c84f84\": container with ID starting with ca2a9d54ff7588156e0f72c7533f365360f5b128903481eae6536b0b12c84f84 not found: ID does not exist" containerID="ca2a9d54ff7588156e0f72c7533f365360f5b128903481eae6536b0b12c84f84"
May 25 13:10:36.290783 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:36.290735    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca2a9d54ff7588156e0f72c7533f365360f5b128903481eae6536b0b12c84f84"} err="failed to get container status \"ca2a9d54ff7588156e0f72c7533f365360f5b128903481eae6536b0b12c84f84\": rpc error: code = NotFound desc = could not find container \"ca2a9d54ff7588156e0f72c7533f365360f5b128903481eae6536b0b12c84f84\": container with ID starting with ca2a9d54ff7588156e0f72c7533f365360f5b128903481eae6536b0b12c84f84 not found: ID does not exist"
May 25 13:10:36.290783 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:36.290753    2555 scope.go:117] "RemoveContainer" containerID="9dc769cdc3753a8deb634f7936511f4991a8c5e32522eef7c6bede4ec0d9480c"
May 25 13:10:36.290965 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:10:36.290951    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dc769cdc3753a8deb634f7936511f4991a8c5e32522eef7c6bede4ec0d9480c\": container with ID starting with 9dc769cdc3753a8deb634f7936511f4991a8c5e32522eef7c6bede4ec0d9480c not found: ID does not exist" containerID="9dc769cdc3753a8deb634f7936511f4991a8c5e32522eef7c6bede4ec0d9480c"
May 25 13:10:36.291008 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:36.290968    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dc769cdc3753a8deb634f7936511f4991a8c5e32522eef7c6bede4ec0d9480c"} err="failed to get container status \"9dc769cdc3753a8deb634f7936511f4991a8c5e32522eef7c6bede4ec0d9480c\": rpc error: code = NotFound desc = could not find container \"9dc769cdc3753a8deb634f7936511f4991a8c5e32522eef7c6bede4ec0d9480c\": container with ID starting with 9dc769cdc3753a8deb634f7936511f4991a8c5e32522eef7c6bede4ec0d9480c not found: ID does not exist"
May 25 13:10:36.291008 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:36.290980    2555 scope.go:117] "RemoveContainer" containerID="e998947ec2b73577cda44a984bc1a4b225024997484e6bfadb3bdfef701b2bc4"
May 25 13:10:36.291209 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:10:36.291190    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e998947ec2b73577cda44a984bc1a4b225024997484e6bfadb3bdfef701b2bc4\": container with ID starting with e998947ec2b73577cda44a984bc1a4b225024997484e6bfadb3bdfef701b2bc4 not found: ID does not exist" containerID="e998947ec2b73577cda44a984bc1a4b225024997484e6bfadb3bdfef701b2bc4"
May 25 13:10:36.291258 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:36.291211    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e998947ec2b73577cda44a984bc1a4b225024997484e6bfadb3bdfef701b2bc4"} err="failed to get container status \"e998947ec2b73577cda44a984bc1a4b225024997484e6bfadb3bdfef701b2bc4\": rpc error: code = NotFound desc = could not find container \"e998947ec2b73577cda44a984bc1a4b225024997484e6bfadb3bdfef701b2bc4\": container with ID starting with e998947ec2b73577cda44a984bc1a4b225024997484e6bfadb3bdfef701b2bc4 not found: ID does not exist"
May 25 13:10:37.599509 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:37.599475    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114" path="/var/lib/kubelet/pods/7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114/volumes"
May 25 13:10:42.271967 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:10:42.271938    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-s5ssz"
May 25 13:11:12.276265 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:11:12.276236    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-s5ssz"
May 25 13:11:19.542750 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:11:19.542712    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-s5ssz"]
May 25 13:11:19.543207 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:11:19.543153    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-s5ssz" podUID="b7c6b019-57cc-43f3-a475-7755083971f3" containerName="kserve-container" containerID="cri-o://63d94457eb8d817715578d463d081fc8712a2d58186899762dbf8cbe7140dd6c" gracePeriod=30
May 25 13:11:19.543494 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:11:19.543409    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-s5ssz" podUID="b7c6b019-57cc-43f3-a475-7755083971f3" containerName="kube-rbac-proxy" containerID="cri-o://d70baa681f62370ac182ea8f0c979948970c973647fdb528ff0b78d9d7c0c429" gracePeriod=30
May 25 13:11:20.413871 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:11:20.413835    2555 generic.go:358] "Generic (PLEG): container finished" podID="b7c6b019-57cc-43f3-a475-7755083971f3" containerID="d70baa681f62370ac182ea8f0c979948970c973647fdb528ff0b78d9d7c0c429" exitCode=2
May 25 13:11:20.414042 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:11:20.413909    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-s5ssz" event={"ID":"b7c6b019-57cc-43f3-a475-7755083971f3","Type":"ContainerDied","Data":"d70baa681f62370ac182ea8f0c979948970c973647fdb528ff0b78d9d7c0c429"}
May 25 13:11:22.267893 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:11:22.267854    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-s5ssz" podUID="b7c6b019-57cc-43f3-a475-7755083971f3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.60:8643/healthz\": dial tcp 10.134.0.60:8643: connect: connection refused"
May 25 13:11:22.272823 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:11:22.272798    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-s5ssz" podUID="b7c6b019-57cc-43f3-a475-7755083971f3" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.60:8080/v2/models/xgboost-v2-mlserver/ready\": dial tcp 10.134.0.60:8080: connect: connection refused"
May 25 13:11:25.377919 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:11:25.377898    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-s5ssz"
May 25 13:11:25.437327 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:11:25.437264    2555 generic.go:358] "Generic (PLEG): container finished" podID="b7c6b019-57cc-43f3-a475-7755083971f3" containerID="63d94457eb8d817715578d463d081fc8712a2d58186899762dbf8cbe7140dd6c" exitCode=0
May 25 13:11:25.437438 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:11:25.437339    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-s5ssz"
May 25 13:11:25.437438 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:11:25.437342    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-s5ssz" event={"ID":"b7c6b019-57cc-43f3-a475-7755083971f3","Type":"ContainerDied","Data":"63d94457eb8d817715578d463d081fc8712a2d58186899762dbf8cbe7140dd6c"}
May 25 13:11:25.437438 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:11:25.437379    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-s5ssz" event={"ID":"b7c6b019-57cc-43f3-a475-7755083971f3","Type":"ContainerDied","Data":"2f754dd9fd2c09b3312db3fa1076a19460af44bb2735e431d1b187ed73991411"}
May 25 13:11:25.437438 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:11:25.437395    2555 scope.go:117] "RemoveContainer" containerID="d70baa681f62370ac182ea8f0c979948970c973647fdb528ff0b78d9d7c0c429"
May 25 13:11:25.445890 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:11:25.445874    2555 scope.go:117] "RemoveContainer" containerID="63d94457eb8d817715578d463d081fc8712a2d58186899762dbf8cbe7140dd6c"
May 25 13:11:25.453056 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:11:25.453026    2555 scope.go:117] "RemoveContainer" containerID="043436619e2c1785bb0ea5bbb16efe0e6b813891df247d788715c2ff24ecc941"
May 25 13:11:25.460771 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:11:25.460584    2555 scope.go:117] "RemoveContainer" containerID="d70baa681f62370ac182ea8f0c979948970c973647fdb528ff0b78d9d7c0c429"
May 25 13:11:25.462555 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:11:25.462521    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d70baa681f62370ac182ea8f0c979948970c973647fdb528ff0b78d9d7c0c429\": container with ID starting with d70baa681f62370ac182ea8f0c979948970c973647fdb528ff0b78d9d7c0c429 not found: ID does not exist" containerID="d70baa681f62370ac182ea8f0c979948970c973647fdb528ff0b78d9d7c0c429"
May 25 13:11:25.462670 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:11:25.462564    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d70baa681f62370ac182ea8f0c979948970c973647fdb528ff0b78d9d7c0c429"} err="failed to get container status \"d70baa681f62370ac182ea8f0c979948970c973647fdb528ff0b78d9d7c0c429\": rpc error: code = NotFound desc = could not find container \"d70baa681f62370ac182ea8f0c979948970c973647fdb528ff0b78d9d7c0c429\": container with ID starting with d70baa681f62370ac182ea8f0c979948970c973647fdb528ff0b78d9d7c0c429 not found: ID does not exist"
May 25 13:11:25.462670 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:11:25.462587    2555 scope.go:117] "RemoveContainer" containerID="63d94457eb8d817715578d463d081fc8712a2d58186899762dbf8cbe7140dd6c"
May 25 13:11:25.462866 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:11:25.462850    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63d94457eb8d817715578d463d081fc8712a2d58186899762dbf8cbe7140dd6c\": container with ID starting with 63d94457eb8d817715578d463d081fc8712a2d58186899762dbf8cbe7140dd6c not found: ID does not exist" containerID="63d94457eb8d817715578d463d081fc8712a2d58186899762dbf8cbe7140dd6c"
May 25 13:11:25.462918 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:11:25.462872    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63d94457eb8d817715578d463d081fc8712a2d58186899762dbf8cbe7140dd6c"} err="failed to get container status \"63d94457eb8d817715578d463d081fc8712a2d58186899762dbf8cbe7140dd6c\": rpc error: code = NotFound desc = could not find container \"63d94457eb8d817715578d463d081fc8712a2d58186899762dbf8cbe7140dd6c\": container with ID starting with 63d94457eb8d817715578d463d081fc8712a2d58186899762dbf8cbe7140dd6c not found: ID does not exist"
May 25 13:11:25.462918 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:11:25.462885    2555 scope.go:117] "RemoveContainer" containerID="043436619e2c1785bb0ea5bbb16efe0e6b813891df247d788715c2ff24ecc941"
May 25 13:11:25.463153 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:11:25.463134    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"043436619e2c1785bb0ea5bbb16efe0e6b813891df247d788715c2ff24ecc941\": container with ID starting with 043436619e2c1785bb0ea5bbb16efe0e6b813891df247d788715c2ff24ecc941 not found: ID does not exist" containerID="043436619e2c1785bb0ea5bbb16efe0e6b813891df247d788715c2ff24ecc941"
May 25 13:11:25.463212 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:11:25.463173    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"043436619e2c1785bb0ea5bbb16efe0e6b813891df247d788715c2ff24ecc941"} err="failed to get container status \"043436619e2c1785bb0ea5bbb16efe0e6b813891df247d788715c2ff24ecc941\": rpc error: code = NotFound desc = could not find container \"043436619e2c1785bb0ea5bbb16efe0e6b813891df247d788715c2ff24ecc941\": container with ID starting with 043436619e2c1785bb0ea5bbb16efe0e6b813891df247d788715c2ff24ecc941 not found: ID does not exist"
May 25 13:11:25.543290 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:11:25.543266    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b7c6b019-57cc-43f3-a475-7755083971f3-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"b7c6b019-57cc-43f3-a475-7755083971f3\" (UID: \"b7c6b019-57cc-43f3-a475-7755083971f3\") "
May 25 13:11:25.543392 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:11:25.543337    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b7c6b019-57cc-43f3-a475-7755083971f3-kserve-provision-location\") pod \"b7c6b019-57cc-43f3-a475-7755083971f3\" (UID: \"b7c6b019-57cc-43f3-a475-7755083971f3\") "
May 25 13:11:25.543392 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:11:25.543374    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5c7c6\" (UniqueName: \"kubernetes.io/projected/b7c6b019-57cc-43f3-a475-7755083971f3-kube-api-access-5c7c6\") pod \"b7c6b019-57cc-43f3-a475-7755083971f3\" (UID: \"b7c6b019-57cc-43f3-a475-7755083971f3\") "
May 25 13:11:25.543473 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:11:25.543399    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7c6b019-57cc-43f3-a475-7755083971f3-proxy-tls\") pod \"b7c6b019-57cc-43f3-a475-7755083971f3\" (UID: \"b7c6b019-57cc-43f3-a475-7755083971f3\") "
May 25 13:11:25.543634 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:11:25.543613    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7c6b019-57cc-43f3-a475-7755083971f3-xgboost-v2-mlserver-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "xgboost-v2-mlserver-kube-rbac-proxy-sar-config") pod "b7c6b019-57cc-43f3-a475-7755083971f3" (UID: "b7c6b019-57cc-43f3-a475-7755083971f3"). InnerVolumeSpecName "xgboost-v2-mlserver-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:11:25.543744 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:11:25.543692    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7c6b019-57cc-43f3-a475-7755083971f3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b7c6b019-57cc-43f3-a475-7755083971f3" (UID: "b7c6b019-57cc-43f3-a475-7755083971f3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:11:25.545322 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:11:25.545302    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7c6b019-57cc-43f3-a475-7755083971f3-kube-api-access-5c7c6" (OuterVolumeSpecName: "kube-api-access-5c7c6") pod "b7c6b019-57cc-43f3-a475-7755083971f3" (UID: "b7c6b019-57cc-43f3-a475-7755083971f3"). InnerVolumeSpecName "kube-api-access-5c7c6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:11:25.545402 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:11:25.545381    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7c6b019-57cc-43f3-a475-7755083971f3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b7c6b019-57cc-43f3-a475-7755083971f3" (UID: "b7c6b019-57cc-43f3-a475-7755083971f3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:11:25.644487 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:11:25.644469    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b7c6b019-57cc-43f3-a475-7755083971f3-proxy-tls\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:11:25.644574 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:11:25.644490    2555 reconciler_common.go:299] "Volume detached for volume \"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b7c6b019-57cc-43f3-a475-7755083971f3-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:11:25.644574 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:11:25.644501    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b7c6b019-57cc-43f3-a475-7755083971f3-kserve-provision-location\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:11:25.644574 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:11:25.644512    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5c7c6\" (UniqueName: \"kubernetes.io/projected/b7c6b019-57cc-43f3-a475-7755083971f3-kube-api-access-5c7c6\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:11:25.752933 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:11:25.752852    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-s5ssz"]
May 25 13:11:25.756491 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:11:25.756467    2555 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-s5ssz"]
May 25 13:11:27.599536 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:11:27.599460    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7c6b019-57cc-43f3-a475-7755083971f3" path="/var/lib/kubelet/pods/b7c6b019-57cc-43f3-a475-7755083971f3/volumes"
May 25 13:12:59.896134 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:12:59.896095    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-tn6s7"]
May 25 13:12:59.896589 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:12:59.896483    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114" containerName="storage-initializer"
May 25 13:12:59.896589 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:12:59.896496    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114" containerName="storage-initializer"
May 25 13:12:59.896589 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:12:59.896505    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114" containerName="kserve-container"
May 25 13:12:59.896589 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:12:59.896511    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114" containerName="kserve-container"
May 25 13:12:59.896589 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:12:59.896520    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7c6b019-57cc-43f3-a475-7755083971f3" containerName="storage-initializer"
May 25 13:12:59.896589 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:12:59.896526    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7c6b019-57cc-43f3-a475-7755083971f3" containerName="storage-initializer"
May 25 13:12:59.896589 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:12:59.896535    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7c6b019-57cc-43f3-a475-7755083971f3" containerName="kserve-container"
May 25 13:12:59.896589 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:12:59.896541    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7c6b019-57cc-43f3-a475-7755083971f3" containerName="kserve-container"
May 25 13:12:59.896589 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:12:59.896553    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7c6b019-57cc-43f3-a475-7755083971f3" containerName="kube-rbac-proxy"
May 25 13:12:59.896589 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:12:59.896558    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7c6b019-57cc-43f3-a475-7755083971f3" containerName="kube-rbac-proxy"
May 25 13:12:59.896589 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:12:59.896573    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114" containerName="kube-rbac-proxy"
May 25 13:12:59.896589 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:12:59.896578    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114" containerName="kube-rbac-proxy"
May 25 13:12:59.897028 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:12:59.896631    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114" containerName="kserve-container"
May 25 13:12:59.897028 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:12:59.896656    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="7c20f3fc-7f16-4a7d-afc2-6cf38cbc7114" containerName="kube-rbac-proxy"
May 25 13:12:59.897028 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:12:59.896663    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="b7c6b019-57cc-43f3-a475-7755083971f3" containerName="kserve-container"
May 25 13:12:59.897028 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:12:59.896671    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="b7c6b019-57cc-43f3-a475-7755083971f3" containerName="kube-rbac-proxy"
May 25 13:12:59.899844 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:12:59.899824    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-tn6s7"
May 25 13:12:59.901876 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:12:59.901853    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 25 13:12:59.901994 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:12:59.901914    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 25 13:12:59.901994 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:12:59.901860    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6qbkc\""
May 25 13:12:59.902091 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:12:59.901992    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
May 25 13:12:59.902145 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:12:59.902128    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
May 25 13:12:59.912109 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:12:59.912088    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-tn6s7"]
May 25 13:12:59.999496 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:12:59.999409    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0e3b816-826f-442a-b688-632825398903-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-tn6s7\" (UID: \"a0e3b816-826f-442a-b688-632825398903\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-tn6s7"
May 25 13:12:59.999496 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:12:59.999456    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0e3b816-826f-442a-b688-632825398903-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-tn6s7\" (UID: \"a0e3b816-826f-442a-b688-632825398903\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-tn6s7"
May 25 13:12:59.999496 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:12:59.999489    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0e3b816-826f-442a-b688-632825398903-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-tn6s7\" (UID: \"a0e3b816-826f-442a-b688-632825398903\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-tn6s7"
May 25 13:12:59.999792 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:12:59.999620    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5f6r\" (UniqueName: \"kubernetes.io/projected/a0e3b816-826f-442a-b688-632825398903-kube-api-access-b5f6r\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-tn6s7\" (UID: \"a0e3b816-826f-442a-b688-632825398903\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-tn6s7"
May 25 13:13:00.100985 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:13:00.100947    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-b5f6r\" (UniqueName: \"kubernetes.io/projected/a0e3b816-826f-442a-b688-632825398903-kube-api-access-b5f6r\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-tn6s7\" (UID: \"a0e3b816-826f-442a-b688-632825398903\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-tn6s7"
May 25 13:13:00.101178 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:13:00.100990    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0e3b816-826f-442a-b688-632825398903-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-tn6s7\" (UID: \"a0e3b816-826f-442a-b688-632825398903\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-tn6s7"
May 25 13:13:00.101178 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:13:00.101040    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0e3b816-826f-442a-b688-632825398903-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-tn6s7\" (UID: \"a0e3b816-826f-442a-b688-632825398903\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-tn6s7"
May 25 13:13:00.101178 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:13:00.101086    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0e3b816-826f-442a-b688-632825398903-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-tn6s7\" (UID: \"a0e3b816-826f-442a-b688-632825398903\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-tn6s7"
May 25 13:13:00.101178 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:13:00.101162    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-serving-cert: secret "isvc-xgboost-v2-runtime-predictor-serving-cert" not found
May 25 13:13:00.101404 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:13:00.101251    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0e3b816-826f-442a-b688-632825398903-proxy-tls podName:a0e3b816-826f-442a-b688-632825398903 nodeName:}" failed. No retries permitted until 2026-05-25 13:13:00.601233673 +0000 UTC m=+3593.561284075 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a0e3b816-826f-442a-b688-632825398903-proxy-tls") pod "isvc-xgboost-v2-runtime-predictor-6dc5954dc-tn6s7" (UID: "a0e3b816-826f-442a-b688-632825398903") : secret "isvc-xgboost-v2-runtime-predictor-serving-cert" not found
May 25 13:13:00.101553 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:13:00.101534    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0e3b816-826f-442a-b688-632825398903-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-tn6s7\" (UID: \"a0e3b816-826f-442a-b688-632825398903\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-tn6s7"
May 25 13:13:00.101755 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:13:00.101730    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0e3b816-826f-442a-b688-632825398903-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-tn6s7\" (UID: \"a0e3b816-826f-442a-b688-632825398903\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-tn6s7"
May 25 13:13:00.108882 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:13:00.108862    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5f6r\" (UniqueName: \"kubernetes.io/projected/a0e3b816-826f-442a-b688-632825398903-kube-api-access-b5f6r\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-tn6s7\" (UID: \"a0e3b816-826f-442a-b688-632825398903\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-tn6s7"
May 25 13:13:00.605580 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:13:00.605540    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0e3b816-826f-442a-b688-632825398903-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-tn6s7\" (UID: \"a0e3b816-826f-442a-b688-632825398903\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-tn6s7"
May 25 13:13:00.607930 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:13:00.607906    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0e3b816-826f-442a-b688-632825398903-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-tn6s7\" (UID: \"a0e3b816-826f-442a-b688-632825398903\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-tn6s7"
May 25 13:13:00.810415 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:13:00.810381    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-tn6s7"
May 25 13:13:00.938797 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:13:00.938768    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-tn6s7"]
May 25 13:13:00.941249 ip-10-0-134-255 kubenswrapper[2555]: W0525 13:13:00.941217    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0e3b816_826f_442a_b688_632825398903.slice/crio-40aed1792778d8a4364875e831f1d9856e1eaff83ac45b336bbc81f1c54355d4 WatchSource:0}: Error finding container 40aed1792778d8a4364875e831f1d9856e1eaff83ac45b336bbc81f1c54355d4: Status 404 returned error can't find the container with id 40aed1792778d8a4364875e831f1d9856e1eaff83ac45b336bbc81f1c54355d4
May 25 13:13:01.745948 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:13:01.745911    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-tn6s7" event={"ID":"a0e3b816-826f-442a-b688-632825398903","Type":"ContainerStarted","Data":"90c3476ff5180d7fba06d301954b0ba2d971cab13eb002963f6e3f99993a51be"}
May 25 13:13:01.745948 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:13:01.745949    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-tn6s7" event={"ID":"a0e3b816-826f-442a-b688-632825398903","Type":"ContainerStarted","Data":"40aed1792778d8a4364875e831f1d9856e1eaff83ac45b336bbc81f1c54355d4"}
May 25 13:13:04.756301 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:13:04.756271    2555 generic.go:358] "Generic (PLEG): container finished" podID="a0e3b816-826f-442a-b688-632825398903" containerID="90c3476ff5180d7fba06d301954b0ba2d971cab13eb002963f6e3f99993a51be" exitCode=0
May 25 13:13:04.756626 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:13:04.756338    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-tn6s7" event={"ID":"a0e3b816-826f-442a-b688-632825398903","Type":"ContainerDied","Data":"90c3476ff5180d7fba06d301954b0ba2d971cab13eb002963f6e3f99993a51be"}
May 25 13:13:05.761375 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:13:05.761338    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-tn6s7" event={"ID":"a0e3b816-826f-442a-b688-632825398903","Type":"ContainerStarted","Data":"1ac71ab4ed952a7b18be7c7df06dbeb56f479d8447d8887f92c171da5dfef846"}
May 25 13:13:05.761782 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:13:05.761385    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-tn6s7" event={"ID":"a0e3b816-826f-442a-b688-632825398903","Type":"ContainerStarted","Data":"bf7bf3251c4f0cad2255fb9259f6514c90610c915a4630e15b0baf08e388772a"}
May 25 13:13:05.761782 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:13:05.761687    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-tn6s7"
May 25 13:13:05.779976 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:13:05.779934    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-tn6s7" podStartSLOduration=6.779922301 podStartE2EDuration="6.779922301s" podCreationTimestamp="2026-05-25 13:12:59 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:13:05.778577105 +0000 UTC m=+3598.738627512" watchObservedRunningTime="2026-05-25 13:13:05.779922301 +0000 UTC m=+3598.739972708"
May 25 13:13:06.764142 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:13:06.764116    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-tn6s7"
May 25 13:13:07.760926 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:13:07.760900    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-2tqtm_00e6f79c-8f05-4330-aa16-1ada2ae9e9cc/console-operator/1.log"
May 25 13:13:07.770404 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:13:07.770383    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-2tqtm_00e6f79c-8f05-4330-aa16-1ada2ae9e9cc/console-operator/1.log"
May 25 13:13:12.772516 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:13:12.772480    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-tn6s7"
May 25 13:13:42.779820 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:13:42.779778    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-tn6s7" podUID="a0e3b816-826f-442a-b688-632825398903" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
May 25 13:13:52.775566 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:13:52.775537    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-tn6s7"
May 25 13:13:59.990711 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:13:59.990618    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-tn6s7"]
May 25 13:13:59.991151 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:13:59.990944    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-tn6s7" podUID="a0e3b816-826f-442a-b688-632825398903" containerName="kserve-container" containerID="cri-o://bf7bf3251c4f0cad2255fb9259f6514c90610c915a4630e15b0baf08e388772a" gracePeriod=30
May 25 13:13:59.991151 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:13:59.990997    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-tn6s7" podUID="a0e3b816-826f-442a-b688-632825398903" containerName="kube-rbac-proxy" containerID="cri-o://1ac71ab4ed952a7b18be7c7df06dbeb56f479d8447d8887f92c171da5dfef846" gracePeriod=30
May 25 13:14:00.942897 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:14:00.942863    2555 generic.go:358] "Generic (PLEG): container finished" podID="a0e3b816-826f-442a-b688-632825398903" containerID="1ac71ab4ed952a7b18be7c7df06dbeb56f479d8447d8887f92c171da5dfef846" exitCode=2
May 25 13:14:00.943113 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:14:00.942915    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-tn6s7" event={"ID":"a0e3b816-826f-442a-b688-632825398903","Type":"ContainerDied","Data":"1ac71ab4ed952a7b18be7c7df06dbeb56f479d8447d8887f92c171da5dfef846"}
May 25 13:14:02.767279 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:14:02.767239    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-tn6s7" podUID="a0e3b816-826f-442a-b688-632825398903" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.61:8643/healthz\": dial tcp 10.134.0.61:8643: connect: connection refused"
May 25 13:14:07.142225 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:14:07.142202    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-tn6s7"
May 25 13:14:07.275248 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:14:07.275176    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0e3b816-826f-442a-b688-632825398903-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\") pod \"a0e3b816-826f-442a-b688-632825398903\" (UID: \"a0e3b816-826f-442a-b688-632825398903\") "
May 25 13:14:07.275248 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:14:07.275216    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0e3b816-826f-442a-b688-632825398903-proxy-tls\") pod \"a0e3b816-826f-442a-b688-632825398903\" (UID: \"a0e3b816-826f-442a-b688-632825398903\") "
May 25 13:14:07.275473 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:14:07.275252    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0e3b816-826f-442a-b688-632825398903-kserve-provision-location\") pod \"a0e3b816-826f-442a-b688-632825398903\" (UID: \"a0e3b816-826f-442a-b688-632825398903\") "
May 25 13:14:07.275473 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:14:07.275367    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b5f6r\" (UniqueName: \"kubernetes.io/projected/a0e3b816-826f-442a-b688-632825398903-kube-api-access-b5f6r\") pod \"a0e3b816-826f-442a-b688-632825398903\" (UID: \"a0e3b816-826f-442a-b688-632825398903\") "
May 25 13:14:07.275573 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:14:07.275556    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0e3b816-826f-442a-b688-632825398903-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a0e3b816-826f-442a-b688-632825398903" (UID: "a0e3b816-826f-442a-b688-632825398903"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:14:07.275615 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:14:07.275575    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0e3b816-826f-442a-b688-632825398903-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config") pod "a0e3b816-826f-442a-b688-632825398903" (UID: "a0e3b816-826f-442a-b688-632825398903"). InnerVolumeSpecName "isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:14:07.275738 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:14:07.275722    2555 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0e3b816-826f-442a-b688-632825398903-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:14:07.275779 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:14:07.275741    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0e3b816-826f-442a-b688-632825398903-kserve-provision-location\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:14:07.277352 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:14:07.277327    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0e3b816-826f-442a-b688-632825398903-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a0e3b816-826f-442a-b688-632825398903" (UID: "a0e3b816-826f-442a-b688-632825398903"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:14:07.277462 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:14:07.277362    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0e3b816-826f-442a-b688-632825398903-kube-api-access-b5f6r" (OuterVolumeSpecName: "kube-api-access-b5f6r") pod "a0e3b816-826f-442a-b688-632825398903" (UID: "a0e3b816-826f-442a-b688-632825398903"). InnerVolumeSpecName "kube-api-access-b5f6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:14:07.377096 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:14:07.377068    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0e3b816-826f-442a-b688-632825398903-proxy-tls\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:14:07.377096 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:14:07.377092    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5f6r\" (UniqueName: \"kubernetes.io/projected/a0e3b816-826f-442a-b688-632825398903-kube-api-access-b5f6r\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:14:07.970255 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:14:07.970219    2555 generic.go:358] "Generic (PLEG): container finished" podID="a0e3b816-826f-442a-b688-632825398903" containerID="bf7bf3251c4f0cad2255fb9259f6514c90610c915a4630e15b0baf08e388772a" exitCode=0
May 25 13:14:07.970428 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:14:07.970301    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-tn6s7"
May 25 13:14:07.970428 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:14:07.970303    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-tn6s7" event={"ID":"a0e3b816-826f-442a-b688-632825398903","Type":"ContainerDied","Data":"bf7bf3251c4f0cad2255fb9259f6514c90610c915a4630e15b0baf08e388772a"}
May 25 13:14:07.970428 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:14:07.970400    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-tn6s7" event={"ID":"a0e3b816-826f-442a-b688-632825398903","Type":"ContainerDied","Data":"40aed1792778d8a4364875e831f1d9856e1eaff83ac45b336bbc81f1c54355d4"}
May 25 13:14:07.970428 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:14:07.970426    2555 scope.go:117] "RemoveContainer" containerID="1ac71ab4ed952a7b18be7c7df06dbeb56f479d8447d8887f92c171da5dfef846"
May 25 13:14:07.979155 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:14:07.979137    2555 scope.go:117] "RemoveContainer" containerID="bf7bf3251c4f0cad2255fb9259f6514c90610c915a4630e15b0baf08e388772a"
May 25 13:14:07.986550 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:14:07.986528    2555 scope.go:117] "RemoveContainer" containerID="90c3476ff5180d7fba06d301954b0ba2d971cab13eb002963f6e3f99993a51be"
May 25 13:14:07.986636 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:14:07.986553    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-tn6s7"]
May 25 13:14:07.989746 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:14:07.989723    2555 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-tn6s7"]
May 25 13:14:07.993451 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:14:07.993431    2555 scope.go:117] "RemoveContainer" containerID="1ac71ab4ed952a7b18be7c7df06dbeb56f479d8447d8887f92c171da5dfef846"
May 25 13:14:07.993774 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:14:07.993754    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ac71ab4ed952a7b18be7c7df06dbeb56f479d8447d8887f92c171da5dfef846\": container with ID starting with 1ac71ab4ed952a7b18be7c7df06dbeb56f479d8447d8887f92c171da5dfef846 not found: ID does not exist" containerID="1ac71ab4ed952a7b18be7c7df06dbeb56f479d8447d8887f92c171da5dfef846"
May 25 13:14:07.993865 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:14:07.993783    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ac71ab4ed952a7b18be7c7df06dbeb56f479d8447d8887f92c171da5dfef846"} err="failed to get container status \"1ac71ab4ed952a7b18be7c7df06dbeb56f479d8447d8887f92c171da5dfef846\": rpc error: code = NotFound desc = could not find container \"1ac71ab4ed952a7b18be7c7df06dbeb56f479d8447d8887f92c171da5dfef846\": container with ID starting with 1ac71ab4ed952a7b18be7c7df06dbeb56f479d8447d8887f92c171da5dfef846 not found: ID does not exist"
May 25 13:14:07.993865 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:14:07.993809    2555 scope.go:117] "RemoveContainer" containerID="bf7bf3251c4f0cad2255fb9259f6514c90610c915a4630e15b0baf08e388772a"
May 25 13:14:07.994068 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:14:07.994051    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf7bf3251c4f0cad2255fb9259f6514c90610c915a4630e15b0baf08e388772a\": container with ID starting with bf7bf3251c4f0cad2255fb9259f6514c90610c915a4630e15b0baf08e388772a not found: ID does not exist" containerID="bf7bf3251c4f0cad2255fb9259f6514c90610c915a4630e15b0baf08e388772a"
May 25 13:14:07.994108 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:14:07.994076    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf7bf3251c4f0cad2255fb9259f6514c90610c915a4630e15b0baf08e388772a"} err="failed to get container status \"bf7bf3251c4f0cad2255fb9259f6514c90610c915a4630e15b0baf08e388772a\": rpc error: code = NotFound desc = could not find container \"bf7bf3251c4f0cad2255fb9259f6514c90610c915a4630e15b0baf08e388772a\": container with ID starting with bf7bf3251c4f0cad2255fb9259f6514c90610c915a4630e15b0baf08e388772a not found: ID does not exist"
May 25 13:14:07.994108 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:14:07.994093    2555 scope.go:117] "RemoveContainer" containerID="90c3476ff5180d7fba06d301954b0ba2d971cab13eb002963f6e3f99993a51be"
May 25 13:14:07.994301 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:14:07.994283    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90c3476ff5180d7fba06d301954b0ba2d971cab13eb002963f6e3f99993a51be\": container with ID starting with 90c3476ff5180d7fba06d301954b0ba2d971cab13eb002963f6e3f99993a51be not found: ID does not exist" containerID="90c3476ff5180d7fba06d301954b0ba2d971cab13eb002963f6e3f99993a51be"
May 25 13:14:07.994354 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:14:07.994311    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90c3476ff5180d7fba06d301954b0ba2d971cab13eb002963f6e3f99993a51be"} err="failed to get container status \"90c3476ff5180d7fba06d301954b0ba2d971cab13eb002963f6e3f99993a51be\": rpc error: code = NotFound desc = could not find container \"90c3476ff5180d7fba06d301954b0ba2d971cab13eb002963f6e3f99993a51be\": container with ID starting with 90c3476ff5180d7fba06d301954b0ba2d971cab13eb002963f6e3f99993a51be not found: ID does not exist"
May 25 13:14:09.598284 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:14:09.598253    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0e3b816-826f-442a-b688-632825398903" path="/var/lib/kubelet/pods/a0e3b816-826f-442a-b688-632825398903/volumes"
May 25 13:15:20.476664 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:15:20.476605    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-n7zjh"]
May 25 13:15:20.477077 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:15:20.476946    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0e3b816-826f-442a-b688-632825398903" containerName="storage-initializer"
May 25 13:15:20.477077 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:15:20.476967    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0e3b816-826f-442a-b688-632825398903" containerName="storage-initializer"
May 25 13:15:20.477077 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:15:20.476975    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0e3b816-826f-442a-b688-632825398903" containerName="kube-rbac-proxy"
May 25 13:15:20.477077 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:15:20.476980    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0e3b816-826f-442a-b688-632825398903" containerName="kube-rbac-proxy"
May 25 13:15:20.477077 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:15:20.476991    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0e3b816-826f-442a-b688-632825398903" containerName="kserve-container"
May 25 13:15:20.477077 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:15:20.476997    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0e3b816-826f-442a-b688-632825398903" containerName="kserve-container"
May 25 13:15:20.477077 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:15:20.477062    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="a0e3b816-826f-442a-b688-632825398903" containerName="kube-rbac-proxy"
May 25 13:15:20.477077 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:15:20.477070    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="a0e3b816-826f-442a-b688-632825398903" containerName="kserve-container"
May 25 13:15:20.480424 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:15:20.480405    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-n7zjh"
May 25 13:15:20.482555 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:15:20.482537    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
May 25 13:15:20.482808 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:15:20.482781    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 25 13:15:20.482808 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:15:20.482782    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-predictor-serving-cert\""
May 25 13:15:20.482970 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:15:20.482845    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
May 25 13:15:20.483296 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:15:20.483278    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6qbkc\""
May 25 13:15:20.483379 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:15:20.483303    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 25 13:15:20.490744 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:15:20.490725    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-n7zjh"]
May 25 13:15:20.551626 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:15:20.551597    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bbff0d45-7021-4fb1-aea1-1ec85cfc54d4-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-89bbdf6fb-n7zjh\" (UID: \"bbff0d45-7021-4fb1-aea1-1ec85cfc54d4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-n7zjh"
May 25 13:15:20.551778 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:15:20.551722    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bbff0d45-7021-4fb1-aea1-1ec85cfc54d4-proxy-tls\") pod \"isvc-sklearn-s3-predictor-89bbdf6fb-n7zjh\" (UID: \"bbff0d45-7021-4fb1-aea1-1ec85cfc54d4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-n7zjh"
May 25 13:15:20.551778 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:15:20.551774    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bbff0d45-7021-4fb1-aea1-1ec85cfc54d4-isvc-sklearn-s3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-89bbdf6fb-n7zjh\" (UID: \"bbff0d45-7021-4fb1-aea1-1ec85cfc54d4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-n7zjh"
May 25 13:15:20.551913 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:15:20.551801    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4bsj\" (UniqueName: \"kubernetes.io/projected/bbff0d45-7021-4fb1-aea1-1ec85cfc54d4-kube-api-access-g4bsj\") pod \"isvc-sklearn-s3-predictor-89bbdf6fb-n7zjh\" (UID: \"bbff0d45-7021-4fb1-aea1-1ec85cfc54d4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-n7zjh"
May 25 13:15:20.652846 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:15:20.652814    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bbff0d45-7021-4fb1-aea1-1ec85cfc54d4-proxy-tls\") pod \"isvc-sklearn-s3-predictor-89bbdf6fb-n7zjh\" (UID: \"bbff0d45-7021-4fb1-aea1-1ec85cfc54d4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-n7zjh"
May 25 13:15:20.652990 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:15:20.652861    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bbff0d45-7021-4fb1-aea1-1ec85cfc54d4-isvc-sklearn-s3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-89bbdf6fb-n7zjh\" (UID: \"bbff0d45-7021-4fb1-aea1-1ec85cfc54d4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-n7zjh"
May 25 13:15:20.652990 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:15:20.652889    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-g4bsj\" (UniqueName: \"kubernetes.io/projected/bbff0d45-7021-4fb1-aea1-1ec85cfc54d4-kube-api-access-g4bsj\") pod \"isvc-sklearn-s3-predictor-89bbdf6fb-n7zjh\" (UID: \"bbff0d45-7021-4fb1-aea1-1ec85cfc54d4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-n7zjh"
May 25 13:15:20.652990 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:15:20.652937    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bbff0d45-7021-4fb1-aea1-1ec85cfc54d4-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-89bbdf6fb-n7zjh\" (UID: \"bbff0d45-7021-4fb1-aea1-1ec85cfc54d4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-n7zjh"
May 25 13:15:20.653349 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:15:20.653322    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bbff0d45-7021-4fb1-aea1-1ec85cfc54d4-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-89bbdf6fb-n7zjh\" (UID: \"bbff0d45-7021-4fb1-aea1-1ec85cfc54d4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-n7zjh"
May 25 13:15:20.653443 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:15:20.653424    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bbff0d45-7021-4fb1-aea1-1ec85cfc54d4-isvc-sklearn-s3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-89bbdf6fb-n7zjh\" (UID: \"bbff0d45-7021-4fb1-aea1-1ec85cfc54d4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-n7zjh"
May 25 13:15:20.655400 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:15:20.655380    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bbff0d45-7021-4fb1-aea1-1ec85cfc54d4-proxy-tls\") pod \"isvc-sklearn-s3-predictor-89bbdf6fb-n7zjh\" (UID: \"bbff0d45-7021-4fb1-aea1-1ec85cfc54d4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-n7zjh"
May 25 13:15:20.663860 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:15:20.663837    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4bsj\" (UniqueName: \"kubernetes.io/projected/bbff0d45-7021-4fb1-aea1-1ec85cfc54d4-kube-api-access-g4bsj\") pod \"isvc-sklearn-s3-predictor-89bbdf6fb-n7zjh\" (UID: \"bbff0d45-7021-4fb1-aea1-1ec85cfc54d4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-n7zjh"
May 25 13:15:20.791497 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:15:20.791436    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-n7zjh"
May 25 13:15:20.930098 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:15:20.930070    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-n7zjh"]
May 25 13:15:20.931528 ip-10-0-134-255 kubenswrapper[2555]: W0525 13:15:20.931494    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbff0d45_7021_4fb1_aea1_1ec85cfc54d4.slice/crio-eaa73a5f4d7376a5a2e34edca01a003c12babc793276aba49868e4cc755a9ce5 WatchSource:0}: Error finding container eaa73a5f4d7376a5a2e34edca01a003c12babc793276aba49868e4cc755a9ce5: Status 404 returned error can't find the container with id eaa73a5f4d7376a5a2e34edca01a003c12babc793276aba49868e4cc755a9ce5
May 25 13:15:20.933792 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:15:20.933775    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 13:15:21.223837 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:15:21.223799    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-n7zjh" event={"ID":"bbff0d45-7021-4fb1-aea1-1ec85cfc54d4","Type":"ContainerStarted","Data":"3bced5d1dae94eaa3ae710e8eadd914070e2c3bb1dc948d14d2c994f5883e982"}
May 25 13:15:21.223837 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:15:21.223836    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-n7zjh" event={"ID":"bbff0d45-7021-4fb1-aea1-1ec85cfc54d4","Type":"ContainerStarted","Data":"eaa73a5f4d7376a5a2e34edca01a003c12babc793276aba49868e4cc755a9ce5"}
May 25 13:15:22.228428 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:15:22.228391    2555 generic.go:358] "Generic (PLEG): container finished" podID="bbff0d45-7021-4fb1-aea1-1ec85cfc54d4" containerID="3bced5d1dae94eaa3ae710e8eadd914070e2c3bb1dc948d14d2c994f5883e982" exitCode=0
May 25 13:15:22.228851 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:15:22.228452    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-n7zjh" event={"ID":"bbff0d45-7021-4fb1-aea1-1ec85cfc54d4","Type":"ContainerDied","Data":"3bced5d1dae94eaa3ae710e8eadd914070e2c3bb1dc948d14d2c994f5883e982"}
May 25 13:15:23.233373 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:15:23.233340    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-n7zjh" event={"ID":"bbff0d45-7021-4fb1-aea1-1ec85cfc54d4","Type":"ContainerStarted","Data":"39fe93378e1e29679bad1c267455f22f92eb8501bad8a723a4bc18948042570d"}
May 25 13:15:23.233373 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:15:23.233375    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-n7zjh" event={"ID":"bbff0d45-7021-4fb1-aea1-1ec85cfc54d4","Type":"ContainerStarted","Data":"cc420e1e121e07269e2ef2621ce7275b1931d245442216fae560046fafb0d1df"}
May 25 13:15:23.233825 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:15:23.233525    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-n7zjh"
May 25 13:15:23.258529 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:15:23.258485    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-n7zjh" podStartSLOduration=3.258473465 podStartE2EDuration="3.258473465s" podCreationTimestamp="2026-05-25 13:15:20 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:15:23.25780822 +0000 UTC m=+3736.217858627" watchObservedRunningTime="2026-05-25 13:15:23.258473465 +0000 UTC m=+3736.218523870"
May 25 13:15:24.236451 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:15:24.236416    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-n7zjh"
May 25 13:15:24.237667 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:15:24.237618    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-n7zjh" podUID="bbff0d45-7021-4fb1-aea1-1ec85cfc54d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
May 25 13:15:25.240086 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:15:25.240042    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-n7zjh" podUID="bbff0d45-7021-4fb1-aea1-1ec85cfc54d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
May 25 13:15:30.244072 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:15:30.244041    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-n7zjh"
May 25 13:15:30.244701 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:15:30.244673    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-n7zjh" podUID="bbff0d45-7021-4fb1-aea1-1ec85cfc54d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
May 25 13:15:40.245126 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:15:40.245083    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-n7zjh" podUID="bbff0d45-7021-4fb1-aea1-1ec85cfc54d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
May 25 13:15:50.245605 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:15:50.245560    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-n7zjh" podUID="bbff0d45-7021-4fb1-aea1-1ec85cfc54d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
May 25 13:16:00.245480 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:00.245440    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-n7zjh" podUID="bbff0d45-7021-4fb1-aea1-1ec85cfc54d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
May 25 13:16:10.245345 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:10.245299    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-n7zjh" podUID="bbff0d45-7021-4fb1-aea1-1ec85cfc54d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
May 25 13:16:20.244993 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:20.244947    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-n7zjh" podUID="bbff0d45-7021-4fb1-aea1-1ec85cfc54d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
May 25 13:16:30.244793 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:30.244765    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-n7zjh"
May 25 13:16:40.474342 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:40.474306    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-n7zjh"]
May 25 13:16:40.474838 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:40.474735    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-n7zjh" podUID="bbff0d45-7021-4fb1-aea1-1ec85cfc54d4" containerName="kserve-container" containerID="cri-o://cc420e1e121e07269e2ef2621ce7275b1931d245442216fae560046fafb0d1df" gracePeriod=30
May 25 13:16:40.474838 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:40.474802    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-n7zjh" podUID="bbff0d45-7021-4fb1-aea1-1ec85cfc54d4" containerName="kube-rbac-proxy" containerID="cri-o://39fe93378e1e29679bad1c267455f22f92eb8501bad8a723a4bc18948042570d" gracePeriod=30
May 25 13:16:40.603757 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:40.603727    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp"]
May 25 13:16:40.607070 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:40.607050    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp"
May 25 13:16:40.609015 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:40.608994    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
May 25 13:16:40.609160 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:40.609138    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
May 25 13:16:40.609275 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:40.609257    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
May 25 13:16:40.614322 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:40.614244    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp"]
May 25 13:16:40.798381 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:40.798291    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92442eef-081e-474a-b2fa-66defce35f56-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp\" (UID: \"92442eef-081e-474a-b2fa-66defce35f56\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp"
May 25 13:16:40.798381 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:40.798341    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvjtx\" (UniqueName: \"kubernetes.io/projected/92442eef-081e-474a-b2fa-66defce35f56-kube-api-access-gvjtx\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp\" (UID: \"92442eef-081e-474a-b2fa-66defce35f56\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp"
May 25 13:16:40.798616 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:40.798414    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92442eef-081e-474a-b2fa-66defce35f56-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp\" (UID: \"92442eef-081e-474a-b2fa-66defce35f56\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp"
May 25 13:16:40.798616 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:40.798533    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92442eef-081e-474a-b2fa-66defce35f56-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp\" (UID: \"92442eef-081e-474a-b2fa-66defce35f56\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp"
May 25 13:16:40.798616 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:40.798568    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92442eef-081e-474a-b2fa-66defce35f56-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp\" (UID: \"92442eef-081e-474a-b2fa-66defce35f56\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp"
May 25 13:16:40.899841 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:40.899804    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92442eef-081e-474a-b2fa-66defce35f56-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp\" (UID: \"92442eef-081e-474a-b2fa-66defce35f56\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp"
May 25 13:16:40.899841 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:40.899841    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92442eef-081e-474a-b2fa-66defce35f56-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp\" (UID: \"92442eef-081e-474a-b2fa-66defce35f56\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp"
May 25 13:16:40.900079 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:40.899875    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92442eef-081e-474a-b2fa-66defce35f56-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp\" (UID: \"92442eef-081e-474a-b2fa-66defce35f56\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp"
May 25 13:16:40.900079 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:40.899912    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gvjtx\" (UniqueName: \"kubernetes.io/projected/92442eef-081e-474a-b2fa-66defce35f56-kube-api-access-gvjtx\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp\" (UID: \"92442eef-081e-474a-b2fa-66defce35f56\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp"
May 25 13:16:40.900079 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:40.899943    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92442eef-081e-474a-b2fa-66defce35f56-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp\" (UID: \"92442eef-081e-474a-b2fa-66defce35f56\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp"
May 25 13:16:40.900303 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:40.900278    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92442eef-081e-474a-b2fa-66defce35f56-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp\" (UID: \"92442eef-081e-474a-b2fa-66defce35f56\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp"
May 25 13:16:40.900558 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:40.900540    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92442eef-081e-474a-b2fa-66defce35f56-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp\" (UID: \"92442eef-081e-474a-b2fa-66defce35f56\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp"
May 25 13:16:40.900718 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:40.900699    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92442eef-081e-474a-b2fa-66defce35f56-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp\" (UID: \"92442eef-081e-474a-b2fa-66defce35f56\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp"
May 25 13:16:40.902439 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:40.902414    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92442eef-081e-474a-b2fa-66defce35f56-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp\" (UID: \"92442eef-081e-474a-b2fa-66defce35f56\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp"
May 25 13:16:40.907598 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:40.907575    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvjtx\" (UniqueName: \"kubernetes.io/projected/92442eef-081e-474a-b2fa-66defce35f56-kube-api-access-gvjtx\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp\" (UID: \"92442eef-081e-474a-b2fa-66defce35f56\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp"
May 25 13:16:40.918352 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:40.918328    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp"
May 25 13:16:41.047265 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:41.047237    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp"]
May 25 13:16:41.049498 ip-10-0-134-255 kubenswrapper[2555]: W0525 13:16:41.049435    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92442eef_081e_474a_b2fa_66defce35f56.slice/crio-bab9562f23a9b82cf79f3253e238c99dc9111e5649201b943f21c33c7e5ec64b WatchSource:0}: Error finding container bab9562f23a9b82cf79f3253e238c99dc9111e5649201b943f21c33c7e5ec64b: Status 404 returned error can't find the container with id bab9562f23a9b82cf79f3253e238c99dc9111e5649201b943f21c33c7e5ec64b
May 25 13:16:41.503417 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:41.503382    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp" event={"ID":"92442eef-081e-474a-b2fa-66defce35f56","Type":"ContainerStarted","Data":"aca80494c2d8c5ddeb6dfc351249ddd0b54ae4c447bdcd4b545e8082107dbe13"}
May 25 13:16:41.503417 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:41.503422    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp" event={"ID":"92442eef-081e-474a-b2fa-66defce35f56","Type":"ContainerStarted","Data":"bab9562f23a9b82cf79f3253e238c99dc9111e5649201b943f21c33c7e5ec64b"}
May 25 13:16:41.505551 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:41.505521    2555 generic.go:358] "Generic (PLEG): container finished" podID="bbff0d45-7021-4fb1-aea1-1ec85cfc54d4" containerID="39fe93378e1e29679bad1c267455f22f92eb8501bad8a723a4bc18948042570d" exitCode=2
May 25 13:16:41.505692 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:41.505606    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-n7zjh" event={"ID":"bbff0d45-7021-4fb1-aea1-1ec85cfc54d4","Type":"ContainerDied","Data":"39fe93378e1e29679bad1c267455f22f92eb8501bad8a723a4bc18948042570d"}
May 25 13:16:42.510795 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:42.510761    2555 generic.go:358] "Generic (PLEG): container finished" podID="92442eef-081e-474a-b2fa-66defce35f56" containerID="aca80494c2d8c5ddeb6dfc351249ddd0b54ae4c447bdcd4b545e8082107dbe13" exitCode=0
May 25 13:16:42.511205 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:42.510820    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp" event={"ID":"92442eef-081e-474a-b2fa-66defce35f56","Type":"ContainerDied","Data":"aca80494c2d8c5ddeb6dfc351249ddd0b54ae4c447bdcd4b545e8082107dbe13"}
May 25 13:16:43.516171 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:43.516133    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp" event={"ID":"92442eef-081e-474a-b2fa-66defce35f56","Type":"ContainerStarted","Data":"e5ac71f662845f6b090f5ac70ebcb7ec0efee492e9614c84af86420da1cec127"}
May 25 13:16:43.516171 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:43.516173    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp" event={"ID":"92442eef-081e-474a-b2fa-66defce35f56","Type":"ContainerStarted","Data":"78fcbfb9c9c61c66a7704def13f60a0c6be29a128247f1360e2e2c06ba25d09b"}
May 25 13:16:43.516581 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:43.516225    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp"
May 25 13:16:43.534395 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:43.534345    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp" podStartSLOduration=3.534333752 podStartE2EDuration="3.534333752s" podCreationTimestamp="2026-05-25 13:16:40 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:16:43.532816919 +0000 UTC m=+3816.492867325" watchObservedRunningTime="2026-05-25 13:16:43.534333752 +0000 UTC m=+3816.494384158"
May 25 13:16:44.522314 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:44.522278    2555 generic.go:358] "Generic (PLEG): container finished" podID="bbff0d45-7021-4fb1-aea1-1ec85cfc54d4" containerID="cc420e1e121e07269e2ef2621ce7275b1931d245442216fae560046fafb0d1df" exitCode=0
May 25 13:16:44.522797 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:44.522351    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-n7zjh" event={"ID":"bbff0d45-7021-4fb1-aea1-1ec85cfc54d4","Type":"ContainerDied","Data":"cc420e1e121e07269e2ef2621ce7275b1931d245442216fae560046fafb0d1df"}
May 25 13:16:44.522797 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:44.522680    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp"
May 25 13:16:44.523637 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:44.523606    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp" podUID="92442eef-081e-474a-b2fa-66defce35f56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
May 25 13:16:44.622881 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:44.622855    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-n7zjh"
May 25 13:16:44.626658 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:44.626622    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bbff0d45-7021-4fb1-aea1-1ec85cfc54d4-proxy-tls\") pod \"bbff0d45-7021-4fb1-aea1-1ec85cfc54d4\" (UID: \"bbff0d45-7021-4fb1-aea1-1ec85cfc54d4\") "
May 25 13:16:44.626777 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:44.626744    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4bsj\" (UniqueName: \"kubernetes.io/projected/bbff0d45-7021-4fb1-aea1-1ec85cfc54d4-kube-api-access-g4bsj\") pod \"bbff0d45-7021-4fb1-aea1-1ec85cfc54d4\" (UID: \"bbff0d45-7021-4fb1-aea1-1ec85cfc54d4\") "
May 25 13:16:44.626828 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:44.626773    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bbff0d45-7021-4fb1-aea1-1ec85cfc54d4-isvc-sklearn-s3-kube-rbac-proxy-sar-config\") pod \"bbff0d45-7021-4fb1-aea1-1ec85cfc54d4\" (UID: \"bbff0d45-7021-4fb1-aea1-1ec85cfc54d4\") "
May 25 13:16:44.626828 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:44.626804    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bbff0d45-7021-4fb1-aea1-1ec85cfc54d4-kserve-provision-location\") pod \"bbff0d45-7021-4fb1-aea1-1ec85cfc54d4\" (UID: \"bbff0d45-7021-4fb1-aea1-1ec85cfc54d4\") "
May 25 13:16:44.627155 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:44.627128    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbff0d45-7021-4fb1-aea1-1ec85cfc54d4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bbff0d45-7021-4fb1-aea1-1ec85cfc54d4" (UID: "bbff0d45-7021-4fb1-aea1-1ec85cfc54d4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:16:44.627155 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:44.627142    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbff0d45-7021-4fb1-aea1-1ec85cfc54d4-isvc-sklearn-s3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-kube-rbac-proxy-sar-config") pod "bbff0d45-7021-4fb1-aea1-1ec85cfc54d4" (UID: "bbff0d45-7021-4fb1-aea1-1ec85cfc54d4"). InnerVolumeSpecName "isvc-sklearn-s3-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:16:44.628621 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:44.628600    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbff0d45-7021-4fb1-aea1-1ec85cfc54d4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bbff0d45-7021-4fb1-aea1-1ec85cfc54d4" (UID: "bbff0d45-7021-4fb1-aea1-1ec85cfc54d4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:16:44.628728 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:44.628676    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbff0d45-7021-4fb1-aea1-1ec85cfc54d4-kube-api-access-g4bsj" (OuterVolumeSpecName: "kube-api-access-g4bsj") pod "bbff0d45-7021-4fb1-aea1-1ec85cfc54d4" (UID: "bbff0d45-7021-4fb1-aea1-1ec85cfc54d4"). InnerVolumeSpecName "kube-api-access-g4bsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:16:44.727319 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:44.727286    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bbff0d45-7021-4fb1-aea1-1ec85cfc54d4-proxy-tls\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:16:44.727319 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:44.727314    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4bsj\" (UniqueName: \"kubernetes.io/projected/bbff0d45-7021-4fb1-aea1-1ec85cfc54d4-kube-api-access-g4bsj\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:16:44.727319 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:44.727324    2555 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bbff0d45-7021-4fb1-aea1-1ec85cfc54d4-isvc-sklearn-s3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:16:44.727566 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:44.727334    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bbff0d45-7021-4fb1-aea1-1ec85cfc54d4-kserve-provision-location\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:16:45.527608 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:45.527576    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-n7zjh"
May 25 13:16:45.528036 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:45.527579    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-n7zjh" event={"ID":"bbff0d45-7021-4fb1-aea1-1ec85cfc54d4","Type":"ContainerDied","Data":"eaa73a5f4d7376a5a2e34edca01a003c12babc793276aba49868e4cc755a9ce5"}
May 25 13:16:45.528036 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:45.527703    2555 scope.go:117] "RemoveContainer" containerID="39fe93378e1e29679bad1c267455f22f92eb8501bad8a723a4bc18948042570d"
May 25 13:16:45.528173 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:45.528144    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp" podUID="92442eef-081e-474a-b2fa-66defce35f56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
May 25 13:16:45.536700 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:45.536679    2555 scope.go:117] "RemoveContainer" containerID="cc420e1e121e07269e2ef2621ce7275b1931d245442216fae560046fafb0d1df"
May 25 13:16:45.543684 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:45.543656    2555 scope.go:117] "RemoveContainer" containerID="3bced5d1dae94eaa3ae710e8eadd914070e2c3bb1dc948d14d2c994f5883e982"
May 25 13:16:45.546426 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:45.546401    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-n7zjh"]
May 25 13:16:45.551854 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:45.551834    2555 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-n7zjh"]
May 25 13:16:45.599170 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:45.599138    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbff0d45-7021-4fb1-aea1-1ec85cfc54d4" path="/var/lib/kubelet/pods/bbff0d45-7021-4fb1-aea1-1ec85cfc54d4/volumes"
May 25 13:16:50.531887 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:50.531860    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp"
May 25 13:16:50.532501 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:16:50.532470    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp" podUID="92442eef-081e-474a-b2fa-66defce35f56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
May 25 13:17:00.532637 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:17:00.532595    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp" podUID="92442eef-081e-474a-b2fa-66defce35f56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
May 25 13:17:10.532616 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:17:10.532567    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp" podUID="92442eef-081e-474a-b2fa-66defce35f56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
May 25 13:17:20.532743 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:17:20.532698    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp" podUID="92442eef-081e-474a-b2fa-66defce35f56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
May 25 13:17:30.533276 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:17:30.533193    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp" podUID="92442eef-081e-474a-b2fa-66defce35f56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
May 25 13:17:40.533424 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:17:40.533382    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp" podUID="92442eef-081e-474a-b2fa-66defce35f56" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
May 25 13:17:50.532816 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:17:50.532785    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp"
May 25 13:18:00.638208 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:00.638173    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp"]
May 25 13:18:00.638680 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:00.638495    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp" podUID="92442eef-081e-474a-b2fa-66defce35f56" containerName="kserve-container" containerID="cri-o://78fcbfb9c9c61c66a7704def13f60a0c6be29a128247f1360e2e2c06ba25d09b" gracePeriod=30
May 25 13:18:00.638680 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:00.638531    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp" podUID="92442eef-081e-474a-b2fa-66defce35f56" containerName="kube-rbac-proxy" containerID="cri-o://e5ac71f662845f6b090f5ac70ebcb7ec0efee492e9614c84af86420da1cec127" gracePeriod=30
May 25 13:18:00.781716 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:00.781681    2555 generic.go:358] "Generic (PLEG): container finished" podID="92442eef-081e-474a-b2fa-66defce35f56" containerID="e5ac71f662845f6b090f5ac70ebcb7ec0efee492e9614c84af86420da1cec127" exitCode=2
May 25 13:18:00.781842 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:00.781739    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp" event={"ID":"92442eef-081e-474a-b2fa-66defce35f56","Type":"ContainerDied","Data":"e5ac71f662845f6b090f5ac70ebcb7ec0efee492e9614c84af86420da1cec127"}
May 25 13:18:01.720061 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:01.720023    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-fbs5w"]
May 25 13:18:01.720437 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:01.720362    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbff0d45-7021-4fb1-aea1-1ec85cfc54d4" containerName="kserve-container"
May 25 13:18:01.720437 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:01.720373    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbff0d45-7021-4fb1-aea1-1ec85cfc54d4" containerName="kserve-container"
May 25 13:18:01.720437 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:01.720391    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbff0d45-7021-4fb1-aea1-1ec85cfc54d4" containerName="storage-initializer"
May 25 13:18:01.720437 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:01.720398    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbff0d45-7021-4fb1-aea1-1ec85cfc54d4" containerName="storage-initializer"
May 25 13:18:01.720437 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:01.720406    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbff0d45-7021-4fb1-aea1-1ec85cfc54d4" containerName="kube-rbac-proxy"
May 25 13:18:01.720437 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:01.720412    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbff0d45-7021-4fb1-aea1-1ec85cfc54d4" containerName="kube-rbac-proxy"
May 25 13:18:01.720632 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:01.720467    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="bbff0d45-7021-4fb1-aea1-1ec85cfc54d4" containerName="kserve-container"
May 25 13:18:01.720632 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:01.720476    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="bbff0d45-7021-4fb1-aea1-1ec85cfc54d4" containerName="kube-rbac-proxy"
May 25 13:18:01.723387 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:01.723369    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-fbs5w"
May 25 13:18:01.725134 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:01.725113    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
May 25 13:18:01.725246 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:01.725115    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
May 25 13:18:01.733868 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:01.733842    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-fbs5w"]
May 25 13:18:01.777127 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:01.777095    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2qlb\" (UniqueName: \"kubernetes.io/projected/a99c9fa5-024b-4147-8c59-e7a13a42b76a-kube-api-access-j2qlb\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-fbs5w\" (UID: \"a99c9fa5-024b-4147-8c59-e7a13a42b76a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-fbs5w"
May 25 13:18:01.777127 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:01.777129    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a99c9fa5-024b-4147-8c59-e7a13a42b76a-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-fbs5w\" (UID: \"a99c9fa5-024b-4147-8c59-e7a13a42b76a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-fbs5w"
May 25 13:18:01.777334 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:01.777177    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a99c9fa5-024b-4147-8c59-e7a13a42b76a-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-fbs5w\" (UID: \"a99c9fa5-024b-4147-8c59-e7a13a42b76a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-fbs5w"
May 25 13:18:01.777334 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:01.777218    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a99c9fa5-024b-4147-8c59-e7a13a42b76a-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-fbs5w\" (UID: \"a99c9fa5-024b-4147-8c59-e7a13a42b76a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-fbs5w"
May 25 13:18:01.877769 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:01.877735    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a99c9fa5-024b-4147-8c59-e7a13a42b76a-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-fbs5w\" (UID: \"a99c9fa5-024b-4147-8c59-e7a13a42b76a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-fbs5w"
May 25 13:18:01.877952 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:01.877803    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-j2qlb\" (UniqueName: \"kubernetes.io/projected/a99c9fa5-024b-4147-8c59-e7a13a42b76a-kube-api-access-j2qlb\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-fbs5w\" (UID: \"a99c9fa5-024b-4147-8c59-e7a13a42b76a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-fbs5w"
May 25 13:18:01.877952 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:01.877824    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a99c9fa5-024b-4147-8c59-e7a13a42b76a-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-fbs5w\" (UID: \"a99c9fa5-024b-4147-8c59-e7a13a42b76a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-fbs5w"
May 25 13:18:01.877952 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:01.877857    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a99c9fa5-024b-4147-8c59-e7a13a42b76a-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-fbs5w\" (UID: \"a99c9fa5-024b-4147-8c59-e7a13a42b76a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-fbs5w"
May 25 13:18:01.877952 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:18:01.877902    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
May 25 13:18:01.878169 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:18:01.877974    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a99c9fa5-024b-4147-8c59-e7a13a42b76a-proxy-tls podName:a99c9fa5-024b-4147-8c59-e7a13a42b76a nodeName:}" failed. No retries permitted until 2026-05-25 13:18:02.377952134 +0000 UTC m=+3895.338002519 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a99c9fa5-024b-4147-8c59-e7a13a42b76a-proxy-tls") pod "isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-fbs5w" (UID: "a99c9fa5-024b-4147-8c59-e7a13a42b76a") : secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
May 25 13:18:01.878267 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:01.878248    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a99c9fa5-024b-4147-8c59-e7a13a42b76a-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-fbs5w\" (UID: \"a99c9fa5-024b-4147-8c59-e7a13a42b76a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-fbs5w"
May 25 13:18:01.878534 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:01.878515    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a99c9fa5-024b-4147-8c59-e7a13a42b76a-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-fbs5w\" (UID: \"a99c9fa5-024b-4147-8c59-e7a13a42b76a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-fbs5w"
May 25 13:18:01.887778 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:01.887754    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2qlb\" (UniqueName: \"kubernetes.io/projected/a99c9fa5-024b-4147-8c59-e7a13a42b76a-kube-api-access-j2qlb\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-fbs5w\" (UID: \"a99c9fa5-024b-4147-8c59-e7a13a42b76a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-fbs5w"
May 25 13:18:02.382630 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:02.382589    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a99c9fa5-024b-4147-8c59-e7a13a42b76a-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-fbs5w\" (UID: \"a99c9fa5-024b-4147-8c59-e7a13a42b76a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-fbs5w"
May 25 13:18:02.385206 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:02.385176    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a99c9fa5-024b-4147-8c59-e7a13a42b76a-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-fbs5w\" (UID: \"a99c9fa5-024b-4147-8c59-e7a13a42b76a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-fbs5w"
May 25 13:18:02.635548 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:02.635450    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-fbs5w"
May 25 13:18:02.761382 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:02.761332    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-fbs5w"]
May 25 13:18:02.763670 ip-10-0-134-255 kubenswrapper[2555]: W0525 13:18:02.763624    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda99c9fa5_024b_4147_8c59_e7a13a42b76a.slice/crio-4ecb3df412213d14f87a6808821271fb6c1a6b2d5e429a3302f900c59a75d13d WatchSource:0}: Error finding container 4ecb3df412213d14f87a6808821271fb6c1a6b2d5e429a3302f900c59a75d13d: Status 404 returned error can't find the container with id 4ecb3df412213d14f87a6808821271fb6c1a6b2d5e429a3302f900c59a75d13d
May 25 13:18:02.790223 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:02.790105    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-fbs5w" event={"ID":"a99c9fa5-024b-4147-8c59-e7a13a42b76a","Type":"ContainerStarted","Data":"4ecb3df412213d14f87a6808821271fb6c1a6b2d5e429a3302f900c59a75d13d"}
May 25 13:18:03.794593 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:03.794552    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-fbs5w" event={"ID":"a99c9fa5-024b-4147-8c59-e7a13a42b76a","Type":"ContainerStarted","Data":"b824130a84616b2f480d890e6741c5ce49e9eb12e273322be19073c133badcac"}
May 25 13:18:04.992839 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:04.992814    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp"
May 25 13:18:05.108413 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:05.108319    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92442eef-081e-474a-b2fa-66defce35f56-cabundle-cert\") pod \"92442eef-081e-474a-b2fa-66defce35f56\" (UID: \"92442eef-081e-474a-b2fa-66defce35f56\") "
May 25 13:18:05.108413 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:05.108368    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92442eef-081e-474a-b2fa-66defce35f56-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\") pod \"92442eef-081e-474a-b2fa-66defce35f56\" (UID: \"92442eef-081e-474a-b2fa-66defce35f56\") "
May 25 13:18:05.108676 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:05.108418    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92442eef-081e-474a-b2fa-66defce35f56-kserve-provision-location\") pod \"92442eef-081e-474a-b2fa-66defce35f56\" (UID: \"92442eef-081e-474a-b2fa-66defce35f56\") "
May 25 13:18:05.108676 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:05.108449    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92442eef-081e-474a-b2fa-66defce35f56-proxy-tls\") pod \"92442eef-081e-474a-b2fa-66defce35f56\" (UID: \"92442eef-081e-474a-b2fa-66defce35f56\") "
May 25 13:18:05.108676 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:05.108469    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvjtx\" (UniqueName: \"kubernetes.io/projected/92442eef-081e-474a-b2fa-66defce35f56-kube-api-access-gvjtx\") pod \"92442eef-081e-474a-b2fa-66defce35f56\" (UID: \"92442eef-081e-474a-b2fa-66defce35f56\") "
May 25 13:18:05.108847 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:05.108823    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92442eef-081e-474a-b2fa-66defce35f56-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config") pod "92442eef-081e-474a-b2fa-66defce35f56" (UID: "92442eef-081e-474a-b2fa-66defce35f56"). InnerVolumeSpecName "isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:18:05.108900 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:05.108873    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92442eef-081e-474a-b2fa-66defce35f56-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "92442eef-081e-474a-b2fa-66defce35f56" (UID: "92442eef-081e-474a-b2fa-66defce35f56"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:18:05.108952 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:05.108901    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92442eef-081e-474a-b2fa-66defce35f56-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "92442eef-081e-474a-b2fa-66defce35f56" (UID: "92442eef-081e-474a-b2fa-66defce35f56"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:18:05.110664 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:05.110628    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92442eef-081e-474a-b2fa-66defce35f56-kube-api-access-gvjtx" (OuterVolumeSpecName: "kube-api-access-gvjtx") pod "92442eef-081e-474a-b2fa-66defce35f56" (UID: "92442eef-081e-474a-b2fa-66defce35f56"). InnerVolumeSpecName "kube-api-access-gvjtx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:18:05.110772 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:05.110664    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92442eef-081e-474a-b2fa-66defce35f56-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "92442eef-081e-474a-b2fa-66defce35f56" (UID: "92442eef-081e-474a-b2fa-66defce35f56"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:18:05.209396 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:05.209357    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92442eef-081e-474a-b2fa-66defce35f56-cabundle-cert\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:18:05.209396 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:05.209393    2555 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92442eef-081e-474a-b2fa-66defce35f56-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:18:05.209396 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:05.209404    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92442eef-081e-474a-b2fa-66defce35f56-kserve-provision-location\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:18:05.209666 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:05.209415    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92442eef-081e-474a-b2fa-66defce35f56-proxy-tls\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:18:05.209666 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:05.209424    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvjtx\" (UniqueName: \"kubernetes.io/projected/92442eef-081e-474a-b2fa-66defce35f56-kube-api-access-gvjtx\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:18:05.804124 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:05.804087    2555 generic.go:358] "Generic (PLEG): container finished" podID="92442eef-081e-474a-b2fa-66defce35f56" containerID="78fcbfb9c9c61c66a7704def13f60a0c6be29a128247f1360e2e2c06ba25d09b" exitCode=0
May 25 13:18:05.804303 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:05.804145    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp" event={"ID":"92442eef-081e-474a-b2fa-66defce35f56","Type":"ContainerDied","Data":"78fcbfb9c9c61c66a7704def13f60a0c6be29a128247f1360e2e2c06ba25d09b"}
May 25 13:18:05.804303 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:05.804170    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp" event={"ID":"92442eef-081e-474a-b2fa-66defce35f56","Type":"ContainerDied","Data":"bab9562f23a9b82cf79f3253e238c99dc9111e5649201b943f21c33c7e5ec64b"}
May 25 13:18:05.804303 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:05.804178    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp"
May 25 13:18:05.804303 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:05.804187    2555 scope.go:117] "RemoveContainer" containerID="e5ac71f662845f6b090f5ac70ebcb7ec0efee492e9614c84af86420da1cec127"
May 25 13:18:05.812398 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:05.812367    2555 scope.go:117] "RemoveContainer" containerID="78fcbfb9c9c61c66a7704def13f60a0c6be29a128247f1360e2e2c06ba25d09b"
May 25 13:18:05.818253 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:05.818230    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp"]
May 25 13:18:05.819844 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:05.819786    2555 scope.go:117] "RemoveContainer" containerID="aca80494c2d8c5ddeb6dfc351249ddd0b54ae4c447bdcd4b545e8082107dbe13"
May 25 13:18:05.821531 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:05.821510    2555 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-99hdp"]
May 25 13:18:05.826826 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:05.826811    2555 scope.go:117] "RemoveContainer" containerID="e5ac71f662845f6b090f5ac70ebcb7ec0efee492e9614c84af86420da1cec127"
May 25 13:18:05.827088 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:18:05.827068    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5ac71f662845f6b090f5ac70ebcb7ec0efee492e9614c84af86420da1cec127\": container with ID starting with e5ac71f662845f6b090f5ac70ebcb7ec0efee492e9614c84af86420da1cec127 not found: ID does not exist" containerID="e5ac71f662845f6b090f5ac70ebcb7ec0efee492e9614c84af86420da1cec127"
May 25 13:18:05.827145 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:05.827096    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5ac71f662845f6b090f5ac70ebcb7ec0efee492e9614c84af86420da1cec127"} err="failed to get container status \"e5ac71f662845f6b090f5ac70ebcb7ec0efee492e9614c84af86420da1cec127\": rpc error: code = NotFound desc = could not find container \"e5ac71f662845f6b090f5ac70ebcb7ec0efee492e9614c84af86420da1cec127\": container with ID starting with e5ac71f662845f6b090f5ac70ebcb7ec0efee492e9614c84af86420da1cec127 not found: ID does not exist"
May 25 13:18:05.827145 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:05.827114    2555 scope.go:117] "RemoveContainer" containerID="78fcbfb9c9c61c66a7704def13f60a0c6be29a128247f1360e2e2c06ba25d09b"
May 25 13:18:05.827335 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:18:05.827310    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78fcbfb9c9c61c66a7704def13f60a0c6be29a128247f1360e2e2c06ba25d09b\": container with ID starting with 78fcbfb9c9c61c66a7704def13f60a0c6be29a128247f1360e2e2c06ba25d09b not found: ID does not exist" containerID="78fcbfb9c9c61c66a7704def13f60a0c6be29a128247f1360e2e2c06ba25d09b"
May 25 13:18:05.827404 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:05.827343    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78fcbfb9c9c61c66a7704def13f60a0c6be29a128247f1360e2e2c06ba25d09b"} err="failed to get container status \"78fcbfb9c9c61c66a7704def13f60a0c6be29a128247f1360e2e2c06ba25d09b\": rpc error: code = NotFound desc = could not find container \"78fcbfb9c9c61c66a7704def13f60a0c6be29a128247f1360e2e2c06ba25d09b\": container with ID starting with 78fcbfb9c9c61c66a7704def13f60a0c6be29a128247f1360e2e2c06ba25d09b not found: ID does not exist"
May 25 13:18:05.827404 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:05.827365    2555 scope.go:117] "RemoveContainer" containerID="aca80494c2d8c5ddeb6dfc351249ddd0b54ae4c447bdcd4b545e8082107dbe13"
May 25 13:18:05.827609 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:18:05.827589    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aca80494c2d8c5ddeb6dfc351249ddd0b54ae4c447bdcd4b545e8082107dbe13\": container with ID starting with aca80494c2d8c5ddeb6dfc351249ddd0b54ae4c447bdcd4b545e8082107dbe13 not found: ID does not exist" containerID="aca80494c2d8c5ddeb6dfc351249ddd0b54ae4c447bdcd4b545e8082107dbe13"
May 25 13:18:05.827672 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:05.827616    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aca80494c2d8c5ddeb6dfc351249ddd0b54ae4c447bdcd4b545e8082107dbe13"} err="failed to get container status \"aca80494c2d8c5ddeb6dfc351249ddd0b54ae4c447bdcd4b545e8082107dbe13\": rpc error: code = NotFound desc = could not find container \"aca80494c2d8c5ddeb6dfc351249ddd0b54ae4c447bdcd4b545e8082107dbe13\": container with ID starting with aca80494c2d8c5ddeb6dfc351249ddd0b54ae4c447bdcd4b545e8082107dbe13 not found: ID does not exist"
May 25 13:18:07.599444 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:07.599409    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92442eef-081e-474a-b2fa-66defce35f56" path="/var/lib/kubelet/pods/92442eef-081e-474a-b2fa-66defce35f56/volumes"
May 25 13:18:07.787676 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:07.787635    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-2tqtm_00e6f79c-8f05-4330-aa16-1ada2ae9e9cc/console-operator/1.log"
May 25 13:18:07.798382 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:07.798356    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-2tqtm_00e6f79c-8f05-4330-aa16-1ada2ae9e9cc/console-operator/1.log"
May 25 13:18:08.816189 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:08.816160    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-fbs5w_a99c9fa5-024b-4147-8c59-e7a13a42b76a/storage-initializer/0.log"
May 25 13:18:08.816543 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:08.816201    2555 generic.go:358] "Generic (PLEG): container finished" podID="a99c9fa5-024b-4147-8c59-e7a13a42b76a" containerID="b824130a84616b2f480d890e6741c5ce49e9eb12e273322be19073c133badcac" exitCode=1
May 25 13:18:08.816543 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:08.816265    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-fbs5w" event={"ID":"a99c9fa5-024b-4147-8c59-e7a13a42b76a","Type":"ContainerDied","Data":"b824130a84616b2f480d890e6741c5ce49e9eb12e273322be19073c133badcac"}
May 25 13:18:09.821390 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:09.821361    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-fbs5w_a99c9fa5-024b-4147-8c59-e7a13a42b76a/storage-initializer/0.log"
May 25 13:18:09.821788 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:09.821477    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-fbs5w" event={"ID":"a99c9fa5-024b-4147-8c59-e7a13a42b76a","Type":"ContainerStarted","Data":"ce8ffceeaae98e3f12bcc5aa4e1d95e4117b5d2e8867b7653886238650689aac"}
May 25 13:18:11.711681 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:11.711631    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-fbs5w"]
May 25 13:18:11.712215 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:11.711937    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-fbs5w" podUID="a99c9fa5-024b-4147-8c59-e7a13a42b76a" containerName="storage-initializer" containerID="cri-o://ce8ffceeaae98e3f12bcc5aa4e1d95e4117b5d2e8867b7653886238650689aac" gracePeriod=30
May 25 13:18:12.802193 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:12.802148    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v"]
May 25 13:18:12.802715 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:12.802696    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92442eef-081e-474a-b2fa-66defce35f56" containerName="storage-initializer"
May 25 13:18:12.802803 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:12.802717    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="92442eef-081e-474a-b2fa-66defce35f56" containerName="storage-initializer"
May 25 13:18:12.802803 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:12.802737    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92442eef-081e-474a-b2fa-66defce35f56" containerName="kserve-container"
May 25 13:18:12.802803 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:12.802746    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="92442eef-081e-474a-b2fa-66defce35f56" containerName="kserve-container"
May 25 13:18:12.802803 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:12.802770    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92442eef-081e-474a-b2fa-66defce35f56" containerName="kube-rbac-proxy"
May 25 13:18:12.802803 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:12.802779    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="92442eef-081e-474a-b2fa-66defce35f56" containerName="kube-rbac-proxy"
May 25 13:18:12.803067 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:12.802862    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="92442eef-081e-474a-b2fa-66defce35f56" containerName="kserve-container"
May 25 13:18:12.803067 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:12.802876    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="92442eef-081e-474a-b2fa-66defce35f56" containerName="kube-rbac-proxy"
May 25 13:18:12.805851 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:12.805829    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v"
May 25 13:18:12.807628 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:12.807607    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
May 25 13:18:12.807776 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:12.807665    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
May 25 13:18:12.807853 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:12.807837    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
May 25 13:18:12.814675 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:12.814630    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v"]
May 25 13:18:12.881238 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:12.881207    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pld8t\" (UniqueName: \"kubernetes.io/projected/e1e821a9-67d0-419d-8460-5d52132a3bff-kube-api-access-pld8t\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v\" (UID: \"e1e821a9-67d0-419d-8460-5d52132a3bff\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v"
May 25 13:18:12.881429 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:12.881252    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e1e821a9-67d0-419d-8460-5d52132a3bff-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v\" (UID: \"e1e821a9-67d0-419d-8460-5d52132a3bff\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v"
May 25 13:18:12.881429 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:12.881273    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1e821a9-67d0-419d-8460-5d52132a3bff-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v\" (UID: \"e1e821a9-67d0-419d-8460-5d52132a3bff\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v"
May 25 13:18:12.881429 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:12.881294    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1e821a9-67d0-419d-8460-5d52132a3bff-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v\" (UID: \"e1e821a9-67d0-419d-8460-5d52132a3bff\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v"
May 25 13:18:12.881429 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:12.881371    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1e821a9-67d0-419d-8460-5d52132a3bff-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v\" (UID: \"e1e821a9-67d0-419d-8460-5d52132a3bff\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v"
May 25 13:18:12.982706 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:12.982670    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pld8t\" (UniqueName: \"kubernetes.io/projected/e1e821a9-67d0-419d-8460-5d52132a3bff-kube-api-access-pld8t\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v\" (UID: \"e1e821a9-67d0-419d-8460-5d52132a3bff\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v"
May 25 13:18:12.982901 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:12.982723    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e1e821a9-67d0-419d-8460-5d52132a3bff-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v\" (UID: \"e1e821a9-67d0-419d-8460-5d52132a3bff\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v"
May 25 13:18:12.982901 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:12.982754    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1e821a9-67d0-419d-8460-5d52132a3bff-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v\" (UID: \"e1e821a9-67d0-419d-8460-5d52132a3bff\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v"
May 25 13:18:12.982901 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:12.982785    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1e821a9-67d0-419d-8460-5d52132a3bff-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v\" (UID: \"e1e821a9-67d0-419d-8460-5d52132a3bff\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v"
May 25 13:18:12.982901 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:12.982812    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1e821a9-67d0-419d-8460-5d52132a3bff-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v\" (UID: \"e1e821a9-67d0-419d-8460-5d52132a3bff\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v"
May 25 13:18:12.983125 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:18:12.982936    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert" not found
May 25 13:18:12.983125 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:18:12.983004    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1e821a9-67d0-419d-8460-5d52132a3bff-proxy-tls podName:e1e821a9-67d0-419d-8460-5d52132a3bff nodeName:}" failed. No retries permitted until 2026-05-25 13:18:13.48298315 +0000 UTC m=+3906.443033536 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e1e821a9-67d0-419d-8460-5d52132a3bff-proxy-tls") pod "isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v" (UID: "e1e821a9-67d0-419d-8460-5d52132a3bff") : secret "isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert" not found
May 25 13:18:12.983239 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:12.983152    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1e821a9-67d0-419d-8460-5d52132a3bff-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v\" (UID: \"e1e821a9-67d0-419d-8460-5d52132a3bff\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v"
May 25 13:18:12.983433 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:12.983408    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e1e821a9-67d0-419d-8460-5d52132a3bff-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v\" (UID: \"e1e821a9-67d0-419d-8460-5d52132a3bff\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v"
May 25 13:18:12.983551 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:12.983518    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1e821a9-67d0-419d-8460-5d52132a3bff-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v\" (UID: \"e1e821a9-67d0-419d-8460-5d52132a3bff\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v"
May 25 13:18:12.991767 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:12.991744    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pld8t\" (UniqueName: \"kubernetes.io/projected/e1e821a9-67d0-419d-8460-5d52132a3bff-kube-api-access-pld8t\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v\" (UID: \"e1e821a9-67d0-419d-8460-5d52132a3bff\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v"
May 25 13:18:13.486484 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:13.486446    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1e821a9-67d0-419d-8460-5d52132a3bff-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v\" (UID: \"e1e821a9-67d0-419d-8460-5d52132a3bff\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v"
May 25 13:18:13.488856 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:13.488833    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1e821a9-67d0-419d-8460-5d52132a3bff-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v\" (UID: \"e1e821a9-67d0-419d-8460-5d52132a3bff\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v"
May 25 13:18:13.717680 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:13.717622    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v"
May 25 13:18:13.847678 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:13.847616    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v"]
May 25 13:18:13.850565 ip-10-0-134-255 kubenswrapper[2555]: W0525 13:18:13.850532    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1e821a9_67d0_419d_8460_5d52132a3bff.slice/crio-fd5a7704fafcdf28ef84e39022f134e738a081038c84bd478903941c4acc241f WatchSource:0}: Error finding container fd5a7704fafcdf28ef84e39022f134e738a081038c84bd478903941c4acc241f: Status 404 returned error can't find the container with id fd5a7704fafcdf28ef84e39022f134e738a081038c84bd478903941c4acc241f
May 25 13:18:14.841014 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:14.840980    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v" event={"ID":"e1e821a9-67d0-419d-8460-5d52132a3bff","Type":"ContainerStarted","Data":"d5f7c182eef41f9c637572978cb3ab655f650650d6f5611b8967aa3024883e3f"}
May 25 13:18:14.841149 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:14.841027    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v" event={"ID":"e1e821a9-67d0-419d-8460-5d52132a3bff","Type":"ContainerStarted","Data":"fd5a7704fafcdf28ef84e39022f134e738a081038c84bd478903941c4acc241f"}
May 25 13:18:15.660145 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:15.660119    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-fbs5w_a99c9fa5-024b-4147-8c59-e7a13a42b76a/storage-initializer/1.log"
May 25 13:18:15.660455 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:15.660432    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-fbs5w_a99c9fa5-024b-4147-8c59-e7a13a42b76a/storage-initializer/0.log"
May 25 13:18:15.660504 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:15.660494    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-fbs5w"
May 25 13:18:15.807478 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:15.807397    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a99c9fa5-024b-4147-8c59-e7a13a42b76a-proxy-tls\") pod \"a99c9fa5-024b-4147-8c59-e7a13a42b76a\" (UID: \"a99c9fa5-024b-4147-8c59-e7a13a42b76a\") "
May 25 13:18:15.807478 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:15.807434    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a99c9fa5-024b-4147-8c59-e7a13a42b76a-kserve-provision-location\") pod \"a99c9fa5-024b-4147-8c59-e7a13a42b76a\" (UID: \"a99c9fa5-024b-4147-8c59-e7a13a42b76a\") "
May 25 13:18:15.807478 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:15.807458    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j2qlb\" (UniqueName: \"kubernetes.io/projected/a99c9fa5-024b-4147-8c59-e7a13a42b76a-kube-api-access-j2qlb\") pod \"a99c9fa5-024b-4147-8c59-e7a13a42b76a\" (UID: \"a99c9fa5-024b-4147-8c59-e7a13a42b76a\") "
May 25 13:18:15.807795 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:15.807590    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a99c9fa5-024b-4147-8c59-e7a13a42b76a-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\") pod \"a99c9fa5-024b-4147-8c59-e7a13a42b76a\" (UID: \"a99c9fa5-024b-4147-8c59-e7a13a42b76a\") "
May 25 13:18:15.807795 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:15.807710    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a99c9fa5-024b-4147-8c59-e7a13a42b76a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a99c9fa5-024b-4147-8c59-e7a13a42b76a" (UID: "a99c9fa5-024b-4147-8c59-e7a13a42b76a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:18:15.807927 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:15.807908    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a99c9fa5-024b-4147-8c59-e7a13a42b76a-kserve-provision-location\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:18:15.807976 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:15.807919    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a99c9fa5-024b-4147-8c59-e7a13a42b76a-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config") pod "a99c9fa5-024b-4147-8c59-e7a13a42b76a" (UID: "a99c9fa5-024b-4147-8c59-e7a13a42b76a"). InnerVolumeSpecName "isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:18:15.809544 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:15.809518    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a99c9fa5-024b-4147-8c59-e7a13a42b76a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a99c9fa5-024b-4147-8c59-e7a13a42b76a" (UID: "a99c9fa5-024b-4147-8c59-e7a13a42b76a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:18:15.809634 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:15.809559    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a99c9fa5-024b-4147-8c59-e7a13a42b76a-kube-api-access-j2qlb" (OuterVolumeSpecName: "kube-api-access-j2qlb") pod "a99c9fa5-024b-4147-8c59-e7a13a42b76a" (UID: "a99c9fa5-024b-4147-8c59-e7a13a42b76a"). InnerVolumeSpecName "kube-api-access-j2qlb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:18:15.845587 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:15.845552    2555 generic.go:358] "Generic (PLEG): container finished" podID="e1e821a9-67d0-419d-8460-5d52132a3bff" containerID="d5f7c182eef41f9c637572978cb3ab655f650650d6f5611b8967aa3024883e3f" exitCode=0
May 25 13:18:15.845764 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:15.845636    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v" event={"ID":"e1e821a9-67d0-419d-8460-5d52132a3bff","Type":"ContainerDied","Data":"d5f7c182eef41f9c637572978cb3ab655f650650d6f5611b8967aa3024883e3f"}
May 25 13:18:15.846946 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:15.846926    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-fbs5w_a99c9fa5-024b-4147-8c59-e7a13a42b76a/storage-initializer/1.log"
May 25 13:18:15.847273 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:15.847258    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-fbs5w_a99c9fa5-024b-4147-8c59-e7a13a42b76a/storage-initializer/0.log"
May 25 13:18:15.847346 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:15.847287    2555 generic.go:358] "Generic (PLEG): container finished" podID="a99c9fa5-024b-4147-8c59-e7a13a42b76a" containerID="ce8ffceeaae98e3f12bcc5aa4e1d95e4117b5d2e8867b7653886238650689aac" exitCode=1
May 25 13:18:15.847346 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:15.847323    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-fbs5w" event={"ID":"a99c9fa5-024b-4147-8c59-e7a13a42b76a","Type":"ContainerDied","Data":"ce8ffceeaae98e3f12bcc5aa4e1d95e4117b5d2e8867b7653886238650689aac"}
May 25 13:18:15.847438 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:15.847352    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-fbs5w" event={"ID":"a99c9fa5-024b-4147-8c59-e7a13a42b76a","Type":"ContainerDied","Data":"4ecb3df412213d14f87a6808821271fb6c1a6b2d5e429a3302f900c59a75d13d"}
May 25 13:18:15.847438 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:15.847361    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-fbs5w"
May 25 13:18:15.847438 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:15.847371    2555 scope.go:117] "RemoveContainer" containerID="ce8ffceeaae98e3f12bcc5aa4e1d95e4117b5d2e8867b7653886238650689aac"
May 25 13:18:15.855752 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:15.855736    2555 scope.go:117] "RemoveContainer" containerID="b824130a84616b2f480d890e6741c5ce49e9eb12e273322be19073c133badcac"
May 25 13:18:15.863374 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:15.863350    2555 scope.go:117] "RemoveContainer" containerID="ce8ffceeaae98e3f12bcc5aa4e1d95e4117b5d2e8867b7653886238650689aac"
May 25 13:18:15.863694 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:18:15.863670    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce8ffceeaae98e3f12bcc5aa4e1d95e4117b5d2e8867b7653886238650689aac\": container with ID starting with ce8ffceeaae98e3f12bcc5aa4e1d95e4117b5d2e8867b7653886238650689aac not found: ID does not exist" containerID="ce8ffceeaae98e3f12bcc5aa4e1d95e4117b5d2e8867b7653886238650689aac"
May 25 13:18:15.863801 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:15.863703    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce8ffceeaae98e3f12bcc5aa4e1d95e4117b5d2e8867b7653886238650689aac"} err="failed to get container status \"ce8ffceeaae98e3f12bcc5aa4e1d95e4117b5d2e8867b7653886238650689aac\": rpc error: code = NotFound desc = could not find container \"ce8ffceeaae98e3f12bcc5aa4e1d95e4117b5d2e8867b7653886238650689aac\": container with ID starting with ce8ffceeaae98e3f12bcc5aa4e1d95e4117b5d2e8867b7653886238650689aac not found: ID does not exist"
May 25 13:18:15.863801 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:15.863727    2555 scope.go:117] "RemoveContainer" containerID="b824130a84616b2f480d890e6741c5ce49e9eb12e273322be19073c133badcac"
May 25 13:18:15.863997 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:18:15.863977    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b824130a84616b2f480d890e6741c5ce49e9eb12e273322be19073c133badcac\": container with ID starting with b824130a84616b2f480d890e6741c5ce49e9eb12e273322be19073c133badcac not found: ID does not exist" containerID="b824130a84616b2f480d890e6741c5ce49e9eb12e273322be19073c133badcac"
May 25 13:18:15.864068 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:15.864005    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b824130a84616b2f480d890e6741c5ce49e9eb12e273322be19073c133badcac"} err="failed to get container status \"b824130a84616b2f480d890e6741c5ce49e9eb12e273322be19073c133badcac\": rpc error: code = NotFound desc = could not find container \"b824130a84616b2f480d890e6741c5ce49e9eb12e273322be19073c133badcac\": container with ID starting with b824130a84616b2f480d890e6741c5ce49e9eb12e273322be19073c133badcac not found: ID does not exist"
May 25 13:18:15.889001 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:15.888970    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-fbs5w"]
May 25 13:18:15.890715 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:15.890636    2555 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-fbs5w"]
May 25 13:18:15.908973 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:15.908942    2555 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a99c9fa5-024b-4147-8c59-e7a13a42b76a-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:18:15.909097 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:15.908976    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a99c9fa5-024b-4147-8c59-e7a13a42b76a-proxy-tls\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:18:15.909097 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:15.908994    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2qlb\" (UniqueName: \"kubernetes.io/projected/a99c9fa5-024b-4147-8c59-e7a13a42b76a-kube-api-access-j2qlb\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:18:16.852220 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:16.852181    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v" event={"ID":"e1e821a9-67d0-419d-8460-5d52132a3bff","Type":"ContainerStarted","Data":"8b1658afcd3e90dc2d62f40b4b1644d2cec5a7fc8d85d254d5b4dace3a34b7f1"}
May 25 13:18:16.852688 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:16.852228    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v" event={"ID":"e1e821a9-67d0-419d-8460-5d52132a3bff","Type":"ContainerStarted","Data":"49ffcab4a1f022c9a3d6f21d55dfc79c51bb45c1d816d154665c1035f806f676"}
May 25 13:18:16.852688 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:16.852400    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v"
May 25 13:18:16.852688 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:16.852535    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v"
May 25 13:18:16.853882 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:16.853856    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v" podUID="e1e821a9-67d0-419d-8460-5d52132a3bff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
May 25 13:18:16.870160 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:16.870114    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v" podStartSLOduration=4.870101545 podStartE2EDuration="4.870101545s" podCreationTimestamp="2026-05-25 13:18:12 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:18:16.868435506 +0000 UTC m=+3909.828485912" watchObservedRunningTime="2026-05-25 13:18:16.870101545 +0000 UTC m=+3909.830151950"
May 25 13:18:17.599085 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:17.599050    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a99c9fa5-024b-4147-8c59-e7a13a42b76a" path="/var/lib/kubelet/pods/a99c9fa5-024b-4147-8c59-e7a13a42b76a/volumes"
May 25 13:18:17.857072 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:17.856984    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v" podUID="e1e821a9-67d0-419d-8460-5d52132a3bff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
May 25 13:18:22.861681 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:22.861633    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v"
May 25 13:18:22.862247 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:22.862213    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v" podUID="e1e821a9-67d0-419d-8460-5d52132a3bff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
May 25 13:18:32.862137 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:32.862097    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v" podUID="e1e821a9-67d0-419d-8460-5d52132a3bff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
May 25 13:18:42.862602 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:42.862561    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v" podUID="e1e821a9-67d0-419d-8460-5d52132a3bff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
May 25 13:18:52.862171 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:18:52.862128    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v" podUID="e1e821a9-67d0-419d-8460-5d52132a3bff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
May 25 13:19:02.862412 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:02.862369    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v" podUID="e1e821a9-67d0-419d-8460-5d52132a3bff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
May 25 13:19:12.862174 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:12.862131    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v" podUID="e1e821a9-67d0-419d-8460-5d52132a3bff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
May 25 13:19:22.862833 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:22.862797    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v"
May 25 13:19:32.815512 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:32.815469    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v"]
May 25 13:19:32.816124 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:32.815915    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v" podUID="e1e821a9-67d0-419d-8460-5d52132a3bff" containerName="kserve-container" containerID="cri-o://49ffcab4a1f022c9a3d6f21d55dfc79c51bb45c1d816d154665c1035f806f676" gracePeriod=30
May 25 13:19:32.816124 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:32.815984    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v" podUID="e1e821a9-67d0-419d-8460-5d52132a3bff" containerName="kube-rbac-proxy" containerID="cri-o://8b1658afcd3e90dc2d62f40b4b1644d2cec5a7fc8d85d254d5b4dace3a34b7f1" gracePeriod=30
May 25 13:19:32.858093 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:32.858041    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v" podUID="e1e821a9-67d0-419d-8460-5d52132a3bff" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.65:8643/healthz\": dial tcp 10.134.0.65:8643: connect: connection refused"
May 25 13:19:33.117206 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:33.117120    2555 generic.go:358] "Generic (PLEG): container finished" podID="e1e821a9-67d0-419d-8460-5d52132a3bff" containerID="8b1658afcd3e90dc2d62f40b4b1644d2cec5a7fc8d85d254d5b4dace3a34b7f1" exitCode=2
May 25 13:19:33.117206 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:33.117194    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v" event={"ID":"e1e821a9-67d0-419d-8460-5d52132a3bff","Type":"ContainerDied","Data":"8b1658afcd3e90dc2d62f40b4b1644d2cec5a7fc8d85d254d5b4dace3a34b7f1"}
May 25 13:19:33.924048 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:33.924006    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-jd8v2"]
May 25 13:19:33.924429 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:33.924382    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a99c9fa5-024b-4147-8c59-e7a13a42b76a" containerName="storage-initializer"
May 25 13:19:33.924429 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:33.924394    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a99c9fa5-024b-4147-8c59-e7a13a42b76a" containerName="storage-initializer"
May 25 13:19:33.924429 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:33.924402    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a99c9fa5-024b-4147-8c59-e7a13a42b76a" containerName="storage-initializer"
May 25 13:19:33.924429 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:33.924407    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a99c9fa5-024b-4147-8c59-e7a13a42b76a" containerName="storage-initializer"
May 25 13:19:33.924563 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:33.924476    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="a99c9fa5-024b-4147-8c59-e7a13a42b76a" containerName="storage-initializer"
May 25 13:19:33.924602 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:33.924593    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="a99c9fa5-024b-4147-8c59-e7a13a42b76a" containerName="storage-initializer"
May 25 13:19:33.927699 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:33.927675    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-jd8v2"
May 25 13:19:33.929607 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:33.929587    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
May 25 13:19:33.929724 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:33.929628    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
May 25 13:19:33.944358 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:33.944327    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-jd8v2"]
May 25 13:19:34.094960 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:34.094924    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0887a890-c7fd-4e27-8aaa-acbe8364240f-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-jd8v2\" (UID: \"0887a890-c7fd-4e27-8aaa-acbe8364240f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-jd8v2"
May 25 13:19:34.094960 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:34.094958    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0887a890-c7fd-4e27-8aaa-acbe8364240f-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-jd8v2\" (UID: \"0887a890-c7fd-4e27-8aaa-acbe8364240f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-jd8v2"
May 25 13:19:34.095189 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:34.095040    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s67s\" (UniqueName: \"kubernetes.io/projected/0887a890-c7fd-4e27-8aaa-acbe8364240f-kube-api-access-2s67s\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-jd8v2\" (UID: \"0887a890-c7fd-4e27-8aaa-acbe8364240f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-jd8v2"
May 25 13:19:34.095189 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:34.095069    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0887a890-c7fd-4e27-8aaa-acbe8364240f-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-jd8v2\" (UID: \"0887a890-c7fd-4e27-8aaa-acbe8364240f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-jd8v2"
May 25 13:19:34.195779 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:34.195749    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0887a890-c7fd-4e27-8aaa-acbe8364240f-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-jd8v2\" (UID: \"0887a890-c7fd-4e27-8aaa-acbe8364240f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-jd8v2"
May 25 13:19:34.195952 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:34.195790    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0887a890-c7fd-4e27-8aaa-acbe8364240f-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-jd8v2\" (UID: \"0887a890-c7fd-4e27-8aaa-acbe8364240f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-jd8v2"
May 25 13:19:34.195952 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:34.195843    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2s67s\" (UniqueName: \"kubernetes.io/projected/0887a890-c7fd-4e27-8aaa-acbe8364240f-kube-api-access-2s67s\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-jd8v2\" (UID: \"0887a890-c7fd-4e27-8aaa-acbe8364240f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-jd8v2"
May 25 13:19:34.195952 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:34.195868    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0887a890-c7fd-4e27-8aaa-acbe8364240f-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-jd8v2\" (UID: \"0887a890-c7fd-4e27-8aaa-acbe8364240f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-jd8v2"
May 25 13:19:34.196347 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:34.196319    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0887a890-c7fd-4e27-8aaa-acbe8364240f-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-jd8v2\" (UID: \"0887a890-c7fd-4e27-8aaa-acbe8364240f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-jd8v2"
May 25 13:19:34.196638 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:34.196608    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0887a890-c7fd-4e27-8aaa-acbe8364240f-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-jd8v2\" (UID: \"0887a890-c7fd-4e27-8aaa-acbe8364240f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-jd8v2"
May 25 13:19:34.198196 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:34.198174    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0887a890-c7fd-4e27-8aaa-acbe8364240f-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-jd8v2\" (UID: \"0887a890-c7fd-4e27-8aaa-acbe8364240f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-jd8v2"
May 25 13:19:34.202893 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:34.202870    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s67s\" (UniqueName: \"kubernetes.io/projected/0887a890-c7fd-4e27-8aaa-acbe8364240f-kube-api-access-2s67s\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-jd8v2\" (UID: \"0887a890-c7fd-4e27-8aaa-acbe8364240f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-jd8v2"
May 25 13:19:34.237827 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:34.237790    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-jd8v2"
May 25 13:19:34.366622 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:34.366592    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-jd8v2"]
May 25 13:19:34.369134 ip-10-0-134-255 kubenswrapper[2555]: W0525 13:19:34.369106    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0887a890_c7fd_4e27_8aaa_acbe8364240f.slice/crio-fd95ab04a8b1593406adcf3734722a47c53a446f3bba3b3556ec78a767ad88b1 WatchSource:0}: Error finding container fd95ab04a8b1593406adcf3734722a47c53a446f3bba3b3556ec78a767ad88b1: Status 404 returned error can't find the container with id fd95ab04a8b1593406adcf3734722a47c53a446f3bba3b3556ec78a767ad88b1
May 25 13:19:35.126041 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:35.126001    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-jd8v2" event={"ID":"0887a890-c7fd-4e27-8aaa-acbe8364240f","Type":"ContainerStarted","Data":"85d04ec6b985303726e83497451f4e8be50eb0be75dd446877ddf5e9ea530e8e"}
May 25 13:19:35.126041 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:35.126048    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-jd8v2" event={"ID":"0887a890-c7fd-4e27-8aaa-acbe8364240f","Type":"ContainerStarted","Data":"fd95ab04a8b1593406adcf3734722a47c53a446f3bba3b3556ec78a767ad88b1"}
May 25 13:19:37.370994 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:37.370969    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v"
May 25 13:19:37.419249 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:37.419205    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1e821a9-67d0-419d-8460-5d52132a3bff-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\") pod \"e1e821a9-67d0-419d-8460-5d52132a3bff\" (UID: \"e1e821a9-67d0-419d-8460-5d52132a3bff\") "
May 25 13:19:37.419431 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:37.419261    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1e821a9-67d0-419d-8460-5d52132a3bff-kserve-provision-location\") pod \"e1e821a9-67d0-419d-8460-5d52132a3bff\" (UID: \"e1e821a9-67d0-419d-8460-5d52132a3bff\") "
May 25 13:19:37.419431 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:37.419315    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1e821a9-67d0-419d-8460-5d52132a3bff-proxy-tls\") pod \"e1e821a9-67d0-419d-8460-5d52132a3bff\" (UID: \"e1e821a9-67d0-419d-8460-5d52132a3bff\") "
May 25 13:19:37.419431 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:37.419351    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e1e821a9-67d0-419d-8460-5d52132a3bff-cabundle-cert\") pod \"e1e821a9-67d0-419d-8460-5d52132a3bff\" (UID: \"e1e821a9-67d0-419d-8460-5d52132a3bff\") "
May 25 13:19:37.419431 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:37.419387    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pld8t\" (UniqueName: \"kubernetes.io/projected/e1e821a9-67d0-419d-8460-5d52132a3bff-kube-api-access-pld8t\") pod \"e1e821a9-67d0-419d-8460-5d52132a3bff\" (UID: \"e1e821a9-67d0-419d-8460-5d52132a3bff\") "
May 25 13:19:37.419741 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:37.419714    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1e821a9-67d0-419d-8460-5d52132a3bff-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config") pod "e1e821a9-67d0-419d-8460-5d52132a3bff" (UID: "e1e821a9-67d0-419d-8460-5d52132a3bff"). InnerVolumeSpecName "isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:19:37.419804 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:37.419769    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1e821a9-67d0-419d-8460-5d52132a3bff-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e1e821a9-67d0-419d-8460-5d52132a3bff" (UID: "e1e821a9-67d0-419d-8460-5d52132a3bff"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:19:37.419842 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:37.419799    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1e821a9-67d0-419d-8460-5d52132a3bff-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e1e821a9-67d0-419d-8460-5d52132a3bff" (UID: "e1e821a9-67d0-419d-8460-5d52132a3bff"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:19:37.421507 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:37.421485    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1e821a9-67d0-419d-8460-5d52132a3bff-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e1e821a9-67d0-419d-8460-5d52132a3bff" (UID: "e1e821a9-67d0-419d-8460-5d52132a3bff"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:19:37.421622 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:37.421504    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1e821a9-67d0-419d-8460-5d52132a3bff-kube-api-access-pld8t" (OuterVolumeSpecName: "kube-api-access-pld8t") pod "e1e821a9-67d0-419d-8460-5d52132a3bff" (UID: "e1e821a9-67d0-419d-8460-5d52132a3bff"). InnerVolumeSpecName "kube-api-access-pld8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:19:37.520032 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:37.519935    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1e821a9-67d0-419d-8460-5d52132a3bff-kserve-provision-location\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:19:37.520032 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:37.519971    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1e821a9-67d0-419d-8460-5d52132a3bff-proxy-tls\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:19:37.520032 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:37.519982    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e1e821a9-67d0-419d-8460-5d52132a3bff-cabundle-cert\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:19:37.520032 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:37.519993    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pld8t\" (UniqueName: \"kubernetes.io/projected/e1e821a9-67d0-419d-8460-5d52132a3bff-kube-api-access-pld8t\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:19:37.520032 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:37.520004    2555 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1e821a9-67d0-419d-8460-5d52132a3bff-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:19:38.137624 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:38.137582    2555 generic.go:358] "Generic (PLEG): container finished" podID="e1e821a9-67d0-419d-8460-5d52132a3bff" containerID="49ffcab4a1f022c9a3d6f21d55dfc79c51bb45c1d816d154665c1035f806f676" exitCode=0
May 25 13:19:38.137826 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:38.137671    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v" event={"ID":"e1e821a9-67d0-419d-8460-5d52132a3bff","Type":"ContainerDied","Data":"49ffcab4a1f022c9a3d6f21d55dfc79c51bb45c1d816d154665c1035f806f676"}
May 25 13:19:38.137826 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:38.137713    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v" event={"ID":"e1e821a9-67d0-419d-8460-5d52132a3bff","Type":"ContainerDied","Data":"fd5a7704fafcdf28ef84e39022f134e738a081038c84bd478903941c4acc241f"}
May 25 13:19:38.137826 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:38.137728    2555 scope.go:117] "RemoveContainer" containerID="8b1658afcd3e90dc2d62f40b4b1644d2cec5a7fc8d85d254d5b4dace3a34b7f1"
May 25 13:19:38.137826 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:38.137739    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v"
May 25 13:19:38.146850 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:38.146828    2555 scope.go:117] "RemoveContainer" containerID="49ffcab4a1f022c9a3d6f21d55dfc79c51bb45c1d816d154665c1035f806f676"
May 25 13:19:38.156105 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:38.156082    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v"]
May 25 13:19:38.156182 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:38.156119    2555 scope.go:117] "RemoveContainer" containerID="d5f7c182eef41f9c637572978cb3ab655f650650d6f5611b8967aa3024883e3f"
May 25 13:19:38.159381 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:38.159357    2555 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-rwl9v"]
May 25 13:19:38.164224 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:38.164199    2555 scope.go:117] "RemoveContainer" containerID="8b1658afcd3e90dc2d62f40b4b1644d2cec5a7fc8d85d254d5b4dace3a34b7f1"
May 25 13:19:38.164498 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:19:38.164480    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b1658afcd3e90dc2d62f40b4b1644d2cec5a7fc8d85d254d5b4dace3a34b7f1\": container with ID starting with 8b1658afcd3e90dc2d62f40b4b1644d2cec5a7fc8d85d254d5b4dace3a34b7f1 not found: ID does not exist" containerID="8b1658afcd3e90dc2d62f40b4b1644d2cec5a7fc8d85d254d5b4dace3a34b7f1"
May 25 13:19:38.164558 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:38.164508    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b1658afcd3e90dc2d62f40b4b1644d2cec5a7fc8d85d254d5b4dace3a34b7f1"} err="failed to get container status \"8b1658afcd3e90dc2d62f40b4b1644d2cec5a7fc8d85d254d5b4dace3a34b7f1\": rpc error: code = NotFound desc = could not find container \"8b1658afcd3e90dc2d62f40b4b1644d2cec5a7fc8d85d254d5b4dace3a34b7f1\": container with ID starting with 8b1658afcd3e90dc2d62f40b4b1644d2cec5a7fc8d85d254d5b4dace3a34b7f1 not found: ID does not exist"
May 25 13:19:38.164558 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:38.164527    2555 scope.go:117] "RemoveContainer" containerID="49ffcab4a1f022c9a3d6f21d55dfc79c51bb45c1d816d154665c1035f806f676"
May 25 13:19:38.164835 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:19:38.164815    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49ffcab4a1f022c9a3d6f21d55dfc79c51bb45c1d816d154665c1035f806f676\": container with ID starting with 49ffcab4a1f022c9a3d6f21d55dfc79c51bb45c1d816d154665c1035f806f676 not found: ID does not exist" containerID="49ffcab4a1f022c9a3d6f21d55dfc79c51bb45c1d816d154665c1035f806f676"
May 25 13:19:38.164927 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:38.164846    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49ffcab4a1f022c9a3d6f21d55dfc79c51bb45c1d816d154665c1035f806f676"} err="failed to get container status \"49ffcab4a1f022c9a3d6f21d55dfc79c51bb45c1d816d154665c1035f806f676\": rpc error: code = NotFound desc = could not find container \"49ffcab4a1f022c9a3d6f21d55dfc79c51bb45c1d816d154665c1035f806f676\": container with ID starting with 49ffcab4a1f022c9a3d6f21d55dfc79c51bb45c1d816d154665c1035f806f676 not found: ID does not exist"
May 25 13:19:38.164927 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:38.164871    2555 scope.go:117] "RemoveContainer" containerID="d5f7c182eef41f9c637572978cb3ab655f650650d6f5611b8967aa3024883e3f"
May 25 13:19:38.165185 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:19:38.165151    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5f7c182eef41f9c637572978cb3ab655f650650d6f5611b8967aa3024883e3f\": container with ID starting with d5f7c182eef41f9c637572978cb3ab655f650650d6f5611b8967aa3024883e3f not found: ID does not exist" containerID="d5f7c182eef41f9c637572978cb3ab655f650650d6f5611b8967aa3024883e3f"
May 25 13:19:38.165270 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:38.165190    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5f7c182eef41f9c637572978cb3ab655f650650d6f5611b8967aa3024883e3f"} err="failed to get container status \"d5f7c182eef41f9c637572978cb3ab655f650650d6f5611b8967aa3024883e3f\": rpc error: code = NotFound desc = could not find container \"d5f7c182eef41f9c637572978cb3ab655f650650d6f5611b8967aa3024883e3f\": container with ID starting with d5f7c182eef41f9c637572978cb3ab655f650650d6f5611b8967aa3024883e3f not found: ID does not exist"
May 25 13:19:39.599750 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:39.599665    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1e821a9-67d0-419d-8460-5d52132a3bff" path="/var/lib/kubelet/pods/e1e821a9-67d0-419d-8460-5d52132a3bff/volumes"
May 25 13:19:40.147293 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:40.147260    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-jd8v2_0887a890-c7fd-4e27-8aaa-acbe8364240f/storage-initializer/0.log"
May 25 13:19:40.147474 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:40.147303    2555 generic.go:358] "Generic (PLEG): container finished" podID="0887a890-c7fd-4e27-8aaa-acbe8364240f" containerID="85d04ec6b985303726e83497451f4e8be50eb0be75dd446877ddf5e9ea530e8e" exitCode=1
May 25 13:19:40.147474 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:40.147387    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-jd8v2" event={"ID":"0887a890-c7fd-4e27-8aaa-acbe8364240f","Type":"ContainerDied","Data":"85d04ec6b985303726e83497451f4e8be50eb0be75dd446877ddf5e9ea530e8e"}
May 25 13:19:41.152700 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:41.152637    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-jd8v2_0887a890-c7fd-4e27-8aaa-acbe8364240f/storage-initializer/0.log"
May 25 13:19:41.153107 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:41.152741    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-jd8v2" event={"ID":"0887a890-c7fd-4e27-8aaa-acbe8364240f","Type":"ContainerStarted","Data":"7a3ef1ab7cb3f91c108b1f5403969a82904c36660b9ae1a87b41afbc566b013c"}
May 25 13:19:43.927303 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:43.927267    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-jd8v2"]
May 25 13:19:43.927766 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:43.927557    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-jd8v2" podUID="0887a890-c7fd-4e27-8aaa-acbe8364240f" containerName="storage-initializer" containerID="cri-o://7a3ef1ab7cb3f91c108b1f5403969a82904c36660b9ae1a87b41afbc566b013c" gracePeriod=30
May 25 13:19:44.068257 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:44.068233    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-jd8v2_0887a890-c7fd-4e27-8aaa-acbe8364240f/storage-initializer/1.log"
May 25 13:19:44.068581 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:44.068564    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-jd8v2_0887a890-c7fd-4e27-8aaa-acbe8364240f/storage-initializer/0.log"
May 25 13:19:44.068665 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:44.068633    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-jd8v2"
May 25 13:19:44.165629 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:44.165601    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-jd8v2_0887a890-c7fd-4e27-8aaa-acbe8364240f/storage-initializer/1.log"
May 25 13:19:44.166001 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:44.165985    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-jd8v2_0887a890-c7fd-4e27-8aaa-acbe8364240f/storage-initializer/0.log"
May 25 13:19:44.166078 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:44.166021    2555 generic.go:358] "Generic (PLEG): container finished" podID="0887a890-c7fd-4e27-8aaa-acbe8364240f" containerID="7a3ef1ab7cb3f91c108b1f5403969a82904c36660b9ae1a87b41afbc566b013c" exitCode=1
May 25 13:19:44.166143 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:44.166092    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-jd8v2" event={"ID":"0887a890-c7fd-4e27-8aaa-acbe8364240f","Type":"ContainerDied","Data":"7a3ef1ab7cb3f91c108b1f5403969a82904c36660b9ae1a87b41afbc566b013c"}
May 25 13:19:44.166143 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:44.166128    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-jd8v2" event={"ID":"0887a890-c7fd-4e27-8aaa-acbe8364240f","Type":"ContainerDied","Data":"fd95ab04a8b1593406adcf3734722a47c53a446f3bba3b3556ec78a767ad88b1"}
May 25 13:19:44.166143 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:44.166128    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-jd8v2"
May 25 13:19:44.166247 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:44.166188    2555 scope.go:117] "RemoveContainer" containerID="7a3ef1ab7cb3f91c108b1f5403969a82904c36660b9ae1a87b41afbc566b013c"
May 25 13:19:44.173139 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:44.173118    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0887a890-c7fd-4e27-8aaa-acbe8364240f-proxy-tls\") pod \"0887a890-c7fd-4e27-8aaa-acbe8364240f\" (UID: \"0887a890-c7fd-4e27-8aaa-acbe8364240f\") "
May 25 13:19:44.173230 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:44.173147    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2s67s\" (UniqueName: \"kubernetes.io/projected/0887a890-c7fd-4e27-8aaa-acbe8364240f-kube-api-access-2s67s\") pod \"0887a890-c7fd-4e27-8aaa-acbe8364240f\" (UID: \"0887a890-c7fd-4e27-8aaa-acbe8364240f\") "
May 25 13:19:44.173230 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:44.173171    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0887a890-c7fd-4e27-8aaa-acbe8364240f-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\") pod \"0887a890-c7fd-4e27-8aaa-acbe8364240f\" (UID: \"0887a890-c7fd-4e27-8aaa-acbe8364240f\") "
May 25 13:19:44.173320 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:44.173270    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0887a890-c7fd-4e27-8aaa-acbe8364240f-kserve-provision-location\") pod \"0887a890-c7fd-4e27-8aaa-acbe8364240f\" (UID: \"0887a890-c7fd-4e27-8aaa-acbe8364240f\") "
May 25 13:19:44.173545 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:44.173515    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0887a890-c7fd-4e27-8aaa-acbe8364240f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0887a890-c7fd-4e27-8aaa-acbe8364240f" (UID: "0887a890-c7fd-4e27-8aaa-acbe8364240f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:19:44.173694 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:44.173670    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0887a890-c7fd-4e27-8aaa-acbe8364240f-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config") pod "0887a890-c7fd-4e27-8aaa-acbe8364240f" (UID: "0887a890-c7fd-4e27-8aaa-acbe8364240f"). InnerVolumeSpecName "isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:19:44.174382 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:44.174365    2555 scope.go:117] "RemoveContainer" containerID="85d04ec6b985303726e83497451f4e8be50eb0be75dd446877ddf5e9ea530e8e"
May 25 13:19:44.175393 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:44.175284    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0887a890-c7fd-4e27-8aaa-acbe8364240f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0887a890-c7fd-4e27-8aaa-acbe8364240f" (UID: "0887a890-c7fd-4e27-8aaa-acbe8364240f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:19:44.175393 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:44.175381    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0887a890-c7fd-4e27-8aaa-acbe8364240f-kube-api-access-2s67s" (OuterVolumeSpecName: "kube-api-access-2s67s") pod "0887a890-c7fd-4e27-8aaa-acbe8364240f" (UID: "0887a890-c7fd-4e27-8aaa-acbe8364240f"). InnerVolumeSpecName "kube-api-access-2s67s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:19:44.189749 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:44.189729    2555 scope.go:117] "RemoveContainer" containerID="7a3ef1ab7cb3f91c108b1f5403969a82904c36660b9ae1a87b41afbc566b013c"
May 25 13:19:44.190008 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:19:44.189988    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a3ef1ab7cb3f91c108b1f5403969a82904c36660b9ae1a87b41afbc566b013c\": container with ID starting with 7a3ef1ab7cb3f91c108b1f5403969a82904c36660b9ae1a87b41afbc566b013c not found: ID does not exist" containerID="7a3ef1ab7cb3f91c108b1f5403969a82904c36660b9ae1a87b41afbc566b013c"
May 25 13:19:44.190091 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:44.190022    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a3ef1ab7cb3f91c108b1f5403969a82904c36660b9ae1a87b41afbc566b013c"} err="failed to get container status \"7a3ef1ab7cb3f91c108b1f5403969a82904c36660b9ae1a87b41afbc566b013c\": rpc error: code = NotFound desc = could not find container \"7a3ef1ab7cb3f91c108b1f5403969a82904c36660b9ae1a87b41afbc566b013c\": container with ID starting with 7a3ef1ab7cb3f91c108b1f5403969a82904c36660b9ae1a87b41afbc566b013c not found: ID does not exist"
May 25 13:19:44.190091 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:44.190069    2555 scope.go:117] "RemoveContainer" containerID="85d04ec6b985303726e83497451f4e8be50eb0be75dd446877ddf5e9ea530e8e"
May 25 13:19:44.190305 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:19:44.190286    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85d04ec6b985303726e83497451f4e8be50eb0be75dd446877ddf5e9ea530e8e\": container with ID starting with 85d04ec6b985303726e83497451f4e8be50eb0be75dd446877ddf5e9ea530e8e not found: ID does not exist" containerID="85d04ec6b985303726e83497451f4e8be50eb0be75dd446877ddf5e9ea530e8e"
May 25 13:19:44.190349 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:44.190312    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85d04ec6b985303726e83497451f4e8be50eb0be75dd446877ddf5e9ea530e8e"} err="failed to get container status \"85d04ec6b985303726e83497451f4e8be50eb0be75dd446877ddf5e9ea530e8e\": rpc error: code = NotFound desc = could not find container \"85d04ec6b985303726e83497451f4e8be50eb0be75dd446877ddf5e9ea530e8e\": container with ID starting with 85d04ec6b985303726e83497451f4e8be50eb0be75dd446877ddf5e9ea530e8e not found: ID does not exist"
May 25 13:19:44.274756 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:44.274723    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0887a890-c7fd-4e27-8aaa-acbe8364240f-proxy-tls\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:19:44.274756 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:44.274755    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s67s\" (UniqueName: \"kubernetes.io/projected/0887a890-c7fd-4e27-8aaa-acbe8364240f-kube-api-access-2s67s\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:19:44.274971 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:44.274769    2555 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0887a890-c7fd-4e27-8aaa-acbe8364240f-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:19:44.274971 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:44.274786    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0887a890-c7fd-4e27-8aaa-acbe8364240f-kserve-provision-location\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:19:44.498903 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:44.498866    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-jd8v2"]
May 25 13:19:44.502286 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:44.502260    2555 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-jd8v2"]
May 25 13:19:44.978040 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:44.977999    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm"]
May 25 13:19:44.978416 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:44.978373    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0887a890-c7fd-4e27-8aaa-acbe8364240f" containerName="storage-initializer"
May 25 13:19:44.978416 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:44.978385    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0887a890-c7fd-4e27-8aaa-acbe8364240f" containerName="storage-initializer"
May 25 13:19:44.978416 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:44.978396    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1e821a9-67d0-419d-8460-5d52132a3bff" containerName="kserve-container"
May 25 13:19:44.978416 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:44.978402    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1e821a9-67d0-419d-8460-5d52132a3bff" containerName="kserve-container"
May 25 13:19:44.978550 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:44.978416    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1e821a9-67d0-419d-8460-5d52132a3bff" containerName="kube-rbac-proxy"
May 25 13:19:44.978550 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:44.978425    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1e821a9-67d0-419d-8460-5d52132a3bff" containerName="kube-rbac-proxy"
May 25 13:19:44.978550 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:44.978444    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1e821a9-67d0-419d-8460-5d52132a3bff" containerName="storage-initializer"
May 25 13:19:44.978550 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:44.978450    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1e821a9-67d0-419d-8460-5d52132a3bff" containerName="storage-initializer"
May 25 13:19:44.978550 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:44.978507    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="e1e821a9-67d0-419d-8460-5d52132a3bff" containerName="kube-rbac-proxy"
May 25 13:19:44.978550 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:44.978515    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="0887a890-c7fd-4e27-8aaa-acbe8364240f" containerName="storage-initializer"
May 25 13:19:44.978550 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:44.978523    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="0887a890-c7fd-4e27-8aaa-acbe8364240f" containerName="storage-initializer"
May 25 13:19:44.978550 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:44.978531    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="e1e821a9-67d0-419d-8460-5d52132a3bff" containerName="kserve-container"
May 25 13:19:44.978844 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:44.978607    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0887a890-c7fd-4e27-8aaa-acbe8364240f" containerName="storage-initializer"
May 25 13:19:44.978844 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:44.978613    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0887a890-c7fd-4e27-8aaa-acbe8364240f" containerName="storage-initializer"
May 25 13:19:44.983870 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:44.983847    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm"
May 25 13:19:44.985741 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:44.985718    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 25 13:19:44.985871 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:44.985759    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
May 25 13:19:44.985924 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:44.985910    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
May 25 13:19:44.986154 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:44.986137    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6qbkc\""
May 25 13:19:44.986235 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:44.986160    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
May 25 13:19:44.986235 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:44.986185    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 25 13:19:44.986323 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:44.986241    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
May 25 13:19:44.991089 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:44.991065    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm"]
May 25 13:19:45.081992 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:45.081958    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpj8f\" (UniqueName: \"kubernetes.io/projected/17b205cd-931a-4248-bbc0-9fcf8cfdd784-kube-api-access-hpj8f\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm\" (UID: \"17b205cd-931a-4248-bbc0-9fcf8cfdd784\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm"
May 25 13:19:45.082152 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:45.082017    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17b205cd-931a-4248-bbc0-9fcf8cfdd784-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm\" (UID: \"17b205cd-931a-4248-bbc0-9fcf8cfdd784\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm"
May 25 13:19:45.082152 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:45.082110    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17b205cd-931a-4248-bbc0-9fcf8cfdd784-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm\" (UID: \"17b205cd-931a-4248-bbc0-9fcf8cfdd784\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm"
May 25 13:19:45.082152 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:45.082141    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17b205cd-931a-4248-bbc0-9fcf8cfdd784-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm\" (UID: \"17b205cd-931a-4248-bbc0-9fcf8cfdd784\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm"
May 25 13:19:45.082285 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:45.082213    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17b205cd-931a-4248-bbc0-9fcf8cfdd784-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm\" (UID: \"17b205cd-931a-4248-bbc0-9fcf8cfdd784\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm"
May 25 13:19:45.183619 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:45.183575    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17b205cd-931a-4248-bbc0-9fcf8cfdd784-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm\" (UID: \"17b205cd-931a-4248-bbc0-9fcf8cfdd784\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm"
May 25 13:19:45.183619 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:45.183632    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hpj8f\" (UniqueName: \"kubernetes.io/projected/17b205cd-931a-4248-bbc0-9fcf8cfdd784-kube-api-access-hpj8f\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm\" (UID: \"17b205cd-931a-4248-bbc0-9fcf8cfdd784\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm"
May 25 13:19:45.183898 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:45.183690    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17b205cd-931a-4248-bbc0-9fcf8cfdd784-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm\" (UID: \"17b205cd-931a-4248-bbc0-9fcf8cfdd784\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm"
May 25 13:19:45.183898 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:45.183733    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17b205cd-931a-4248-bbc0-9fcf8cfdd784-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm\" (UID: \"17b205cd-931a-4248-bbc0-9fcf8cfdd784\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm"
May 25 13:19:45.183898 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:19:45.183743    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
May 25 13:19:45.183898 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:45.183759    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17b205cd-931a-4248-bbc0-9fcf8cfdd784-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm\" (UID: \"17b205cd-931a-4248-bbc0-9fcf8cfdd784\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm"
May 25 13:19:45.183898 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:19:45.183816    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17b205cd-931a-4248-bbc0-9fcf8cfdd784-proxy-tls podName:17b205cd-931a-4248-bbc0-9fcf8cfdd784 nodeName:}" failed. No retries permitted until 2026-05-25 13:19:45.683795075 +0000 UTC m=+3998.643845460 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/17b205cd-931a-4248-bbc0-9fcf8cfdd784-proxy-tls") pod "isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm" (UID: "17b205cd-931a-4248-bbc0-9fcf8cfdd784") : secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
May 25 13:19:45.184149 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:45.184100    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17b205cd-931a-4248-bbc0-9fcf8cfdd784-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm\" (UID: \"17b205cd-931a-4248-bbc0-9fcf8cfdd784\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm"
May 25 13:19:45.184529 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:45.184506    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17b205cd-931a-4248-bbc0-9fcf8cfdd784-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm\" (UID: \"17b205cd-931a-4248-bbc0-9fcf8cfdd784\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm"
May 25 13:19:45.184567 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:45.184532    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17b205cd-931a-4248-bbc0-9fcf8cfdd784-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm\" (UID: \"17b205cd-931a-4248-bbc0-9fcf8cfdd784\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm"
May 25 13:19:45.191904 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:45.191872    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpj8f\" (UniqueName: \"kubernetes.io/projected/17b205cd-931a-4248-bbc0-9fcf8cfdd784-kube-api-access-hpj8f\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm\" (UID: \"17b205cd-931a-4248-bbc0-9fcf8cfdd784\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm"
May 25 13:19:45.598558 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:45.598522    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0887a890-c7fd-4e27-8aaa-acbe8364240f" path="/var/lib/kubelet/pods/0887a890-c7fd-4e27-8aaa-acbe8364240f/volumes"
May 25 13:19:45.689412 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:45.689372    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17b205cd-931a-4248-bbc0-9fcf8cfdd784-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm\" (UID: \"17b205cd-931a-4248-bbc0-9fcf8cfdd784\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm"
May 25 13:19:45.691861 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:45.691841    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17b205cd-931a-4248-bbc0-9fcf8cfdd784-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm\" (UID: \"17b205cd-931a-4248-bbc0-9fcf8cfdd784\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm"
May 25 13:19:45.895466 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:45.895364    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm"
May 25 13:19:46.020602 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:46.020573    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm"]
May 25 13:19:46.023436 ip-10-0-134-255 kubenswrapper[2555]: W0525 13:19:46.023404    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17b205cd_931a_4248_bbc0_9fcf8cfdd784.slice/crio-5038ccf0594797f8e9075957911c1304464e252f485910fa10b3a2ce9ba15f5c WatchSource:0}: Error finding container 5038ccf0594797f8e9075957911c1304464e252f485910fa10b3a2ce9ba15f5c: Status 404 returned error can't find the container with id 5038ccf0594797f8e9075957911c1304464e252f485910fa10b3a2ce9ba15f5c
May 25 13:19:46.177382 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:46.177290    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm" event={"ID":"17b205cd-931a-4248-bbc0-9fcf8cfdd784","Type":"ContainerStarted","Data":"fd124dc18c2c2e381e66d78e2e47824269b8a49b95609ed4a74f4fef993745c5"}
May 25 13:19:46.177382 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:46.177337    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm" event={"ID":"17b205cd-931a-4248-bbc0-9fcf8cfdd784","Type":"ContainerStarted","Data":"5038ccf0594797f8e9075957911c1304464e252f485910fa10b3a2ce9ba15f5c"}
May 25 13:19:47.183031 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:47.182997    2555 generic.go:358] "Generic (PLEG): container finished" podID="17b205cd-931a-4248-bbc0-9fcf8cfdd784" containerID="fd124dc18c2c2e381e66d78e2e47824269b8a49b95609ed4a74f4fef993745c5" exitCode=0
May 25 13:19:47.183422 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:47.183050    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm" event={"ID":"17b205cd-931a-4248-bbc0-9fcf8cfdd784","Type":"ContainerDied","Data":"fd124dc18c2c2e381e66d78e2e47824269b8a49b95609ed4a74f4fef993745c5"}
May 25 13:19:48.189832 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:48.189794    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm" event={"ID":"17b205cd-931a-4248-bbc0-9fcf8cfdd784","Type":"ContainerStarted","Data":"37fbc584e807ee5d366f28c6ae0cc6c3195af60d051dad03f72314bf16aee72e"}
May 25 13:19:48.189832 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:48.189837    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm" event={"ID":"17b205cd-931a-4248-bbc0-9fcf8cfdd784","Type":"ContainerStarted","Data":"4d3133034b43710206c2614edf9b71825537dea6ea905030b3d5542908c0d2f9"}
May 25 13:19:48.190288 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:48.190026    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm"
May 25 13:19:48.190288 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:48.190116    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm"
May 25 13:19:48.191685 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:48.191655    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm" podUID="17b205cd-931a-4248-bbc0-9fcf8cfdd784" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
May 25 13:19:48.208175 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:48.208121    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm" podStartSLOduration=4.208106156 podStartE2EDuration="4.208106156s" podCreationTimestamp="2026-05-25 13:19:44 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:19:48.206269611 +0000 UTC m=+4001.166320019" watchObservedRunningTime="2026-05-25 13:19:48.208106156 +0000 UTC m=+4001.168156561"
May 25 13:19:49.194036 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:49.193992    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm" podUID="17b205cd-931a-4248-bbc0-9fcf8cfdd784" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
May 25 13:19:54.199285 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:54.199253    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm"
May 25 13:19:54.199902 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:19:54.199875    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm" podUID="17b205cd-931a-4248-bbc0-9fcf8cfdd784" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
May 25 13:20:04.200044 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:20:04.199999    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm" podUID="17b205cd-931a-4248-bbc0-9fcf8cfdd784" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
May 25 13:20:14.200109 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:20:14.200069    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm" podUID="17b205cd-931a-4248-bbc0-9fcf8cfdd784" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
May 25 13:20:24.199877 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:20:24.199829    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm" podUID="17b205cd-931a-4248-bbc0-9fcf8cfdd784" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
May 25 13:20:34.200162 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:20:34.200061    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm" podUID="17b205cd-931a-4248-bbc0-9fcf8cfdd784" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
May 25 13:20:44.199987 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:20:44.199943    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm" podUID="17b205cd-931a-4248-bbc0-9fcf8cfdd784" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
May 25 13:20:54.201425 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:20:54.201385    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm"
May 25 13:20:55.020942 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:20:55.020910    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm"]
May 25 13:20:55.021240 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:20:55.021214    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm" podUID="17b205cd-931a-4248-bbc0-9fcf8cfdd784" containerName="kserve-container" containerID="cri-o://4d3133034b43710206c2614edf9b71825537dea6ea905030b3d5542908c0d2f9" gracePeriod=30
May 25 13:20:55.021323 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:20:55.021290    2555 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm" podUID="17b205cd-931a-4248-bbc0-9fcf8cfdd784" containerName="kube-rbac-proxy" containerID="cri-o://37fbc584e807ee5d366f28c6ae0cc6c3195af60d051dad03f72314bf16aee72e" gracePeriod=30
May 25 13:20:55.425485 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:20:55.425395    2555 generic.go:358] "Generic (PLEG): container finished" podID="17b205cd-931a-4248-bbc0-9fcf8cfdd784" containerID="37fbc584e807ee5d366f28c6ae0cc6c3195af60d051dad03f72314bf16aee72e" exitCode=2
May 25 13:20:55.425859 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:20:55.425477    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm" event={"ID":"17b205cd-931a-4248-bbc0-9fcf8cfdd784","Type":"ContainerDied","Data":"37fbc584e807ee5d366f28c6ae0cc6c3195af60d051dad03f72314bf16aee72e"}
May 25 13:20:56.109030 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:20:56.108986    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-q4dq8"]
May 25 13:20:56.112591 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:20:56.112567    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-q4dq8"
May 25 13:20:56.114273 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:20:56.114242    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
May 25 13:20:56.114398 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:20:56.114311    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
May 25 13:20:56.121707 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:20:56.121682    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-q4dq8"]
May 25 13:20:56.204526 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:20:56.204492    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/430a9386-fedb-4423-a59b-30748fa7c55f-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-q4dq8\" (UID: \"430a9386-fedb-4423-a59b-30748fa7c55f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-q4dq8"
May 25 13:20:56.204526 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:20:56.204531    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r46gs\" (UniqueName: \"kubernetes.io/projected/430a9386-fedb-4423-a59b-30748fa7c55f-kube-api-access-r46gs\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-q4dq8\" (UID: \"430a9386-fedb-4423-a59b-30748fa7c55f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-q4dq8"
May 25 13:20:56.204752 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:20:56.204566    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/430a9386-fedb-4423-a59b-30748fa7c55f-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-q4dq8\" (UID: \"430a9386-fedb-4423-a59b-30748fa7c55f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-q4dq8"
May 25 13:20:56.204752 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:20:56.204668    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/430a9386-fedb-4423-a59b-30748fa7c55f-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-q4dq8\" (UID: \"430a9386-fedb-4423-a59b-30748fa7c55f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-q4dq8"
May 25 13:20:56.305871 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:20:56.305829    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/430a9386-fedb-4423-a59b-30748fa7c55f-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-q4dq8\" (UID: \"430a9386-fedb-4423-a59b-30748fa7c55f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-q4dq8"
May 25 13:20:56.306060 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:20:56.305904    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/430a9386-fedb-4423-a59b-30748fa7c55f-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-q4dq8\" (UID: \"430a9386-fedb-4423-a59b-30748fa7c55f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-q4dq8"
May 25 13:20:56.306060 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:20:56.305923    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-r46gs\" (UniqueName: \"kubernetes.io/projected/430a9386-fedb-4423-a59b-30748fa7c55f-kube-api-access-r46gs\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-q4dq8\" (UID: \"430a9386-fedb-4423-a59b-30748fa7c55f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-q4dq8"
May 25 13:20:56.306060 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:20:56.305951    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/430a9386-fedb-4423-a59b-30748fa7c55f-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-q4dq8\" (UID: \"430a9386-fedb-4423-a59b-30748fa7c55f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-q4dq8"
May 25 13:20:56.306339 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:20:56.306318    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/430a9386-fedb-4423-a59b-30748fa7c55f-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-q4dq8\" (UID: \"430a9386-fedb-4423-a59b-30748fa7c55f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-q4dq8"
May 25 13:20:56.306586 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:20:56.306564    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/430a9386-fedb-4423-a59b-30748fa7c55f-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-q4dq8\" (UID: \"430a9386-fedb-4423-a59b-30748fa7c55f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-q4dq8"
May 25 13:20:56.308587 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:20:56.308560    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/430a9386-fedb-4423-a59b-30748fa7c55f-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-q4dq8\" (UID: \"430a9386-fedb-4423-a59b-30748fa7c55f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-q4dq8"
May 25 13:20:56.313129 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:20:56.313105    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r46gs\" (UniqueName: \"kubernetes.io/projected/430a9386-fedb-4423-a59b-30748fa7c55f-kube-api-access-r46gs\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-q4dq8\" (UID: \"430a9386-fedb-4423-a59b-30748fa7c55f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-q4dq8"
May 25 13:20:56.424155 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:20:56.424066    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-q4dq8"
May 25 13:20:56.556447 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:20:56.556423    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-q4dq8"]
May 25 13:20:56.558818 ip-10-0-134-255 kubenswrapper[2555]: W0525 13:20:56.558782    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod430a9386_fedb_4423_a59b_30748fa7c55f.slice/crio-ee605b072c9753ac6ec3c1c5bf843507b1e83d2ec18d2e19e18c73e4d86c62f4 WatchSource:0}: Error finding container ee605b072c9753ac6ec3c1c5bf843507b1e83d2ec18d2e19e18c73e4d86c62f4: Status 404 returned error can't find the container with id ee605b072c9753ac6ec3c1c5bf843507b1e83d2ec18d2e19e18c73e4d86c62f4
May 25 13:20:56.561018 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:20:56.560999    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 13:20:57.432881 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:20:57.432840    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-q4dq8" event={"ID":"430a9386-fedb-4423-a59b-30748fa7c55f","Type":"ContainerStarted","Data":"3f1439e07c3689392514e18d1873f453dd549da8b913f99f68b70d71708c36ed"}
May 25 13:20:57.432881 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:20:57.432884    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-q4dq8" event={"ID":"430a9386-fedb-4423-a59b-30748fa7c55f","Type":"ContainerStarted","Data":"ee605b072c9753ac6ec3c1c5bf843507b1e83d2ec18d2e19e18c73e4d86c62f4"}
May 25 13:20:59.194474 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:20:59.194425    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm" podUID="17b205cd-931a-4248-bbc0-9fcf8cfdd784" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.67:8643/healthz\": dial tcp 10.134.0.67:8643: connect: connection refused"
May 25 13:20:59.443701 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:20:59.443669    2555 generic.go:358] "Generic (PLEG): container finished" podID="17b205cd-931a-4248-bbc0-9fcf8cfdd784" containerID="4d3133034b43710206c2614edf9b71825537dea6ea905030b3d5542908c0d2f9" exitCode=0
May 25 13:20:59.443857 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:20:59.443743    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm" event={"ID":"17b205cd-931a-4248-bbc0-9fcf8cfdd784","Type":"ContainerDied","Data":"4d3133034b43710206c2614edf9b71825537dea6ea905030b3d5542908c0d2f9"}
May 25 13:20:59.469493 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:20:59.469461    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm"
May 25 13:20:59.534145 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:20:59.534113    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17b205cd-931a-4248-bbc0-9fcf8cfdd784-cabundle-cert\") pod \"17b205cd-931a-4248-bbc0-9fcf8cfdd784\" (UID: \"17b205cd-931a-4248-bbc0-9fcf8cfdd784\") "
May 25 13:20:59.534145 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:20:59.534152    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17b205cd-931a-4248-bbc0-9fcf8cfdd784-proxy-tls\") pod \"17b205cd-931a-4248-bbc0-9fcf8cfdd784\" (UID: \"17b205cd-931a-4248-bbc0-9fcf8cfdd784\") "
May 25 13:20:59.534402 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:20:59.534214    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hpj8f\" (UniqueName: \"kubernetes.io/projected/17b205cd-931a-4248-bbc0-9fcf8cfdd784-kube-api-access-hpj8f\") pod \"17b205cd-931a-4248-bbc0-9fcf8cfdd784\" (UID: \"17b205cd-931a-4248-bbc0-9fcf8cfdd784\") "
May 25 13:20:59.534402 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:20:59.534241    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17b205cd-931a-4248-bbc0-9fcf8cfdd784-kserve-provision-location\") pod \"17b205cd-931a-4248-bbc0-9fcf8cfdd784\" (UID: \"17b205cd-931a-4248-bbc0-9fcf8cfdd784\") "
May 25 13:20:59.534402 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:20:59.534278    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17b205cd-931a-4248-bbc0-9fcf8cfdd784-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\") pod \"17b205cd-931a-4248-bbc0-9fcf8cfdd784\" (UID: \"17b205cd-931a-4248-bbc0-9fcf8cfdd784\") "
May 25 13:20:59.534666 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:20:59.534615    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17b205cd-931a-4248-bbc0-9fcf8cfdd784-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "17b205cd-931a-4248-bbc0-9fcf8cfdd784" (UID: "17b205cd-931a-4248-bbc0-9fcf8cfdd784"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:20:59.534744 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:20:59.534678    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17b205cd-931a-4248-bbc0-9fcf8cfdd784-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "17b205cd-931a-4248-bbc0-9fcf8cfdd784" (UID: "17b205cd-931a-4248-bbc0-9fcf8cfdd784"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:20:59.534744 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:20:59.534716    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17b205cd-931a-4248-bbc0-9fcf8cfdd784-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config") pod "17b205cd-931a-4248-bbc0-9fcf8cfdd784" (UID: "17b205cd-931a-4248-bbc0-9fcf8cfdd784"). InnerVolumeSpecName "isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:20:59.536442 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:20:59.536407    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17b205cd-931a-4248-bbc0-9fcf8cfdd784-kube-api-access-hpj8f" (OuterVolumeSpecName: "kube-api-access-hpj8f") pod "17b205cd-931a-4248-bbc0-9fcf8cfdd784" (UID: "17b205cd-931a-4248-bbc0-9fcf8cfdd784"). InnerVolumeSpecName "kube-api-access-hpj8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:20:59.536569 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:20:59.536505    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17b205cd-931a-4248-bbc0-9fcf8cfdd784-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "17b205cd-931a-4248-bbc0-9fcf8cfdd784" (UID: "17b205cd-931a-4248-bbc0-9fcf8cfdd784"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:20:59.635879 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:20:59.635852    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpj8f\" (UniqueName: \"kubernetes.io/projected/17b205cd-931a-4248-bbc0-9fcf8cfdd784-kube-api-access-hpj8f\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:20:59.635879 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:20:59.635877    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17b205cd-931a-4248-bbc0-9fcf8cfdd784-kserve-provision-location\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:20:59.636043 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:20:59.635888    2555 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17b205cd-931a-4248-bbc0-9fcf8cfdd784-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:20:59.636043 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:20:59.635899    2555 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17b205cd-931a-4248-bbc0-9fcf8cfdd784-cabundle-cert\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:20:59.636043 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:20:59.635908    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17b205cd-931a-4248-bbc0-9fcf8cfdd784-proxy-tls\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:21:00.448164 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:00.448125    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm" event={"ID":"17b205cd-931a-4248-bbc0-9fcf8cfdd784","Type":"ContainerDied","Data":"5038ccf0594797f8e9075957911c1304464e252f485910fa10b3a2ce9ba15f5c"}
May 25 13:21:00.448164 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:00.448162    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm"
May 25 13:21:00.448737 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:00.448170    2555 scope.go:117] "RemoveContainer" containerID="37fbc584e807ee5d366f28c6ae0cc6c3195af60d051dad03f72314bf16aee72e"
May 25 13:21:00.449860 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:00.449841    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-q4dq8_430a9386-fedb-4423-a59b-30748fa7c55f/storage-initializer/0.log"
May 25 13:21:00.449941 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:00.449880    2555 generic.go:358] "Generic (PLEG): container finished" podID="430a9386-fedb-4423-a59b-30748fa7c55f" containerID="3f1439e07c3689392514e18d1873f453dd549da8b913f99f68b70d71708c36ed" exitCode=1
May 25 13:21:00.449941 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:00.449931    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-q4dq8" event={"ID":"430a9386-fedb-4423-a59b-30748fa7c55f","Type":"ContainerDied","Data":"3f1439e07c3689392514e18d1873f453dd549da8b913f99f68b70d71708c36ed"}
May 25 13:21:00.457254 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:00.457237    2555 scope.go:117] "RemoveContainer" containerID="4d3133034b43710206c2614edf9b71825537dea6ea905030b3d5542908c0d2f9"
May 25 13:21:00.465214 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:00.465186    2555 scope.go:117] "RemoveContainer" containerID="fd124dc18c2c2e381e66d78e2e47824269b8a49b95609ed4a74f4fef993745c5"
May 25 13:21:00.466572 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:00.466393    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm"]
May 25 13:21:00.468452 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:00.468426    2555 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-mmgpm"]
May 25 13:21:01.456205 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:01.456178    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-q4dq8_430a9386-fedb-4423-a59b-30748fa7c55f/storage-initializer/0.log"
May 25 13:21:01.456677 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:01.456269    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-q4dq8" event={"ID":"430a9386-fedb-4423-a59b-30748fa7c55f","Type":"ContainerStarted","Data":"c358e1df678c35f84c76c683040fbe0540930cad983d994e80842122f7264445"}
May 25 13:21:01.599166 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:01.599132    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17b205cd-931a-4248-bbc0-9fcf8cfdd784" path="/var/lib/kubelet/pods/17b205cd-931a-4248-bbc0-9fcf8cfdd784/volumes"
May 25 13:21:04.469400 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:04.469375    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-q4dq8_430a9386-fedb-4423-a59b-30748fa7c55f/storage-initializer/1.log"
May 25 13:21:04.469794 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:04.469742    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-q4dq8_430a9386-fedb-4423-a59b-30748fa7c55f/storage-initializer/0.log"
May 25 13:21:04.469794 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:04.469783    2555 generic.go:358] "Generic (PLEG): container finished" podID="430a9386-fedb-4423-a59b-30748fa7c55f" containerID="c358e1df678c35f84c76c683040fbe0540930cad983d994e80842122f7264445" exitCode=1
May 25 13:21:04.469868 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:04.469852    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-q4dq8" event={"ID":"430a9386-fedb-4423-a59b-30748fa7c55f","Type":"ContainerDied","Data":"c358e1df678c35f84c76c683040fbe0540930cad983d994e80842122f7264445"}
May 25 13:21:04.469904 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:04.469890    2555 scope.go:117] "RemoveContainer" containerID="3f1439e07c3689392514e18d1873f453dd549da8b913f99f68b70d71708c36ed"
May 25 13:21:04.470244 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:04.470222    2555 scope.go:117] "RemoveContainer" containerID="3f1439e07c3689392514e18d1873f453dd549da8b913f99f68b70d71708c36ed"
May 25 13:21:04.480231 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:21:04.480203    2555 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-q4dq8_kserve-ci-e2e-test_430a9386-fedb-4423-a59b-30748fa7c55f_0 in pod sandbox ee605b072c9753ac6ec3c1c5bf843507b1e83d2ec18d2e19e18c73e4d86c62f4 from index: no such id: '3f1439e07c3689392514e18d1873f453dd549da8b913f99f68b70d71708c36ed'" containerID="3f1439e07c3689392514e18d1873f453dd549da8b913f99f68b70d71708c36ed"
May 25 13:21:04.480309 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:21:04.480249    2555 kuberuntime_container.go:951] "Unhandled Error" err="failed to remove pod init container \"storage-initializer\": rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-q4dq8_kserve-ci-e2e-test_430a9386-fedb-4423-a59b-30748fa7c55f_0 in pod sandbox ee605b072c9753ac6ec3c1c5bf843507b1e83d2ec18d2e19e18c73e4d86c62f4 from index: no such id: '3f1439e07c3689392514e18d1873f453dd549da8b913f99f68b70d71708c36ed'; Skipping pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-q4dq8_kserve-ci-e2e-test(430a9386-fedb-4423-a59b-30748fa7c55f)\"" logger="UnhandledError"
May 25 13:21:04.482063 ip-10-0-134-255 kubenswrapper[2555]: E0525 13:21:04.482042    2555 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-q4dq8_kserve-ci-e2e-test(430a9386-fedb-4423-a59b-30748fa7c55f)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-q4dq8" podUID="430a9386-fedb-4423-a59b-30748fa7c55f"
May 25 13:21:05.474131 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:05.474102    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-q4dq8_430a9386-fedb-4423-a59b-30748fa7c55f/storage-initializer/1.log"
May 25 13:21:06.099878 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:06.099844    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-q4dq8"]
May 25 13:21:06.225843 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:06.225820    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-q4dq8_430a9386-fedb-4423-a59b-30748fa7c55f/storage-initializer/1.log"
May 25 13:21:06.225949 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:06.225889    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-q4dq8"
May 25 13:21:06.395167 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:06.395073    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/430a9386-fedb-4423-a59b-30748fa7c55f-kserve-provision-location\") pod \"430a9386-fedb-4423-a59b-30748fa7c55f\" (UID: \"430a9386-fedb-4423-a59b-30748fa7c55f\") "
May 25 13:21:06.395334 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:06.395179    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/430a9386-fedb-4423-a59b-30748fa7c55f-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\") pod \"430a9386-fedb-4423-a59b-30748fa7c55f\" (UID: \"430a9386-fedb-4423-a59b-30748fa7c55f\") "
May 25 13:21:06.395334 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:06.395202    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/430a9386-fedb-4423-a59b-30748fa7c55f-proxy-tls\") pod \"430a9386-fedb-4423-a59b-30748fa7c55f\" (UID: \"430a9386-fedb-4423-a59b-30748fa7c55f\") "
May 25 13:21:06.395334 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:06.395243    2555 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r46gs\" (UniqueName: \"kubernetes.io/projected/430a9386-fedb-4423-a59b-30748fa7c55f-kube-api-access-r46gs\") pod \"430a9386-fedb-4423-a59b-30748fa7c55f\" (UID: \"430a9386-fedb-4423-a59b-30748fa7c55f\") "
May 25 13:21:06.395507 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:06.395335    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/430a9386-fedb-4423-a59b-30748fa7c55f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "430a9386-fedb-4423-a59b-30748fa7c55f" (UID: "430a9386-fedb-4423-a59b-30748fa7c55f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:21:06.395564 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:06.395508    2555 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/430a9386-fedb-4423-a59b-30748fa7c55f-kserve-provision-location\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:21:06.395564 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:06.395551    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/430a9386-fedb-4423-a59b-30748fa7c55f-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config") pod "430a9386-fedb-4423-a59b-30748fa7c55f" (UID: "430a9386-fedb-4423-a59b-30748fa7c55f"). InnerVolumeSpecName "isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:21:06.397274 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:06.397249    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/430a9386-fedb-4423-a59b-30748fa7c55f-kube-api-access-r46gs" (OuterVolumeSpecName: "kube-api-access-r46gs") pod "430a9386-fedb-4423-a59b-30748fa7c55f" (UID: "430a9386-fedb-4423-a59b-30748fa7c55f"). InnerVolumeSpecName "kube-api-access-r46gs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:21:06.397362 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:06.397307    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/430a9386-fedb-4423-a59b-30748fa7c55f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "430a9386-fedb-4423-a59b-30748fa7c55f" (UID: "430a9386-fedb-4423-a59b-30748fa7c55f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:21:06.478442 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:06.478413    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-q4dq8_430a9386-fedb-4423-a59b-30748fa7c55f/storage-initializer/1.log"
May 25 13:21:06.478892 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:06.478476    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-q4dq8" event={"ID":"430a9386-fedb-4423-a59b-30748fa7c55f","Type":"ContainerDied","Data":"ee605b072c9753ac6ec3c1c5bf843507b1e83d2ec18d2e19e18c73e4d86c62f4"}
May 25 13:21:06.478892 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:06.478509    2555 scope.go:117] "RemoveContainer" containerID="c358e1df678c35f84c76c683040fbe0540930cad983d994e80842122f7264445"
May 25 13:21:06.478892 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:06.478538    2555 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-q4dq8"
May 25 13:21:06.496527 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:06.496501    2555 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/430a9386-fedb-4423-a59b-30748fa7c55f-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:21:06.496527 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:06.496525    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/430a9386-fedb-4423-a59b-30748fa7c55f-proxy-tls\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:21:06.496660 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:06.496536    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r46gs\" (UniqueName: \"kubernetes.io/projected/430a9386-fedb-4423-a59b-30748fa7c55f-kube-api-access-r46gs\") on node \"ip-10-0-134-255.ec2.internal\" DevicePath \"\""
May 25 13:21:06.512529 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:06.512502    2555 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-q4dq8"]
May 25 13:21:06.515404 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:06.515381    2555 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-q4dq8"]
May 25 13:21:07.599104 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:07.599070    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="430a9386-fedb-4423-a59b-30748fa7c55f" path="/var/lib/kubelet/pods/430a9386-fedb-4423-a59b-30748fa7c55f/volumes"
May 25 13:21:42.458890 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:42.458856    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xtm76_24f7bcc2-ad35-49e6-b69c-175a02068e46/global-pull-secret-syncer/0.log"
May 25 13:21:42.625773 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:42.625742    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-jpm95_72a67a33-cbd6-49ec-a72d-65cdf815803a/konnectivity-agent/0.log"
May 25 13:21:42.675052 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:42.675019    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-255.ec2.internal_e477f180b7e0e366bb0a0001192721d9/haproxy/0.log"
May 25 13:21:46.558011 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:46.557984    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n2rnq_3a57d23a-40f6-46c3-b15e-4d1dd2608e11/node-exporter/0.log"
May 25 13:21:46.580605 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:46.580574    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n2rnq_3a57d23a-40f6-46c3-b15e-4d1dd2608e11/kube-rbac-proxy/0.log"
May 25 13:21:46.601058 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:46.601032    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n2rnq_3a57d23a-40f6-46c3-b15e-4d1dd2608e11/init-textfile/0.log"
May 25 13:21:46.973549 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:46.973516    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7779bbdd78-7ltcx_82327b68-6f0f-4a92-848b-83c34c6625ce/prometheus-operator/0.log"
May 25 13:21:46.989632 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:46.989604    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7779bbdd78-7ltcx_82327b68-6f0f-4a92-848b-83c34c6625ce/kube-rbac-proxy/0.log"
May 25 13:21:47.046897 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:47.046863    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-57b8d79fd4-cgzsk_77329766-b572-41f7-91cb-a0c23fb1cf81/telemeter-client/0.log"
May 25 13:21:47.075064 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:47.075035    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-57b8d79fd4-cgzsk_77329766-b572-41f7-91cb-a0c23fb1cf81/reload/0.log"
May 25 13:21:47.104630 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:47.104602    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-57b8d79fd4-cgzsk_77329766-b572-41f7-91cb-a0c23fb1cf81/kube-rbac-proxy/0.log"
May 25 13:21:48.996320 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:48.996289    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-2tqtm_00e6f79c-8f05-4330-aa16-1ada2ae9e9cc/console-operator/1.log"
May 25 13:21:49.007533 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:49.007503    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-2tqtm_00e6f79c-8f05-4330-aa16-1ada2ae9e9cc/console-operator/2.log"
May 25 13:21:49.359879 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:49.359792    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6db9bc8779-v7b2x_4d71f6ce-1aec-465f-885e-637117a22cce/console/0.log"
May 25 13:21:49.404455 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:49.404424    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-57bd96b888-jmzg5_67702967-b5f5-412e-8d06-ecc66d064257/download-server/0.log"
May 25 13:21:49.761286 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:49.761251    2555 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wj4hj/perf-node-gather-daemonset-479pv"]
May 25 13:21:49.761590 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:49.761578    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17b205cd-931a-4248-bbc0-9fcf8cfdd784" containerName="kserve-container"
May 25 13:21:49.761658 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:49.761592    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="17b205cd-931a-4248-bbc0-9fcf8cfdd784" containerName="kserve-container"
May 25 13:21:49.761658 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:49.761610    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17b205cd-931a-4248-bbc0-9fcf8cfdd784" containerName="kube-rbac-proxy"
May 25 13:21:49.761658 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:49.761618    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="17b205cd-931a-4248-bbc0-9fcf8cfdd784" containerName="kube-rbac-proxy"
May 25 13:21:49.761658 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:49.761627    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="430a9386-fedb-4423-a59b-30748fa7c55f" containerName="storage-initializer"
May 25 13:21:49.761658 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:49.761636    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="430a9386-fedb-4423-a59b-30748fa7c55f" containerName="storage-initializer"
May 25 13:21:49.761819 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:49.761665    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17b205cd-931a-4248-bbc0-9fcf8cfdd784" containerName="storage-initializer"
May 25 13:21:49.761819 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:49.761671    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="17b205cd-931a-4248-bbc0-9fcf8cfdd784" containerName="storage-initializer"
May 25 13:21:49.761819 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:49.761677    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="430a9386-fedb-4423-a59b-30748fa7c55f" containerName="storage-initializer"
May 25 13:21:49.761819 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:49.761682    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="430a9386-fedb-4423-a59b-30748fa7c55f" containerName="storage-initializer"
May 25 13:21:49.761819 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:49.761733    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="430a9386-fedb-4423-a59b-30748fa7c55f" containerName="storage-initializer"
May 25 13:21:49.761819 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:49.761743    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="17b205cd-931a-4248-bbc0-9fcf8cfdd784" containerName="kube-rbac-proxy"
May 25 13:21:49.761819 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:49.761751    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="430a9386-fedb-4423-a59b-30748fa7c55f" containerName="storage-initializer"
May 25 13:21:49.761819 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:49.761758    2555 memory_manager.go:356] "RemoveStaleState removing state" podUID="17b205cd-931a-4248-bbc0-9fcf8cfdd784" containerName="kserve-container"
May 25 13:21:49.764908 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:49.764885    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wj4hj/perf-node-gather-daemonset-479pv"
May 25 13:21:49.766853 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:49.766825    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wj4hj\"/\"openshift-service-ca.crt\""
May 25 13:21:49.767180 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:49.767161    2555 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wj4hj\"/\"default-dockercfg-lb72r\""
May 25 13:21:49.767436 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:49.767419    2555 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wj4hj\"/\"kube-root-ca.crt\""
May 25 13:21:49.771014 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:49.770983    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wj4hj/perf-node-gather-daemonset-479pv"]
May 25 13:21:49.800109 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:49.800079    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5cf8c89d64-4jhg4_ad5bd493-e25b-42d8-b008-df2b89dda3f9/volume-data-source-validator/0.log"
May 25 13:21:49.847616 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:49.847577    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwr4s\" (UniqueName: \"kubernetes.io/projected/20445645-3a12-4160-84bf-1204ddb5c334-kube-api-access-wwr4s\") pod \"perf-node-gather-daemonset-479pv\" (UID: \"20445645-3a12-4160-84bf-1204ddb5c334\") " pod="openshift-must-gather-wj4hj/perf-node-gather-daemonset-479pv"
May 25 13:21:49.847843 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:49.847628    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/20445645-3a12-4160-84bf-1204ddb5c334-lib-modules\") pod \"perf-node-gather-daemonset-479pv\" (UID: \"20445645-3a12-4160-84bf-1204ddb5c334\") " pod="openshift-must-gather-wj4hj/perf-node-gather-daemonset-479pv"
May 25 13:21:49.847843 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:49.847661    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/20445645-3a12-4160-84bf-1204ddb5c334-proc\") pod \"perf-node-gather-daemonset-479pv\" (UID: \"20445645-3a12-4160-84bf-1204ddb5c334\") " pod="openshift-must-gather-wj4hj/perf-node-gather-daemonset-479pv"
May 25 13:21:49.847843 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:49.847744    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20445645-3a12-4160-84bf-1204ddb5c334-sys\") pod \"perf-node-gather-daemonset-479pv\" (UID: \"20445645-3a12-4160-84bf-1204ddb5c334\") " pod="openshift-must-gather-wj4hj/perf-node-gather-daemonset-479pv"
May 25 13:21:49.847843 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:49.847780    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/20445645-3a12-4160-84bf-1204ddb5c334-podres\") pod \"perf-node-gather-daemonset-479pv\" (UID: \"20445645-3a12-4160-84bf-1204ddb5c334\") " pod="openshift-must-gather-wj4hj/perf-node-gather-daemonset-479pv"
May 25 13:21:49.949026 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:49.948979    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/20445645-3a12-4160-84bf-1204ddb5c334-lib-modules\") pod \"perf-node-gather-daemonset-479pv\" (UID: \"20445645-3a12-4160-84bf-1204ddb5c334\") " pod="openshift-must-gather-wj4hj/perf-node-gather-daemonset-479pv"
May 25 13:21:49.949026 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:49.949028    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/20445645-3a12-4160-84bf-1204ddb5c334-proc\") pod \"perf-node-gather-daemonset-479pv\" (UID: \"20445645-3a12-4160-84bf-1204ddb5c334\") " pod="openshift-must-gather-wj4hj/perf-node-gather-daemonset-479pv"
May 25 13:21:49.949292 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:49.949075    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20445645-3a12-4160-84bf-1204ddb5c334-sys\") pod \"perf-node-gather-daemonset-479pv\" (UID: \"20445645-3a12-4160-84bf-1204ddb5c334\") " pod="openshift-must-gather-wj4hj/perf-node-gather-daemonset-479pv"
May 25 13:21:49.949292 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:49.949099    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/20445645-3a12-4160-84bf-1204ddb5c334-podres\") pod \"perf-node-gather-daemonset-479pv\" (UID: \"20445645-3a12-4160-84bf-1204ddb5c334\") " pod="openshift-must-gather-wj4hj/perf-node-gather-daemonset-479pv"
May 25 13:21:49.949292 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:49.949177    2555 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wwr4s\" (UniqueName: \"kubernetes.io/projected/20445645-3a12-4160-84bf-1204ddb5c334-kube-api-access-wwr4s\") pod \"perf-node-gather-daemonset-479pv\" (UID: \"20445645-3a12-4160-84bf-1204ddb5c334\") " pod="openshift-must-gather-wj4hj/perf-node-gather-daemonset-479pv"
May 25 13:21:49.949292 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:49.949175    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/20445645-3a12-4160-84bf-1204ddb5c334-lib-modules\") pod \"perf-node-gather-daemonset-479pv\" (UID: \"20445645-3a12-4160-84bf-1204ddb5c334\") " pod="openshift-must-gather-wj4hj/perf-node-gather-daemonset-479pv"
May 25 13:21:49.949292 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:49.949181    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20445645-3a12-4160-84bf-1204ddb5c334-sys\") pod \"perf-node-gather-daemonset-479pv\" (UID: \"20445645-3a12-4160-84bf-1204ddb5c334\") " pod="openshift-must-gather-wj4hj/perf-node-gather-daemonset-479pv"
May 25 13:21:49.949292 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:49.949175    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/20445645-3a12-4160-84bf-1204ddb5c334-proc\") pod \"perf-node-gather-daemonset-479pv\" (UID: \"20445645-3a12-4160-84bf-1204ddb5c334\") " pod="openshift-must-gather-wj4hj/perf-node-gather-daemonset-479pv"
May 25 13:21:49.949292 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:49.949247    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/20445645-3a12-4160-84bf-1204ddb5c334-podres\") pod \"perf-node-gather-daemonset-479pv\" (UID: \"20445645-3a12-4160-84bf-1204ddb5c334\") " pod="openshift-must-gather-wj4hj/perf-node-gather-daemonset-479pv"
May 25 13:21:49.956350 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:49.956330    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwr4s\" (UniqueName: \"kubernetes.io/projected/20445645-3a12-4160-84bf-1204ddb5c334-kube-api-access-wwr4s\") pod \"perf-node-gather-daemonset-479pv\" (UID: \"20445645-3a12-4160-84bf-1204ddb5c334\") " pod="openshift-must-gather-wj4hj/perf-node-gather-daemonset-479pv"
May 25 13:21:50.075730 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:50.075625    2555 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wj4hj/perf-node-gather-daemonset-479pv"
May 25 13:21:50.203810 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:50.203727    2555 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wj4hj/perf-node-gather-daemonset-479pv"]
May 25 13:21:50.206472 ip-10-0-134-255 kubenswrapper[2555]: W0525 13:21:50.206446    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod20445645_3a12_4160_84bf_1204ddb5c334.slice/crio-28944cf68a928d3330b8a9da3b450cd2f4f035e53c4ccb2a2c0b12d3c8508e06 WatchSource:0}: Error finding container 28944cf68a928d3330b8a9da3b450cd2f4f035e53c4ccb2a2c0b12d3c8508e06: Status 404 returned error can't find the container with id 28944cf68a928d3330b8a9da3b450cd2f4f035e53c4ccb2a2c0b12d3c8508e06
May 25 13:21:50.454660 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:50.454610    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gn5dh_a5b139bd-e9a1-45b9-9d31-11a98965a2e0/dns/0.log"
May 25 13:21:50.474279 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:50.474259    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gn5dh_a5b139bd-e9a1-45b9-9d31-11a98965a2e0/kube-rbac-proxy/0.log"
May 25 13:21:50.539551 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:50.539522    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dtbvj_ab76ee4f-2142-41d5-99e2-0de355f6564d/dns-node-resolver/0.log"
May 25 13:21:50.634170 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:50.634132    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wj4hj/perf-node-gather-daemonset-479pv" event={"ID":"20445645-3a12-4160-84bf-1204ddb5c334","Type":"ContainerStarted","Data":"6c67e56485ef1fd366ef669f253b0b41af4fbeb0bf3abfb3978f025aee304cb9"}
May 25 13:21:50.634170 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:50.634174    2555 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wj4hj/perf-node-gather-daemonset-479pv" event={"ID":"20445645-3a12-4160-84bf-1204ddb5c334","Type":"ContainerStarted","Data":"28944cf68a928d3330b8a9da3b450cd2f4f035e53c4ccb2a2c0b12d3c8508e06"}
May 25 13:21:50.634372 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:50.634245    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wj4hj/perf-node-gather-daemonset-479pv"
May 25 13:21:50.652774 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:50.652722    2555 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-wj4hj/perf-node-gather-daemonset-479pv" podStartSLOduration=1.652707447 podStartE2EDuration="1.652707447s" podCreationTimestamp="2026-05-25 13:21:49 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:21:50.650986244 +0000 UTC m=+4123.611036650" watchObservedRunningTime="2026-05-25 13:21:50.652707447 +0000 UTC m=+4123.612757852"
May 25 13:21:50.986733 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:50.986701    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5bb9b845b5-rznpl_c429816e-6e0c-4c9d-931f-793af9bac4c6/registry/0.log"
May 25 13:21:51.016779 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:51.016750    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-brxmt_b77364d1-80eb-4af1-9764-ca0f0858270a/node-ca/0.log"
May 25 13:21:52.010610 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:52.010584    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8mblp_26ec0201-e1d2-4816-944f-bbdb03159591/serve-healthcheck-canary/0.log"
May 25 13:21:52.389738 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:52.389659    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-78bdfb888c-sp5bh_3b30b319-e998-4354-99b7-a46815d537cf/insights-operator/1.log"
May 25 13:21:52.413402 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:52.413366    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-78bdfb888c-sp5bh_3b30b319-e998-4354-99b7-a46815d537cf/insights-operator/0.log"
May 25 13:21:52.499894 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:52.499868    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bmlx5_486a5c2b-34d6-4041-a7aa-f4bce7b4d1db/kube-rbac-proxy/0.log"
May 25 13:21:52.521273 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:52.521238    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bmlx5_486a5c2b-34d6-4041-a7aa-f4bce7b4d1db/exporter/0.log"
May 25 13:21:52.543242 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:52.543213    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bmlx5_486a5c2b-34d6-4041-a7aa-f4bce7b4d1db/extractor/0.log"
May 25 13:21:54.608834 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:54.608803    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-6768985688-7fsm9_dfcbe65c-9d64-4170-99b4-5f05ae8bcbf4/manager/0.log"
May 25 13:21:54.656609 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:54.656574    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-86f7b4b499-x47mq_6a8f4405-598f-4067-8baa-22ae0ebba087/server/0.log"
May 25 13:21:55.189187 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:55.189153    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-79f457656b-g7fg4_e0276877-74a8-4402-938c-87bed561e826/seaweedfs/0.log"
May 25 13:21:55.217542 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:55.217490    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-custom-5c88b85bb7-ptpbn_19bb1968-d459-488b-b9a8-b2bea71f370b/seaweedfs-tls-custom/0.log"
May 25 13:21:55.243040 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:55.242984    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-serving-7fd5766db9-vjht7_931de070-ae52-45bc-aafe-0f893f3f6df4/seaweedfs-tls-serving/0.log"
May 25 13:21:56.646478 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:56.646453    2555 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wj4hj/perf-node-gather-daemonset-479pv"
May 25 13:21:59.457496 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:59.457463    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-565644f49-cz2wm_01c35034-4da3-4712-b62f-77b9082e49fd/kube-storage-version-migrator-operator/1.log"
May 25 13:21:59.522235 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:21:59.522205    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-565644f49-cz2wm_01c35034-4da3-4712-b62f-77b9082e49fd/kube-storage-version-migrator-operator/0.log"
May 25 13:22:00.390768 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:22:00.390736    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-7hwgl_80908050-ac10-44a0-9f07-4599e6fb0960/kube-multus/0.log"
May 25 13:22:00.575949 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:22:00.575919    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6tbkc_798578bf-c83f-402e-97ad-b3944200e4f6/kube-multus-additional-cni-plugins/0.log"
May 25 13:22:00.597425 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:22:00.597402    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6tbkc_798578bf-c83f-402e-97ad-b3944200e4f6/egress-router-binary-copy/0.log"
May 25 13:22:00.618656 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:22:00.618612    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6tbkc_798578bf-c83f-402e-97ad-b3944200e4f6/cni-plugins/0.log"
May 25 13:22:00.640867 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:22:00.640789    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6tbkc_798578bf-c83f-402e-97ad-b3944200e4f6/bond-cni-plugin/0.log"
May 25 13:22:00.662664 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:22:00.662621    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6tbkc_798578bf-c83f-402e-97ad-b3944200e4f6/routeoverride-cni/0.log"
May 25 13:22:00.685601 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:22:00.685578    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6tbkc_798578bf-c83f-402e-97ad-b3944200e4f6/whereabouts-cni-bincopy/0.log"
May 25 13:22:00.707606 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:22:00.707581    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6tbkc_798578bf-c83f-402e-97ad-b3944200e4f6/whereabouts-cni/0.log"
May 25 13:22:01.114566 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:22:01.114525    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-nwlcx_30f9c156-1292-4927-9159-eba19cf76012/network-metrics-daemon/0.log"
May 25 13:22:01.132936 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:22:01.132903    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-nwlcx_30f9c156-1292-4927-9159-eba19cf76012/kube-rbac-proxy/0.log"
May 25 13:22:02.359407 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:22:02.359378    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jxpcm_94f59572-7f6a-47f6-8091-56651d472fcf/ovn-controller/0.log"
May 25 13:22:02.426786 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:22:02.426757    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jxpcm_94f59572-7f6a-47f6-8091-56651d472fcf/ovn-acl-logging/0.log"
May 25 13:22:02.473125 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:22:02.473088    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jxpcm_94f59572-7f6a-47f6-8091-56651d472fcf/kube-rbac-proxy-node/0.log"
May 25 13:22:02.525154 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:22:02.525125    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jxpcm_94f59572-7f6a-47f6-8091-56651d472fcf/kube-rbac-proxy-ovn-metrics/0.log"
May 25 13:22:02.562606 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:22:02.562572    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jxpcm_94f59572-7f6a-47f6-8091-56651d472fcf/northd/0.log"
May 25 13:22:02.591938 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:22:02.591907    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jxpcm_94f59572-7f6a-47f6-8091-56651d472fcf/nbdb/0.log"
May 25 13:22:02.628946 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:22:02.628860    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jxpcm_94f59572-7f6a-47f6-8091-56651d472fcf/sbdb/0.log"
May 25 13:22:02.844518 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:22:02.844482    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jxpcm_94f59572-7f6a-47f6-8091-56651d472fcf/ovnkube-controller/0.log"
May 25 13:22:04.499180 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:22:04.499102    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-7f44b6dd87-t97h9_0feec15a-bfbe-4544-b178-9889ca727a87/check-endpoints/0.log"
May 25 13:22:04.551717 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:22:04.551672    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-4wlq9_2d835613-0b9d-4b52-bc2f-a40577b71208/network-check-target-container/0.log"
May 25 13:22:05.549020 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:22:05.548988    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-49hs2_53719c4f-a55b-4140-9d2b-1d690a6bc5bb/iptables-alerter/0.log"
May 25 13:22:06.296717 ip-10-0-134-255 kubenswrapper[2555]: I0525 13:22:06.296680    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-vzv5c_d776d0f6-f465-4279-81fc-33535ec8ea83/tuned/0.log"