May 26 16:01:23.411901 ip-10-0-130-83 systemd[1]: Starting Kubernetes Kubelet...
May 26 16:01:23.856782 ip-10-0-130-83 kubenswrapper[2572]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 26 16:01:23.856782 ip-10-0-130-83 kubenswrapper[2572]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
May 26 16:01:23.856782 ip-10-0-130-83 kubenswrapper[2572]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 26 16:01:23.856782 ip-10-0-130-83 kubenswrapper[2572]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
May 26 16:01:23.856782 ip-10-0-130-83 kubenswrapper[2572]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 26 16:01:23.857998 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.857521    2572 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
May 26 16:01:23.861384 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861366    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 16:01:23.861384 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861383    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 16:01:23.861460 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861387    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 16:01:23.861460 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861392    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 16:01:23.861460 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861395    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 16:01:23.861460 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861398    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 16:01:23.861460 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861401    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 16:01:23.861460 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861403    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 16:01:23.861460 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861406    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 16:01:23.861460 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861414    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 16:01:23.861460 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861417    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 16:01:23.861460 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861420    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 16:01:23.861460 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861423    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 16:01:23.861460 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861426    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 16:01:23.861460 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861428    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 16:01:23.861460 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861431    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 16:01:23.861460 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861434    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 16:01:23.861460 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861437    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 16:01:23.861460 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861439    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 16:01:23.861460 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861442    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 16:01:23.861460 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861444    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 16:01:23.861460 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861447    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 16:01:23.861957 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861450    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 16:01:23.861957 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861452    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 16:01:23.861957 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861455    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 16:01:23.861957 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861457    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 16:01:23.861957 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861460    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 16:01:23.861957 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861462    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 16:01:23.861957 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861466    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 16:01:23.861957 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861469    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 16:01:23.861957 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861472    2572 feature_gate.go:328] unrecognized feature gate: Example
May 26 16:01:23.861957 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861476    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 16:01:23.861957 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861479    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 16:01:23.861957 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861482    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 16:01:23.861957 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861484    2572 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 16:01:23.861957 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861487    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 16:01:23.861957 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861490    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 16:01:23.861957 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861492    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 16:01:23.861957 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861495    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 16:01:23.861957 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861497    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 16:01:23.861957 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861500    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 16:01:23.861957 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861502    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 16:01:23.862496 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861505    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 16:01:23.862496 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861507    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 16:01:23.862496 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861510    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 16:01:23.862496 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861512    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 16:01:23.862496 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861514    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 16:01:23.862496 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861517    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 16:01:23.862496 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861519    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 16:01:23.862496 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861522    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 16:01:23.862496 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861524    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 16:01:23.862496 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861527    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 16:01:23.862496 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861529    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 16:01:23.862496 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861532    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 16:01:23.862496 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861534    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 16:01:23.862496 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861538    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 16:01:23.862496 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861540    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 16:01:23.862496 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861543    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 16:01:23.862496 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861546    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 16:01:23.862496 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861548    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 16:01:23.862496 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861551    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 16:01:23.862496 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861553    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 16:01:23.863006 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861556    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 16:01:23.863006 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861559    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 16:01:23.863006 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861562    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 16:01:23.863006 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861564    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 16:01:23.863006 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861567    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 16:01:23.863006 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861570    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 16:01:23.863006 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861573    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 16:01:23.863006 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861575    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 16:01:23.863006 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861577    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 16:01:23.863006 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861580    2572 feature_gate.go:328] unrecognized feature gate: Example2
May 26 16:01:23.863006 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861582    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 16:01:23.863006 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861585    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 16:01:23.863006 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861587    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 16:01:23.863006 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861590    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 16:01:23.863006 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861592    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 16:01:23.863006 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861594    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 16:01:23.863006 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861597    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 16:01:23.863006 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861613    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 16:01:23.863006 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861616    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 16:01:23.863006 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861619    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 16:01:23.863497 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861621    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 16:01:23.863497 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861626    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 16:01:23.863497 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861631    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 16:01:23.863497 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.861635    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 16:01:23.863497 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862082    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 16:01:23.863497 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862090    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 16:01:23.863497 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862094    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 16:01:23.863497 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862097    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 16:01:23.863497 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862100    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 16:01:23.863497 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862104    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 16:01:23.863497 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862108    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 16:01:23.863497 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862111    2572 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 16:01:23.863497 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862114    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 16:01:23.863497 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862116    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 16:01:23.863497 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862119    2572 feature_gate.go:328] unrecognized feature gate: Example
May 26 16:01:23.863497 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862121    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 16:01:23.863497 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862124    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 16:01:23.863497 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862127    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 16:01:23.863497 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862130    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 16:01:23.863970 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862133    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 16:01:23.863970 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862135    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 16:01:23.863970 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862138    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 16:01:23.863970 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862140    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 16:01:23.863970 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862142    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 16:01:23.863970 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862145    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 16:01:23.863970 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862147    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 16:01:23.863970 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862150    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 16:01:23.863970 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862152    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 16:01:23.863970 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862155    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 16:01:23.863970 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862158    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 16:01:23.863970 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862161    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 16:01:23.863970 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862163    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 16:01:23.863970 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862166    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 16:01:23.863970 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862168    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 16:01:23.863970 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862171    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 16:01:23.863970 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862173    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 16:01:23.863970 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862176    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 16:01:23.863970 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862179    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 16:01:23.863970 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862182    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 16:01:23.864475 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862184    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 16:01:23.864475 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862187    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 16:01:23.864475 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862189    2572 feature_gate.go:328] unrecognized feature gate: Example2
May 26 16:01:23.864475 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862191    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 16:01:23.864475 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862194    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 16:01:23.864475 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862196    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 16:01:23.864475 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862199    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 16:01:23.864475 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862201    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 16:01:23.864475 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862205    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 16:01:23.864475 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862208    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 16:01:23.864475 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862211    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 16:01:23.864475 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862215    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 16:01:23.864475 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862217    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 16:01:23.864475 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862220    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 16:01:23.864475 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862223    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 16:01:23.864475 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862225    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 16:01:23.864475 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862228    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 16:01:23.864475 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862231    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 16:01:23.864475 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862234    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 16:01:23.864475 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862236    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 16:01:23.865025 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862239    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 16:01:23.865025 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862242    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 16:01:23.865025 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862245    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 16:01:23.865025 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862247    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 16:01:23.865025 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862250    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 16:01:23.865025 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862252    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 16:01:23.865025 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862254    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 16:01:23.865025 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862257    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 16:01:23.865025 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862259    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 16:01:23.865025 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862261    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 16:01:23.865025 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862265    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 16:01:23.865025 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862267    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 16:01:23.865025 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862270    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 16:01:23.865025 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862272    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 16:01:23.865025 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862275    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 16:01:23.865025 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862277    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 16:01:23.865025 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862279    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 16:01:23.865025 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862282    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 16:01:23.865025 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862284    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 16:01:23.865488 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862287    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 16:01:23.865488 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862289    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 16:01:23.865488 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862292    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 16:01:23.865488 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862294    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 16:01:23.865488 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862297    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 16:01:23.865488 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862299    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 16:01:23.865488 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862302    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 16:01:23.865488 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862304    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 16:01:23.865488 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862307    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 16:01:23.865488 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862310    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 16:01:23.865488 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862312    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 16:01:23.865488 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.862315    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 16:01:23.865488 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862385    2572 flags.go:64] FLAG: --address="0.0.0.0"
May 26 16:01:23.865488 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862401    2572 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
May 26 16:01:23.865488 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862408    2572 flags.go:64] FLAG: --anonymous-auth="true"
May 26 16:01:23.865488 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862412    2572 flags.go:64] FLAG: --application-metrics-count-limit="100"
May 26 16:01:23.865488 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862417    2572 flags.go:64] FLAG: --authentication-token-webhook="false"
May 26 16:01:23.865488 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862420    2572 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
May 26 16:01:23.865488 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862424    2572 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
May 26 16:01:23.865488 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862429    2572 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
May 26 16:01:23.865488 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862433    2572 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
May 26 16:01:23.866018 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862436    2572 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 26 16:01:23.866018 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862440    2572 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
May 26 16:01:23.866018 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862443    2572 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
May 26 16:01:23.866018 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862447    2572 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
May 26 16:01:23.866018 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862450    2572 flags.go:64] FLAG: --cgroup-root=""
May 26 16:01:23.866018 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862453    2572 flags.go:64] FLAG: --cgroups-per-qos="true"
May 26 16:01:23.866018 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862456    2572 flags.go:64] FLAG: --client-ca-file=""
May 26 16:01:23.866018 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862459    2572 flags.go:64] FLAG: --cloud-config=""
May 26 16:01:23.866018 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862461    2572 flags.go:64] FLAG: --cloud-provider="external"
May 26 16:01:23.866018 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862464    2572 flags.go:64] FLAG: --cluster-dns="[]"
May 26 16:01:23.866018 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862469    2572 flags.go:64] FLAG: --cluster-domain=""
May 26 16:01:23.866018 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862472    2572 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
May 26 16:01:23.866018 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862475    2572 flags.go:64] FLAG: --config-dir=""
May 26 16:01:23.866018 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862478    2572 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 26 16:01:23.866018 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862481    2572 flags.go:64] FLAG: --container-log-max-files="5"
May 26 16:01:23.866018 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862485    2572 flags.go:64] FLAG: --container-log-max-size="10Mi"
May 26 16:01:23.866018 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862489    2572 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
May 26 16:01:23.866018 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862492    2572 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
May 26 16:01:23.866018 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862496    2572 flags.go:64] FLAG: --containerd-namespace="k8s.io"
May 26 16:01:23.866018 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862499    2572 flags.go:64] FLAG: --contention-profiling="false"
May 26 16:01:23.866018 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862502    2572 flags.go:64] FLAG: --cpu-cfs-quota="true"
May 26 16:01:23.866018 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862506    2572 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
May 26 16:01:23.866018 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862510    2572 flags.go:64] FLAG: --cpu-manager-policy="none"
May 26 16:01:23.866018 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862513    2572 flags.go:64] FLAG: --cpu-manager-policy-options=""
May 26 16:01:23.866018 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862517    2572 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
May 26 16:01:23.866659 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862520    2572 flags.go:64] FLAG: --enable-controller-attach-detach="true"
May 26 16:01:23.866659 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862523    2572 flags.go:64] FLAG: --enable-debugging-handlers="true"
May 26 16:01:23.866659 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862526    2572 flags.go:64] FLAG: --enable-load-reader="false"
May 26 16:01:23.866659 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862529    2572 flags.go:64] FLAG: --enable-server="true"
May 26 16:01:23.866659 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862532    2572 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
May 26 16:01:23.866659 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862536    2572 flags.go:64] FLAG: --event-burst="100"
May 26 16:01:23.866659 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862539    2572 flags.go:64] FLAG: --event-qps="50"
May 26 16:01:23.866659 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862542    2572 flags.go:64] FLAG: --event-storage-age-limit="default=0"
May 26 16:01:23.866659 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862545    2572 flags.go:64] FLAG: --event-storage-event-limit="default=0"
May 26 16:01:23.866659 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862549    2572 flags.go:64] FLAG: --eviction-hard=""
May 26 16:01:23.866659 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862553    2572 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
May 26 16:01:23.866659 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862555    2572 flags.go:64] FLAG: --eviction-minimum-reclaim=""
May 26 16:01:23.866659 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862558    2572 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
May 26 16:01:23.866659 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862561    2572 flags.go:64] FLAG: --eviction-soft=""
May 26 16:01:23.866659 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862564    2572 flags.go:64] FLAG: --eviction-soft-grace-period=""
May 26 16:01:23.866659 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862567    2572 flags.go:64] FLAG: --exit-on-lock-contention="false"
May 26 16:01:23.866659 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862570    2572 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
May 26 16:01:23.866659 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862572    2572 flags.go:64] FLAG: --experimental-mounter-path=""
May 26 16:01:23.866659 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862575    2572 flags.go:64] FLAG: --fail-cgroupv1="false"
May 26 16:01:23.866659 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862578    2572 flags.go:64] FLAG: --fail-swap-on="true"
May 26 16:01:23.866659 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862581    2572 flags.go:64] FLAG: --feature-gates=""
May 26 16:01:23.866659 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862585    2572 flags.go:64] FLAG: --file-check-frequency="20s"
May 26 16:01:23.866659 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862589    2572 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
May 26 16:01:23.866659 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862592    2572 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
May 26 16:01:23.866659 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862617    2572 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
May 26 16:01:23.867273 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862622    2572 flags.go:64] FLAG: --healthz-port="10248"
May 26 16:01:23.867273 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862626    2572 flags.go:64] FLAG: --help="false"
May 26 16:01:23.867273 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862628    2572 flags.go:64] FLAG: --hostname-override="ip-10-0-130-83.ec2.internal"
May 26 16:01:23.867273 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862632    2572 flags.go:64] FLAG: --housekeeping-interval="10s"
May 26 16:01:23.867273 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862636    2572 flags.go:64] FLAG: --http-check-frequency="20s"
May 26 16:01:23.867273 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862639    2572 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
May 26 16:01:23.867273 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862643    2572 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
May 26 16:01:23.867273 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862646    2572 flags.go:64] FLAG: --image-gc-high-threshold="85"
May 26 16:01:23.867273 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862649    2572 flags.go:64] FLAG: --image-gc-low-threshold="80"
May 26 16:01:23.867273 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862652    2572 flags.go:64] FLAG: --image-service-endpoint=""
May 26 16:01:23.867273 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862654    2572 flags.go:64] FLAG: --kernel-memcg-notification="false"
May 26 16:01:23.867273 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862657    2572 flags.go:64] FLAG: --kube-api-burst="100"
May 26 16:01:23.867273 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862660    2572 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
May 26 16:01:23.867273 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862663    2572 flags.go:64] FLAG: --kube-api-qps="50"
May 26 16:01:23.867273 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862666    2572 flags.go:64] FLAG: --kube-reserved=""
May 26 16:01:23.867273 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862669    2572 flags.go:64] FLAG: --kube-reserved-cgroup=""
May 26 16:01:23.867273 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862672    2572 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
May 26 16:01:23.867273 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862676    2572 flags.go:64] FLAG: --kubelet-cgroups=""
May 26 16:01:23.867273 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862679    2572 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
May 26 16:01:23.867273 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862681    2572 flags.go:64] FLAG: --lock-file=""
May 26 16:01:23.867273 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862684    2572 flags.go:64] FLAG: --log-cadvisor-usage="false"
May 26 16:01:23.867273 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862687    2572 flags.go:64] FLAG: --log-flush-frequency="5s"
May 26 16:01:23.867273 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862690    2572 flags.go:64] FLAG: --log-json-info-buffer-size="0"
May 26 16:01:23.867273 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862696    2572 flags.go:64] FLAG: --log-json-split-stream="false"
May 26 16:01:23.867878 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862698    2572 flags.go:64] FLAG: --log-text-info-buffer-size="0"
May 26 16:01:23.867878 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862701    2572 flags.go:64] FLAG: --log-text-split-stream="false"
May 26 16:01:23.867878 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862704    2572 flags.go:64] FLAG: --logging-format="text"
May 26 16:01:23.867878 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862707    2572 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 26 16:01:23.867878 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862711    2572 flags.go:64] FLAG: --make-iptables-util-chains="true"
May 26 16:01:23.867878 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862714    2572 flags.go:64] FLAG: --manifest-url=""
May 26 16:01:23.867878 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862717    2572 flags.go:64] FLAG: --manifest-url-header=""
May 26 16:01:23.867878 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862721    2572 flags.go:64] FLAG: --max-housekeeping-interval="15s"
May 26 16:01:23.867878 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862727    2572 flags.go:64] FLAG: --max-open-files="1000000"
May 26 16:01:23.867878 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862732    2572 flags.go:64] FLAG: --max-pods="110"
May 26 16:01:23.867878 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862735    2572 flags.go:64] FLAG: --maximum-dead-containers="-1"
May 26 16:01:23.867878 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862738    2572 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
May 26 16:01:23.867878 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862741    2572 flags.go:64] FLAG: --memory-manager-policy="None"
May 26 16:01:23.867878 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862744    2572 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
May 26 16:01:23.867878 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862746    2572 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
May 26 16:01:23.867878 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862749    2572 flags.go:64] FLAG: --node-ip="0.0.0.0"
May 26 16:01:23.867878 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862752    2572 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
May 26 16:01:23.867878 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862759    2572 flags.go:64] FLAG: --node-status-max-images="50"
May 26 16:01:23.867878 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862763    2572 flags.go:64] FLAG: --node-status-update-frequency="10s"
May 26 16:01:23.867878 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862765    2572 flags.go:64] FLAG: --oom-score-adj="-999"
May 26 16:01:23.867878 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862768    2572 flags.go:64] FLAG: --pod-cidr=""
May 26 16:01:23.867878 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862771    2572 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc"
May 26 16:01:23.867878 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862777    2572 flags.go:64] FLAG: --pod-manifest-path=""
May 26 16:01:23.868440 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862780    2572 flags.go:64] FLAG: --pod-max-pids="-1"
May 26 16:01:23.868440 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862784    2572 flags.go:64] FLAG: --pods-per-core="0"
May 26 16:01:23.868440 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862787    2572 flags.go:64] FLAG: --port="10250"
May 26 16:01:23.868440 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862790    2572 flags.go:64] FLAG: --protect-kernel-defaults="false"
May 26 16:01:23.868440 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862793    2572 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0437245b6ee09a5f8"
May 26 16:01:23.868440 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862797    2572 flags.go:64] FLAG: --qos-reserved=""
May 26 16:01:23.868440 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862799    2572 flags.go:64] FLAG: --read-only-port="10255"
May 26 16:01:23.868440 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862802    2572 flags.go:64] FLAG: --register-node="true"
May 26 16:01:23.868440 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862805    2572 flags.go:64] FLAG: --register-schedulable="true"
May 26 16:01:23.868440 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862808    2572 flags.go:64] FLAG: --register-with-taints=""
May 26 16:01:23.868440 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862812    2572 flags.go:64] FLAG: --registry-burst="10"
May 26 16:01:23.868440 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862814    2572 flags.go:64] FLAG: --registry-qps="5"
May 26 16:01:23.868440 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862817    2572 flags.go:64] FLAG: --reserved-cpus=""
May 26 16:01:23.868440 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862823    2572 flags.go:64] FLAG: --reserved-memory=""
May 26 16:01:23.868440 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862827    2572 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
May 26 16:01:23.868440 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862830    2572 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
May 26 16:01:23.868440 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862833    2572 flags.go:64] FLAG: --rotate-certificates="false"
May 26 16:01:23.868440 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862836    2572 flags.go:64] FLAG: --rotate-server-certificates="false"
May 26 16:01:23.868440 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862840    2572 flags.go:64] FLAG: --runonce="false"
May 26 16:01:23.868440 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862843    2572 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
May 26 16:01:23.868440 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862846    2572 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
May 26 16:01:23.868440 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862849    2572 flags.go:64] FLAG: --seccomp-default="false"
May 26 16:01:23.868440 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862852    2572 flags.go:64] FLAG: --serialize-image-pulls="true"
May 26 16:01:23.868440 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862855    2572 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
May 26 16:01:23.868440 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862858    2572 flags.go:64] FLAG: --storage-driver-db="cadvisor"
May 26 16:01:23.868440 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862862    2572 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
May 26 16:01:23.869080 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862865    2572 flags.go:64] FLAG: --storage-driver-password="root"
May 26 16:01:23.869080 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862867    2572 flags.go:64] FLAG: --storage-driver-secure="false"
May 26 16:01:23.869080 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862870    2572 flags.go:64] FLAG: --storage-driver-table="stats"
May 26 16:01:23.869080 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862873    2572 flags.go:64] FLAG: --storage-driver-user="root"
May 26 16:01:23.869080 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862876    2572 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
May 26 16:01:23.869080 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862879    2572 flags.go:64] FLAG: --sync-frequency="1m0s"
May 26 16:01:23.869080 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862882    2572 flags.go:64] FLAG: --system-cgroups=""
May 26 16:01:23.869080 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862890    2572 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
May 26 16:01:23.869080 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862895    2572 flags.go:64] FLAG: --system-reserved-cgroup=""
May 26 16:01:23.869080 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862898    2572 flags.go:64] FLAG: --tls-cert-file=""
May 26 16:01:23.869080 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862901    2572 flags.go:64] FLAG: --tls-cipher-suites="[]"
May 26 16:01:23.869080 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862906    2572 flags.go:64] FLAG: --tls-min-version=""
May 26 16:01:23.869080 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862909    2572 flags.go:64] FLAG: --tls-private-key-file=""
May 26 16:01:23.869080 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862912    2572 flags.go:64] FLAG: --topology-manager-policy="none"
May 26 16:01:23.869080 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862915    2572 flags.go:64] FLAG: --topology-manager-policy-options=""
May 26 16:01:23.869080 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862918    2572 flags.go:64] FLAG: --topology-manager-scope="container"
May 26 16:01:23.869080 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862922    2572 flags.go:64] FLAG: --v="2"
May 26 16:01:23.869080 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862926    2572 flags.go:64] FLAG: --version="false"
May 26 16:01:23.869080 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862930    2572 flags.go:64] FLAG: --vmodule=""
May 26 16:01:23.869080 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862937    2572 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
May 26 16:01:23.869080 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.862940    2572 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
May 26 16:01:23.869080 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863035    2572 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 16:01:23.869080 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863040    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 16:01:23.869080 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863043    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 16:01:23.869715 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863047    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 16:01:23.869715 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863053    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 16:01:23.869715 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863056    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 16:01:23.869715 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863059    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 16:01:23.869715 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863061    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 16:01:23.869715 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863064    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 16:01:23.869715 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863067    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 16:01:23.869715 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863069    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 16:01:23.869715 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863072    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 16:01:23.869715 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863076    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 16:01:23.869715 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863079    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 16:01:23.869715 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863082    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 16:01:23.869715 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863085    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 16:01:23.869715 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863088    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 16:01:23.869715 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863090    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 16:01:23.869715 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863095    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 16:01:23.869715 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863098    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 16:01:23.869715 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863100    2572 feature_gate.go:328] unrecognized feature gate: Example2
May 26 16:01:23.869715 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863103    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 16:01:23.870210 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863105    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 16:01:23.870210 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863108    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 16:01:23.870210 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863110    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 16:01:23.870210 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863113    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 16:01:23.870210 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863115    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 16:01:23.870210 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863118    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 16:01:23.870210 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863120    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 16:01:23.870210 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863123    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 16:01:23.870210 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863127    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 16:01:23.870210 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863129    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 16:01:23.870210 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863132    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 16:01:23.870210 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863134    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 16:01:23.870210 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863137    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 16:01:23.870210 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863140    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 16:01:23.870210 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863142    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 16:01:23.870210 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863145    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 16:01:23.870210 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863147    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 16:01:23.870210 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863150    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 16:01:23.870210 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863153    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 16:01:23.870210 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863155    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 16:01:23.870728 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863158    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 16:01:23.870728 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863160    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 16:01:23.870728 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863163    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 16:01:23.870728 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863165    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 16:01:23.870728 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863168    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 16:01:23.870728 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863170    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 16:01:23.870728 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863173    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 16:01:23.870728 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863175    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 16:01:23.870728 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863179    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 16:01:23.870728 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863182    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 16:01:23.870728 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863184    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 16:01:23.870728 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863187    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 16:01:23.870728 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863189    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 16:01:23.870728 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863192    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 16:01:23.870728 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863194    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 16:01:23.870728 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863197    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 16:01:23.870728 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863199    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 16:01:23.870728 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863202    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 16:01:23.870728 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863204    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 16:01:23.870728 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863207    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 16:01:23.871211 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863211    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 16:01:23.871211 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863213    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 16:01:23.871211 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863216    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 16:01:23.871211 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863218    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 16:01:23.871211 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863221    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 16:01:23.871211 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863223    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 16:01:23.871211 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863226    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 16:01:23.871211 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863228    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 16:01:23.871211 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863236    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 16:01:23.871211 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863239    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 16:01:23.871211 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863242    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 16:01:23.871211 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863245    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 16:01:23.871211 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863247    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 16:01:23.871211 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863250    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 16:01:23.871211 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863252    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 16:01:23.871211 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863255    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 16:01:23.871211 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863257    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 16:01:23.871211 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863260    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 16:01:23.871211 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863262    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 16:01:23.871689 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863265    2572 feature_gate.go:328] unrecognized feature gate: Example
May 26 16:01:23.871689 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863269    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 16:01:23.871689 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863271    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 16:01:23.871689 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863274    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 16:01:23.871689 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.863276    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 16:01:23.871689 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.863281    2572 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 26 16:01:23.871689 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.870641    2572 server.go:530] "Kubelet version" kubeletVersion="v1.33.11"
May 26 16:01:23.871689 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.870658    2572 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
May 26 16:01:23.871689 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870714    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 16:01:23.871689 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870719    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 16:01:23.871689 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870722    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 16:01:23.871689 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870726    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 16:01:23.871689 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870729    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 16:01:23.871689 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870732    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 16:01:23.871689 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870735    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 16:01:23.871689 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870738    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 16:01:23.872088 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870741    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 16:01:23.872088 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870744    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 16:01:23.872088 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870746    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 16:01:23.872088 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870749    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 16:01:23.872088 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870751    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 16:01:23.872088 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870755    2572 feature_gate.go:328] unrecognized feature gate: Example
May 26 16:01:23.872088 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870758    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 16:01:23.872088 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870760    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 16:01:23.872088 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870763    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 16:01:23.872088 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870766    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 16:01:23.872088 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870768    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 16:01:23.872088 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870771    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 16:01:23.872088 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870774    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 16:01:23.872088 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870777    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 16:01:23.872088 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870780    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 16:01:23.872088 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870782    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 16:01:23.872088 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870785    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 16:01:23.872088 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870788    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 16:01:23.872088 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870790    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 16:01:23.872088 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870793    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 16:01:23.872574 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870796    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 16:01:23.872574 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870798    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 16:01:23.872574 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870801    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 16:01:23.872574 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870804    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 16:01:23.872574 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870808    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 16:01:23.872574 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870810    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 16:01:23.872574 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870813    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 16:01:23.872574 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870815    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 16:01:23.872574 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870818    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 16:01:23.872574 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870821    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 16:01:23.872574 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870824    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 16:01:23.872574 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870828    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 16:01:23.872574 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870832    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 16:01:23.872574 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870835    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 16:01:23.872574 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870837    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 16:01:23.872574 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870840    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 16:01:23.872574 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870842    2572 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 16:01:23.872574 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870845    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 16:01:23.872574 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870848    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 16:01:23.873079 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870851    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 16:01:23.873079 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870854    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 16:01:23.873079 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870857    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 16:01:23.873079 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870861    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 16:01:23.873079 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870865    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 16:01:23.873079 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870868    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 16:01:23.873079 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870871    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 16:01:23.873079 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870873    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 16:01:23.873079 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870876    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 16:01:23.873079 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870878    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 16:01:23.873079 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870880    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 16:01:23.873079 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870883    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 16:01:23.873079 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870886    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 16:01:23.873079 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870888    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 16:01:23.873079 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870891    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 16:01:23.873079 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870893    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 16:01:23.873079 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870896    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 16:01:23.873079 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870899    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 16:01:23.873079 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870901    2572 feature_gate.go:328] unrecognized feature gate: Example2
May 26 16:01:23.873079 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870905    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 16:01:23.873627 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870907    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 16:01:23.873627 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870910    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 16:01:23.873627 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870913    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 16:01:23.873627 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870916    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 16:01:23.873627 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870918    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 16:01:23.873627 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870921    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 16:01:23.873627 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870923    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 16:01:23.873627 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870926    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 16:01:23.873627 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870928    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 16:01:23.873627 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870931    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 16:01:23.873627 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870934    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 16:01:23.873627 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870936    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 16:01:23.873627 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870939    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 16:01:23.873627 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870942    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 16:01:23.873627 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870944    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 16:01:23.873627 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870947    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 16:01:23.873627 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870949    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 16:01:23.873627 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870952    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 16:01:23.873627 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.870954    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 16:01:23.874137 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.870960    2572 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 26 16:01:23.874137 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871410    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 16:01:23.874137 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871415    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 16:01:23.874137 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871418    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 16:01:23.874137 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871421    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 16:01:23.874137 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871424    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 16:01:23.874137 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871427    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 16:01:23.874137 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871430    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 16:01:23.874137 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871433    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 16:01:23.874137 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871436    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 16:01:23.874137 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871439    2572 feature_gate.go:328] unrecognized feature gate: Example
May 26 16:01:23.874137 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871442    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 16:01:23.874137 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871444    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 16:01:23.874137 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871447    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 16:01:23.874137 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871450    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 16:01:23.874507 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871453    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 16:01:23.874507 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871456    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 16:01:23.874507 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871458    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 16:01:23.874507 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871461    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 16:01:23.874507 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871463    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 16:01:23.874507 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871466    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 16:01:23.874507 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871469    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 16:01:23.874507 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871472    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 16:01:23.874507 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871476    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 16:01:23.874507 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871478    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 16:01:23.874507 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871481    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 16:01:23.874507 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871484    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 16:01:23.874507 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871486    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 16:01:23.874507 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871489    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 16:01:23.874507 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871492    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 16:01:23.874507 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871494    2572 feature_gate.go:328] unrecognized feature gate: Example2
May 26 16:01:23.874507 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871497    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 16:01:23.874507 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871499    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 16:01:23.874507 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871502    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 16:01:23.875041 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871505    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 16:01:23.875041 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871507    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 16:01:23.875041 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871510    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 16:01:23.875041 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871513    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 16:01:23.875041 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871515    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 16:01:23.875041 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871518    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 16:01:23.875041 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871521    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 16:01:23.875041 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871524    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 16:01:23.875041 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871527    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 16:01:23.875041 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871529    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 16:01:23.875041 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871532    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 16:01:23.875041 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871534    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 16:01:23.875041 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871537    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 16:01:23.875041 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871539    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 16:01:23.875041 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871542    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 16:01:23.875041 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871544    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 16:01:23.875041 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871547    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 16:01:23.875041 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871549    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 16:01:23.875041 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871552    2572 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 16:01:23.875041 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871554    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 16:01:23.875538 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871557    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 16:01:23.875538 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871559    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 16:01:23.875538 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871562    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 16:01:23.875538 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871565    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 16:01:23.875538 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871569    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 16:01:23.875538 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871572    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 16:01:23.875538 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871575    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 16:01:23.875538 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871579    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 16:01:23.875538 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871582    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 16:01:23.875538 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871585    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 16:01:23.875538 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871587    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 16:01:23.875538 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871590    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 16:01:23.875538 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871592    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 16:01:23.875538 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871594    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 16:01:23.875538 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871597    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 16:01:23.875538 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871612    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 16:01:23.875538 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871615    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 16:01:23.875538 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871617    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 16:01:23.875538 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871620    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 16:01:23.875538 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871623    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 16:01:23.876171 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871625    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 16:01:23.876171 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871628    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 16:01:23.876171 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871630    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 16:01:23.876171 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871633    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 16:01:23.876171 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871635    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 16:01:23.876171 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871638    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 16:01:23.876171 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871640    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 16:01:23.876171 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871643    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 16:01:23.876171 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871645    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 16:01:23.876171 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871648    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 16:01:23.876171 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871650    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 16:01:23.876171 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871653    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 16:01:23.876171 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:23.871655    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 16:01:23.876171 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.871660    2572 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 26 16:01:23.876171 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.872461    2572 server.go:962] "Client rotation is on, will bootstrap in background"
May 26 16:01:23.876539 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.874573    2572 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
May 26 16:01:23.876539 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.875518    2572 server.go:1019] "Starting client certificate rotation"
May 26 16:01:23.876539 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.875640    2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 26 16:01:23.877367 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.877354    2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 26 16:01:23.900657 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.900635    2572 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 26 16:01:23.903135 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.902413    2572 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 26 16:01:23.923153 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.923126    2572 log.go:25] "Validated CRI v1 runtime API"
May 26 16:01:23.928146 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.928127    2572 log.go:25] "Validated CRI v1 image API"
May 26 16:01:23.929338 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.929323    2572 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
May 26 16:01:23.933160 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.933137    2572 fs.go:135] Filesystem UUIDs: map[15cc08f6-fcea-4cb0-ad64-4982d6725f99:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 c39b102a-8836-4cc7-8421-df27fcbe5d40:/dev/nvme0n1p3]
May 26 16:01:23.933216 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.933160    2572 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
May 26 16:01:23.935307 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.935289    2572 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 26 16:01:23.937532 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.937410    2572 manager.go:217] Machine: {Timestamp:2026-05-26 16:01:23.936905914 +0000 UTC m=+0.412633255 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3127737 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28beb2c77af080e7910ef0bde63e0d SystemUUID:ec28beb2-c77a-f080-e791-0ef0bde63e0d BootID:cc45c4fe-cf15-4a3d-a9a1-e7f38eed0ede Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:a6:26:4e:69:29 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:a6:26:4e:69:29 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a6:09:15:e9:3f:b6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
May 26 16:01:23.938211 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.938199    2572 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
May 26 16:01:23.938306 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.938293    2572 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
May 26 16:01:23.939983 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.939958    2572 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
May 26 16:01:23.940125 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.939987    2572 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-83.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
May 26 16:01:23.940176 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.940135    2572 topology_manager.go:138] "Creating topology manager with none policy"
May 26 16:01:23.940176 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.940143    2572 container_manager_linux.go:306] "Creating device plugin manager"
May 26 16:01:23.940176 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.940151    2572 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
May 26 16:01:23.940907 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.940895    2572 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
May 26 16:01:23.943139 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.943127    2572 state_mem.go:36] "Initialized new in-memory state store"
May 26 16:01:23.943252 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.943243    2572 server.go:1267] "Using root directory" path="/var/lib/kubelet"
May 26 16:01:23.945736 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.945723    2572 kubelet.go:491] "Attempting to sync node with API server"
May 26 16:01:23.945827 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.945743    2572 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
May 26 16:01:23.945827 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.945756    2572 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
May 26 16:01:23.945827 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.945766    2572 kubelet.go:397] "Adding apiserver pod source"
May 26 16:01:23.945827 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.945775    2572 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
May 26 16:01:23.946872 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.946860    2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 26 16:01:23.946915 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.946879    2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 26 16:01:23.949930 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.949906    2572 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1"
May 26 16:01:23.951722 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.951709    2572 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
May 26 16:01:23.952541 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.952529    2572 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
May 26 16:01:23.952593 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.952545    2572 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
May 26 16:01:23.952593 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.952552    2572 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
May 26 16:01:23.952593 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.952557    2572 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
May 26 16:01:23.952593 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.952565    2572 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
May 26 16:01:23.952593 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.952571    2572 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
May 26 16:01:23.952593 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.952576    2572 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
May 26 16:01:23.952593 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.952582    2572 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
May 26 16:01:23.952593 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.952588    2572 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
May 26 16:01:23.952593 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.952594    2572 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
May 26 16:01:23.952846 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.952617    2572 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
May 26 16:01:23.952897 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.952887    2572 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
May 26 16:01:23.953647 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.953634    2572 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
May 26 16:01:23.953647 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.953648    2572 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
May 26 16:01:23.958502 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.958223    2572 watchdog_linux.go:99] "Systemd watchdog is not enabled"
May 26 16:01:23.958667 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.958647    2572 server.go:1295] "Started kubelet"
May 26 16:01:23.959547 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.959383    2572 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
May 26 16:01:23.959547 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.959456    2572 server_v1.go:47] "podresources" method="list" useActivePods=true
May 26 16:01:23.959663 ip-10-0-130-83 systemd[1]: Started Kubernetes Kubelet.
May 26 16:01:23.960732 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.959814    2572 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
May 26 16:01:23.962478 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.962459    2572 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-83.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
May 26 16:01:23.962585 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:23.962510    2572 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
May 26 16:01:23.962585 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:23.962513    2572 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-83.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
May 26 16:01:23.962807 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.962792    2572 server.go:317] "Adding debug handlers to kubelet server"
May 26 16:01:23.962878 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.962838    2572 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
May 26 16:01:23.968640 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.968620    2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
May 26 16:01:23.969137 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.969126    2572 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
May 26 16:01:23.969249 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.969235    2572 volume_manager.go:295] "The desired_state_of_world populator starts"
May 26 16:01:23.969309 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.969250    2572 volume_manager.go:297] "Starting Kubelet Volume Manager"
May 26 16:01:23.969383 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.969364    2572 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
May 26 16:01:23.969494 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.969445    2572 reconstruct.go:97] "Volume reconstruction finished"
May 26 16:01:23.969494 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.969455    2572 reconciler.go:26] "Reconciler: start to sync state"
May 26 16:01:23.969708 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:23.969687    2572 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
May 26 16:01:23.969815 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.969795    2572 factory.go:55] Registering systemd factory
May 26 16:01:23.969869 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.969821    2572 factory.go:223] Registration of the systemd container factory successfully
May 26 16:01:23.969957 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:23.969938    2572 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-83.ec2.internal\" not found"
May 26 16:01:23.970058 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.970049    2572 factory.go:153] Registering CRI-O factory
May 26 16:01:23.970106 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.970063    2572 factory.go:223] Registration of the crio container factory successfully
May 26 16:01:23.970163 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.970113    2572 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
May 26 16:01:23.970217 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.970164    2572 factory.go:103] Registering Raw factory
May 26 16:01:23.970217 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.970181    2572 manager.go:1196] Started watching for new ooms in manager
May 26 16:01:23.970942 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.970925    2572 manager.go:319] Starting recovery of all containers
May 26 16:01:23.973031 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:23.970817    2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-83.ec2.internal.18b328eed3901640  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-83.ec2.internal,UID:ip-10-0-130-83.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-83.ec2.internal,},FirstTimestamp:2026-05-26 16:01:23.958240832 +0000 UTC m=+0.433968176,LastTimestamp:2026-05-26 16:01:23.958240832 +0000 UTC m=+0.433968176,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-83.ec2.internal,}"
May 26 16:01:23.975509 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:23.975478    2572 reflector.go:200] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
May 26 16:01:23.975628 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:23.975576    2572 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-83.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
May 26 16:01:23.983415 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.983262    2572 manager.go:324] Recovery completed
May 26 16:01:23.984950 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.984931    2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bstf5"
May 26 16:01:23.987678 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.987665    2572 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 16:01:23.989483 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.989469    2572 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-83.ec2.internal" event="NodeHasSufficientMemory"
May 26 16:01:23.989545 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.989496    2572 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-83.ec2.internal" event="NodeHasNoDiskPressure"
May 26 16:01:23.989545 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.989520    2572 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-83.ec2.internal" event="NodeHasSufficientPID"
May 26 16:01:23.990794 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.990775    2572 cpu_manager.go:222] "Starting CPU manager" policy="none"
May 26 16:01:23.990794 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.990794    2572 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
May 26 16:01:23.990925 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.990813    2572 state_mem.go:36] "Initialized new in-memory state store"
May 26 16:01:23.993130 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:23.993062    2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-83.ec2.internal.18b328eed56cd109  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-83.ec2.internal,UID:ip-10-0-130-83.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-83.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-83.ec2.internal,},FirstTimestamp:2026-05-26 16:01:23.989483785 +0000 UTC m=+0.465211125,LastTimestamp:2026-05-26 16:01:23.989483785 +0000 UTC m=+0.465211125,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-83.ec2.internal,}"
May 26 16:01:23.993377 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.993360    2572 policy_none.go:49] "None policy: Start"
May 26 16:01:23.993377 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.993380    2572 memory_manager.go:186] "Starting memorymanager" policy="None"
May 26 16:01:23.993479 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.993391    2572 state_mem.go:35] "Initializing new in-memory state store"
May 26 16:01:23.996734 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:23.996715    2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bstf5"
May 26 16:01:24.031346 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:24.031327    2572 manager.go:341] "Starting Device Plugin manager"
May 26 16:01:24.041027 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:24.031387    2572 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
May 26 16:01:24.041027 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:24.031401    2572 server.go:85] "Starting device plugin registration server"
May 26 16:01:24.041027 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:24.031685    2572 eviction_manager.go:189] "Eviction manager: starting control loop"
May 26 16:01:24.041027 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:24.031696    2572 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
May 26 16:01:24.041027 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:24.031788    2572 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
May 26 16:01:24.041027 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:24.031870    2572 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
May 26 16:01:24.041027 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:24.031880    2572 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
May 26 16:01:24.041027 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:24.032503    2572 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
May 26 16:01:24.041027 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:24.032544    2572 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-83.ec2.internal\" not found"
May 26 16:01:24.073345 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:24.073304    2572 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
May 26 16:01:24.074484 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:24.074467    2572 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
May 26 16:01:24.074582 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:24.074488    2572 status_manager.go:230] "Starting to sync pod status with apiserver"
May 26 16:01:24.074582 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:24.074508    2572 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
May 26 16:01:24.074582 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:24.074515    2572 kubelet.go:2451] "Starting kubelet main sync loop"
May 26 16:01:24.074582 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:24.074555    2572 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
May 26 16:01:24.077116 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:24.077094    2572 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 16:01:24.132189 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:24.132112    2572 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 16:01:24.133082 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:24.133062    2572 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-83.ec2.internal" event="NodeHasSufficientMemory"
May 26 16:01:24.133203 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:24.133095    2572 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-83.ec2.internal" event="NodeHasNoDiskPressure"
May 26 16:01:24.133203 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:24.133106    2572 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-83.ec2.internal" event="NodeHasSufficientPID"
May 26 16:01:24.133203 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:24.133128    2572 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-83.ec2.internal"
May 26 16:01:24.143657 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:24.143638    2572 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-83.ec2.internal"
May 26 16:01:24.143774 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:24.143666    2572 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-83.ec2.internal\": node \"ip-10-0-130-83.ec2.internal\" not found"
May 26 16:01:24.159639 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:24.159618    2572 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-83.ec2.internal\" not found"
May 26 16:01:24.175420 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:24.175396    2572 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-83.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-83.ec2.internal"]
May 26 16:01:24.175498 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:24.175462    2572 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 16:01:24.176347 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:24.176332    2572 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-83.ec2.internal" event="NodeHasSufficientMemory"
May 26 16:01:24.176439 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:24.176364    2572 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-83.ec2.internal" event="NodeHasNoDiskPressure"
May 26 16:01:24.176439 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:24.176377    2572 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-83.ec2.internal" event="NodeHasSufficientPID"
May 26 16:01:24.177826 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:24.177803    2572 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 16:01:24.177968 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:24.177953    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-83.ec2.internal"
May 26 16:01:24.178025 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:24.177983    2572 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 16:01:24.178501 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:24.178471    2572 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-83.ec2.internal" event="NodeHasSufficientMemory"
May 26 16:01:24.178624 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:24.178509    2572 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-83.ec2.internal" event="NodeHasSufficientMemory"
May 26 16:01:24.178624 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:24.178534    2572 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-83.ec2.internal" event="NodeHasNoDiskPressure"
May 26 16:01:24.178624 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:24.178547    2572 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-83.ec2.internal" event="NodeHasSufficientPID"
May 26 16:01:24.178624 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:24.178515    2572 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-83.ec2.internal" event="NodeHasNoDiskPressure"
May 26 16:01:24.178624 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:24.178596    2572 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-83.ec2.internal" event="NodeHasSufficientPID"
May 26 16:01:24.179913 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:24.179896    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-83.ec2.internal"
May 26 16:01:24.179995 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:24.179921    2572 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 16:01:24.180616 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:24.180589    2572 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-83.ec2.internal" event="NodeHasSufficientMemory"
May 26 16:01:24.180691 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:24.180627    2572 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-83.ec2.internal" event="NodeHasNoDiskPressure"
May 26 16:01:24.180691 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:24.180637    2572 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-83.ec2.internal" event="NodeHasSufficientPID"
May 26 16:01:24.207429 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:24.207405    2572 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-83.ec2.internal\" not found" node="ip-10-0-130-83.ec2.internal"
May 26 16:01:24.211939 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:24.211921    2572 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-83.ec2.internal\" not found" node="ip-10-0-130-83.ec2.internal"
May 26 16:01:24.260083 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:24.260059    2572 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-83.ec2.internal\" not found"
May 26 16:01:24.270674 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:24.270653    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/79549bd5c5e41ee1a69fee2a0b6c1223-config\") pod \"kube-apiserver-proxy-ip-10-0-130-83.ec2.internal\" (UID: \"79549bd5c5e41ee1a69fee2a0b6c1223\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-83.ec2.internal"
May 26 16:01:24.270738 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:24.270681    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/75eda3ff71f275b7dd3449d695a0e806-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-83.ec2.internal\" (UID: \"75eda3ff71f275b7dd3449d695a0e806\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-83.ec2.internal"
May 26 16:01:24.270738 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:24.270702    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/75eda3ff71f275b7dd3449d695a0e806-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-83.ec2.internal\" (UID: \"75eda3ff71f275b7dd3449d695a0e806\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-83.ec2.internal"
May 26 16:01:24.360812 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:24.360784    2572 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-83.ec2.internal\" not found"
May 26 16:01:24.371078 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:24.371048    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/79549bd5c5e41ee1a69fee2a0b6c1223-config\") pod \"kube-apiserver-proxy-ip-10-0-130-83.ec2.internal\" (UID: \"79549bd5c5e41ee1a69fee2a0b6c1223\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-83.ec2.internal"
May 26 16:01:24.371078 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:24.371084    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/75eda3ff71f275b7dd3449d695a0e806-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-83.ec2.internal\" (UID: \"75eda3ff71f275b7dd3449d695a0e806\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-83.ec2.internal"
May 26 16:01:24.371183 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:24.371104    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/75eda3ff71f275b7dd3449d695a0e806-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-83.ec2.internal\" (UID: \"75eda3ff71f275b7dd3449d695a0e806\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-83.ec2.internal"
May 26 16:01:24.371183 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:24.371165    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/75eda3ff71f275b7dd3449d695a0e806-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-83.ec2.internal\" (UID: \"75eda3ff71f275b7dd3449d695a0e806\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-83.ec2.internal"
May 26 16:01:24.371183 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:24.371168    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/79549bd5c5e41ee1a69fee2a0b6c1223-config\") pod \"kube-apiserver-proxy-ip-10-0-130-83.ec2.internal\" (UID: \"79549bd5c5e41ee1a69fee2a0b6c1223\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-83.ec2.internal"
May 26 16:01:24.371276 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:24.371200    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/75eda3ff71f275b7dd3449d695a0e806-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-83.ec2.internal\" (UID: \"75eda3ff71f275b7dd3449d695a0e806\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-83.ec2.internal"
May 26 16:01:24.461511 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:24.461442    2572 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-83.ec2.internal\" not found"
May 26 16:01:24.510978 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:24.510939    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-83.ec2.internal"
May 26 16:01:24.514732 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:24.514716    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-83.ec2.internal"
May 26 16:01:24.561597 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:24.561563    2572 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-83.ec2.internal\" not found"
May 26 16:01:24.662065 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:24.662031    2572 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-83.ec2.internal\" not found"
May 26 16:01:24.762575 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:24.762491    2572 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-83.ec2.internal\" not found"
May 26 16:01:24.863093 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:24.863064    2572 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-83.ec2.internal\" not found"
May 26 16:01:24.876510 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:24.876484    2572 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
May 26 16:01:24.876674 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:24.876658    2572 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 26 16:01:24.963189 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:24.963157    2572 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-83.ec2.internal\" not found"
May 26 16:01:24.969087 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:24.969064    2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
May 26 16:01:24.983650 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:24.983628    2572 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 26 16:01:24.992406 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:24.992381    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79549bd5c5e41ee1a69fee2a0b6c1223.slice/crio-8bfd2d477c1402945a5be6963361c8c8c1a696fdddafde4bd347b4e279760a62 WatchSource:0}: Error finding container 8bfd2d477c1402945a5be6963361c8c8c1a696fdddafde4bd347b4e279760a62: Status 404 returned error can't find the container with id 8bfd2d477c1402945a5be6963361c8c8c1a696fdddafde4bd347b4e279760a62
May 26 16:01:24.992570 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:24.992553    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75eda3ff71f275b7dd3449d695a0e806.slice/crio-d8a2bab0b4dbcefc5cf71f2bdf43b1d88eb9ba89accdd9d9a0ea99ca5b0d2087 WatchSource:0}: Error finding container d8a2bab0b4dbcefc5cf71f2bdf43b1d88eb9ba89accdd9d9a0ea99ca5b0d2087: Status 404 returned error can't find the container with id d8a2bab0b4dbcefc5cf71f2bdf43b1d88eb9ba89accdd9d9a0ea99ca5b0d2087
May 26 16:01:24.996433 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:24.996415    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 16:01:24.999188 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:24.999163    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-25 15:56:23 +0000 UTC" deadline="2028-01-11 19:16:53.344281357 +0000 UTC"
May 26 16:01:24.999188 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:24.999188    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14283h15m28.345096008s"
May 26 16:01:25.006413 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.006393    2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-rrfht"
May 26 16:01:25.014679 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.014592    2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rrfht"
May 26 16:01:25.063849 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:25.063818    2572 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-83.ec2.internal\" not found"
May 26 16:01:25.078160 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.078105    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-83.ec2.internal" event={"ID":"75eda3ff71f275b7dd3449d695a0e806","Type":"ContainerStarted","Data":"d8a2bab0b4dbcefc5cf71f2bdf43b1d88eb9ba89accdd9d9a0ea99ca5b0d2087"}
May 26 16:01:25.080939 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.080914    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-83.ec2.internal" event={"ID":"79549bd5c5e41ee1a69fee2a0b6c1223","Type":"ContainerStarted","Data":"8bfd2d477c1402945a5be6963361c8c8c1a696fdddafde4bd347b4e279760a62"}
May 26 16:01:25.164204 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:25.164175    2572 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-83.ec2.internal\" not found"
May 26 16:01:25.264761 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:25.264665    2572 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-83.ec2.internal\" not found"
May 26 16:01:25.293029 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.293001    2572 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 16:01:25.318524 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.318496    2572 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 16:01:25.370109 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.370075    2572 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-83.ec2.internal"
May 26 16:01:25.385032 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.385001    2572 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 26 16:01:25.386142 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.386114    2572 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-83.ec2.internal"
May 26 16:01:25.394142 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.394113    2572 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 26 16:01:25.495092 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.495066    2572 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 16:01:25.947382 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.947343    2572 apiserver.go:52] "Watching apiserver"
May 26 16:01:25.952228 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.952189    2572 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 16:01:25.953534 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.953508    2572 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
May 26 16:01:25.953937 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.953915    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-83.ec2.internal","openshift-multus/multus-additional-cni-plugins-mjznd","openshift-multus/multus-wk2zv","kube-system/konnectivity-agent-tbphl","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64xz7","openshift-cluster-node-tuning-operator/tuned-8hhhx","openshift-image-registry/node-ca-qdk28","openshift-multus/network-metrics-daemon-qpzqg","openshift-network-diagnostics/network-check-target-rvlkc","openshift-network-operator/iptables-alerter-jlndg","openshift-ovn-kubernetes/ovnkube-node-l2lg5","kube-system/kube-apiserver-proxy-ip-10-0-130-83.ec2.internal","openshift-dns/node-resolver-lvfvd"]
May 26 16:01:25.955511 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.955492    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qdk28"
May 26 16:01:25.956626 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.956593    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mjznd"
May 26 16:01:25.957379 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.957361    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zpkcw\""
May 26 16:01:25.957516 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.957498    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
May 26 16:01:25.957832 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.957805    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
May 26 16:01:25.957924 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.957852    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wk2zv"
May 26 16:01:25.958970 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.958939    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-848cw\""
May 26 16:01:25.959068 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.958968    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
May 26 16:01:25.959068 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.958989    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
May 26 16:01:25.959249 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.959228    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
May 26 16:01:25.959345 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.959324    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tbphl"
May 26 16:01:25.959402 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.959234    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
May 26 16:01:25.959734 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.959715    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-rb2nk\""
May 26 16:01:25.959818 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.959753    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
May 26 16:01:25.961120 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.960293    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
May 26 16:01:25.961120 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.960311    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
May 26 16:01:25.961660 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.961423    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
May 26 16:01:25.961660 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.961521    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-lx28l\""
May 26 16:01:25.962729 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.961799    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
May 26 16:01:25.962964 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.962946    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64xz7"
May 26 16:01:25.964333 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.964292    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8hhhx"
May 26 16:01:25.964864 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.964786    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
May 26 16:01:25.964864 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.964786    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
May 26 16:01:25.964864 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.964811    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8rxpt\""
May 26 16:01:25.965108 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.965082    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
May 26 16:01:25.965796 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.965782    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qpzqg"
May 26 16:01:25.965887 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:25.965868    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qpzqg" podUID="8828138d-e7ed-4d5c-ba76-26429679c7b1"
May 26 16:01:25.966231 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.966210    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
May 26 16:01:25.966325 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.966250    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
May 26 16:01:25.966325 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.966316    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2llk5\""
May 26 16:01:25.967283 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.967263    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rvlkc"
May 26 16:01:25.967371 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:25.967329    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rvlkc" podUID="8f54a9e9-f1b3-4073-bf4e-fef6f31dc622"
May 26 16:01:25.968364 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.968345    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jlndg"
May 26 16:01:25.969968 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.969951    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:25.970520 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.970503    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
May 26 16:01:25.970634 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.970535    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
May 26 16:01:25.970634 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.970559    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
May 26 16:01:25.970634 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.970561    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tl48k\""
May 26 16:01:25.971692 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.971670    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lvfvd"
May 26 16:01:25.972339 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.972322    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
May 26 16:01:25.972451 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.972430    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9ncvl\""
May 26 16:01:25.972620 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.972592    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
May 26 16:01:25.973411 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.973395    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
May 26 16:01:25.973458 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.973420    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
May 26 16:01:25.973504 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.973452    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
May 26 16:01:25.974026 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.974010    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
May 26 16:01:25.974342 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.974322    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
May 26 16:01:25.974508 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.974494    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
May 26 16:01:25.974708 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.974694    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pbr6g\""
May 26 16:01:25.980279 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.980261    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/32dd34b2-8588-46f9-b744-7138c2ac7025-konnectivity-ca\") pod \"konnectivity-agent-tbphl\" (UID: \"32dd34b2-8588-46f9-b744-7138c2ac7025\") " pod="kube-system/konnectivity-agent-tbphl"
May 26 16:01:25.980378 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.980291    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9823062-c35f-4aca-b363-912ec43ecb40-host-kubelet\") pod \"ovnkube-node-l2lg5\" (UID: \"f9823062-c35f-4aca-b363-912ec43ecb40\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:25.980378 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.980309    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9823062-c35f-4aca-b363-912ec43ecb40-var-lib-openvswitch\") pod \"ovnkube-node-l2lg5\" (UID: \"f9823062-c35f-4aca-b363-912ec43ecb40\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:25.980378 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.980332    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9823062-c35f-4aca-b363-912ec43ecb40-host-run-ovn-kubernetes\") pod \"ovnkube-node-l2lg5\" (UID: \"f9823062-c35f-4aca-b363-912ec43ecb40\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:25.980378 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.980367    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5ad6cab9-b922-456e-ad28-95f40ce6692c-cnibin\") pod \"multus-wk2zv\" (UID: \"5ad6cab9-b922-456e-ad28-95f40ce6692c\") " pod="openshift-multus/multus-wk2zv"
May 26 16:01:25.980555 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.980394    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/744c275a-8d87-4882-8e2e-73254829ca4a-tmp\") pod \"tuned-8hhhx\" (UID: \"744c275a-8d87-4882-8e2e-73254829ca4a\") " pod="openshift-cluster-node-tuning-operator/tuned-8hhhx"
May 26 16:01:25.980555 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.980416    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8vcn\" (UniqueName: \"kubernetes.io/projected/744c275a-8d87-4882-8e2e-73254829ca4a-kube-api-access-f8vcn\") pod \"tuned-8hhhx\" (UID: \"744c275a-8d87-4882-8e2e-73254829ca4a\") " pod="openshift-cluster-node-tuning-operator/tuned-8hhhx"
May 26 16:01:25.980555 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.980443    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbgsb\" (UniqueName: \"kubernetes.io/projected/8f54a9e9-f1b3-4073-bf4e-fef6f31dc622-kube-api-access-lbgsb\") pod \"network-check-target-rvlkc\" (UID: \"8f54a9e9-f1b3-4073-bf4e-fef6f31dc622\") " pod="openshift-network-diagnostics/network-check-target-rvlkc"
May 26 16:01:25.980555 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.980469    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ef455917-b870-4d4a-b021-45a76d22734d-socket-dir\") pod \"aws-ebs-csi-driver-node-64xz7\" (UID: \"ef455917-b870-4d4a-b021-45a76d22734d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64xz7"
May 26 16:01:25.980555 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.980493    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9823062-c35f-4aca-b363-912ec43ecb40-etc-openvswitch\") pod \"ovnkube-node-l2lg5\" (UID: \"f9823062-c35f-4aca-b363-912ec43ecb40\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:25.980555 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.980514    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ad6cab9-b922-456e-ad28-95f40ce6692c-system-cni-dir\") pod \"multus-wk2zv\" (UID: \"5ad6cab9-b922-456e-ad28-95f40ce6692c\") " pod="openshift-multus/multus-wk2zv"
May 26 16:01:25.980555 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.980545    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ad6cab9-b922-456e-ad28-95f40ce6692c-host-var-lib-kubelet\") pod \"multus-wk2zv\" (UID: \"5ad6cab9-b922-456e-ad28-95f40ce6692c\") " pod="openshift-multus/multus-wk2zv"
May 26 16:01:25.980824 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.980565    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5ad6cab9-b922-456e-ad28-95f40ce6692c-multus-daemon-config\") pod \"multus-wk2zv\" (UID: \"5ad6cab9-b922-456e-ad28-95f40ce6692c\") " pod="openshift-multus/multus-wk2zv"
May 26 16:01:25.980824 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.980579    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/744c275a-8d87-4882-8e2e-73254829ca4a-etc-modprobe-d\") pod \"tuned-8hhhx\" (UID: \"744c275a-8d87-4882-8e2e-73254829ca4a\") " pod="openshift-cluster-node-tuning-operator/tuned-8hhhx"
May 26 16:01:25.980824 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.980592    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/744c275a-8d87-4882-8e2e-73254829ca4a-host\") pod \"tuned-8hhhx\" (UID: \"744c275a-8d87-4882-8e2e-73254829ca4a\") " pod="openshift-cluster-node-tuning-operator/tuned-8hhhx"
May 26 16:01:25.980824 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.980628    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/744c275a-8d87-4882-8e2e-73254829ca4a-etc-tuned\") pod \"tuned-8hhhx\" (UID: \"744c275a-8d87-4882-8e2e-73254829ca4a\") " pod="openshift-cluster-node-tuning-operator/tuned-8hhhx"
May 26 16:01:25.980824 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.980648    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ef455917-b870-4d4a-b021-45a76d22734d-sys-fs\") pod \"aws-ebs-csi-driver-node-64xz7\" (UID: \"ef455917-b870-4d4a-b021-45a76d22734d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64xz7"
May 26 16:01:25.980824 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.980668    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f9823062-c35f-4aca-b363-912ec43ecb40-host-slash\") pod \"ovnkube-node-l2lg5\" (UID: \"f9823062-c35f-4aca-b363-912ec43ecb40\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:25.980824 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.980690    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f9823062-c35f-4aca-b363-912ec43ecb40-run-ovn\") pod \"ovnkube-node-l2lg5\" (UID: \"f9823062-c35f-4aca-b363-912ec43ecb40\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:25.980824 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.980714    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84fpx\" (UniqueName: \"kubernetes.io/projected/f9823062-c35f-4aca-b363-912ec43ecb40-kube-api-access-84fpx\") pod \"ovnkube-node-l2lg5\" (UID: \"f9823062-c35f-4aca-b363-912ec43ecb40\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:25.980824 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.980767    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5ad6cab9-b922-456e-ad28-95f40ce6692c-os-release\") pod \"multus-wk2zv\" (UID: \"5ad6cab9-b922-456e-ad28-95f40ce6692c\") " pod="openshift-multus/multus-wk2zv"
May 26 16:01:25.980824 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.980808    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/23cb5db1-ddc3-46a0-bc9d-3133631ffc58-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mjznd\" (UID: \"23cb5db1-ddc3-46a0-bc9d-3133631ffc58\") " pod="openshift-multus/multus-additional-cni-plugins-mjznd"
May 26 16:01:25.981209 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.980837    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02c134e5-0f87-48c2-80ed-427d242f1435-tmp-dir\") pod \"node-resolver-lvfvd\" (UID: \"02c134e5-0f87-48c2-80ed-427d242f1435\") " pod="openshift-dns/node-resolver-lvfvd"
May 26 16:01:25.981209 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.980862    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5ad6cab9-b922-456e-ad28-95f40ce6692c-host-run-multus-certs\") pod \"multus-wk2zv\" (UID: \"5ad6cab9-b922-456e-ad28-95f40ce6692c\") " pod="openshift-multus/multus-wk2zv"
May 26 16:01:25.981209 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.980884    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/10ea1129-78a4-4d29-ab50-50b69b6d30c1-host\") pod \"node-ca-qdk28\" (UID: \"10ea1129-78a4-4d29-ab50-50b69b6d30c1\") " pod="openshift-image-registry/node-ca-qdk28"
May 26 16:01:25.981209 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.980908    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/23cb5db1-ddc3-46a0-bc9d-3133631ffc58-system-cni-dir\") pod \"multus-additional-cni-plugins-mjznd\" (UID: \"23cb5db1-ddc3-46a0-bc9d-3133631ffc58\") " pod="openshift-multus/multus-additional-cni-plugins-mjznd"
May 26 16:01:25.981209 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.980933    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f9823062-c35f-4aca-b363-912ec43ecb40-ovn-node-metrics-cert\") pod \"ovnkube-node-l2lg5\" (UID: \"f9823062-c35f-4aca-b363-912ec43ecb40\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:25.981209 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.980967    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5ad6cab9-b922-456e-ad28-95f40ce6692c-host-var-lib-cni-bin\") pod \"multus-wk2zv\" (UID: \"5ad6cab9-b922-456e-ad28-95f40ce6692c\") " pod="openshift-multus/multus-wk2zv"
May 26 16:01:25.981209 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.980991    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/744c275a-8d87-4882-8e2e-73254829ca4a-etc-sysctl-d\") pod \"tuned-8hhhx\" (UID: \"744c275a-8d87-4882-8e2e-73254829ca4a\") " pod="openshift-cluster-node-tuning-operator/tuned-8hhhx"
May 26 16:01:25.981209 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.981017    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/744c275a-8d87-4882-8e2e-73254829ca4a-etc-sysctl-conf\") pod \"tuned-8hhhx\" (UID: \"744c275a-8d87-4882-8e2e-73254829ca4a\") " pod="openshift-cluster-node-tuning-operator/tuned-8hhhx"
May 26 16:01:25.981209 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.981068    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/23cb5db1-ddc3-46a0-bc9d-3133631ffc58-os-release\") pod \"multus-additional-cni-plugins-mjznd\" (UID: \"23cb5db1-ddc3-46a0-bc9d-3133631ffc58\") " pod="openshift-multus/multus-additional-cni-plugins-mjznd"
May 26 16:01:25.981209 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.981110    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f9823062-c35f-4aca-b363-912ec43ecb40-systemd-units\") pod \"ovnkube-node-l2lg5\" (UID: \"f9823062-c35f-4aca-b363-912ec43ecb40\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:25.981209 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.981160    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f9823062-c35f-4aca-b363-912ec43ecb40-run-systemd\") pod \"ovnkube-node-l2lg5\" (UID: \"f9823062-c35f-4aca-b363-912ec43ecb40\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:25.981209 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.981199    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ad6cab9-b922-456e-ad28-95f40ce6692c-multus-cni-dir\") pod \"multus-wk2zv\" (UID: \"5ad6cab9-b922-456e-ad28-95f40ce6692c\") " pod="openshift-multus/multus-wk2zv"
May 26 16:01:25.981557 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.981219    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5ad6cab9-b922-456e-ad28-95f40ce6692c-multus-conf-dir\") pod \"multus-wk2zv\" (UID: \"5ad6cab9-b922-456e-ad28-95f40ce6692c\") " pod="openshift-multus/multus-wk2zv"
May 26 16:01:25.981557 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.981234    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/744c275a-8d87-4882-8e2e-73254829ca4a-var-lib-kubelet\") pod \"tuned-8hhhx\" (UID: \"744c275a-8d87-4882-8e2e-73254829ca4a\") " pod="openshift-cluster-node-tuning-operator/tuned-8hhhx"
May 26 16:01:25.981557 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.981260    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c54d2f93-a58c-4f73-966d-2584dc7015e8-host-slash\") pod \"iptables-alerter-jlndg\" (UID: \"c54d2f93-a58c-4f73-966d-2584dc7015e8\") " pod="openshift-network-operator/iptables-alerter-jlndg"
May 26 16:01:25.981557 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.981285    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltn4t\" (UniqueName: \"kubernetes.io/projected/10ea1129-78a4-4d29-ab50-50b69b6d30c1-kube-api-access-ltn4t\") pod \"node-ca-qdk28\" (UID: \"10ea1129-78a4-4d29-ab50-50b69b6d30c1\") " pod="openshift-image-registry/node-ca-qdk28"
May 26 16:01:25.981557 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.981312    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f9823062-c35f-4aca-b363-912ec43ecb40-env-overrides\") pod \"ovnkube-node-l2lg5\" (UID: \"f9823062-c35f-4aca-b363-912ec43ecb40\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:25.981557 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.981333    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frmct\" (UniqueName: \"kubernetes.io/projected/ef455917-b870-4d4a-b021-45a76d22734d-kube-api-access-frmct\") pod \"aws-ebs-csi-driver-node-64xz7\" (UID: \"ef455917-b870-4d4a-b021-45a76d22734d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64xz7"
May 26 16:01:25.981557 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.981369    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f9823062-c35f-4aca-b363-912ec43ecb40-node-log\") pod \"ovnkube-node-l2lg5\" (UID: \"f9823062-c35f-4aca-b363-912ec43ecb40\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:25.981557 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.981411    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/32dd34b2-8588-46f9-b744-7138c2ac7025-agent-certs\") pod \"konnectivity-agent-tbphl\" (UID: \"32dd34b2-8588-46f9-b744-7138c2ac7025\") " pod="kube-system/konnectivity-agent-tbphl"
May 26 16:01:25.981557 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.981438    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ef455917-b870-4d4a-b021-45a76d22734d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-64xz7\" (UID: \"ef455917-b870-4d4a-b021-45a76d22734d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64xz7"
May 26 16:01:25.981557 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.981461    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ef455917-b870-4d4a-b021-45a76d22734d-device-dir\") pod \"aws-ebs-csi-driver-node-64xz7\" (UID: \"ef455917-b870-4d4a-b021-45a76d22734d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64xz7"
May 26 16:01:25.981557 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.981516    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6rlr\" (UniqueName: \"kubernetes.io/projected/02c134e5-0f87-48c2-80ed-427d242f1435-kube-api-access-x6rlr\") pod \"node-resolver-lvfvd\" (UID: \"02c134e5-0f87-48c2-80ed-427d242f1435\") " pod="openshift-dns/node-resolver-lvfvd"
May 26 16:01:25.981557 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.981539    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5ad6cab9-b922-456e-ad28-95f40ce6692c-host-run-k8s-cni-cncf-io\") pod \"multus-wk2zv\" (UID: \"5ad6cab9-b922-456e-ad28-95f40ce6692c\") " pod="openshift-multus/multus-wk2zv"
May 26 16:01:25.981949 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.981563    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/744c275a-8d87-4882-8e2e-73254829ca4a-etc-kubernetes\") pod \"tuned-8hhhx\" (UID: \"744c275a-8d87-4882-8e2e-73254829ca4a\") " pod="openshift-cluster-node-tuning-operator/tuned-8hhhx"
May 26 16:01:25.981949 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.981588    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c54d2f93-a58c-4f73-966d-2584dc7015e8-iptables-alerter-script\") pod \"iptables-alerter-jlndg\" (UID: \"c54d2f93-a58c-4f73-966d-2584dc7015e8\") " pod="openshift-network-operator/iptables-alerter-jlndg"
May 26 16:01:25.981949 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.981633    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/744c275a-8d87-4882-8e2e-73254829ca4a-etc-sysconfig\") pod \"tuned-8hhhx\" (UID: \"744c275a-8d87-4882-8e2e-73254829ca4a\") " pod="openshift-cluster-node-tuning-operator/tuned-8hhhx"
May 26 16:01:25.981949 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.981662    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f9823062-c35f-4aca-b363-912ec43ecb40-ovnkube-config\") pod \"ovnkube-node-l2lg5\" (UID: \"f9823062-c35f-4aca-b363-912ec43ecb40\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:25.981949 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.981699    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f9823062-c35f-4aca-b363-912ec43ecb40-ovnkube-script-lib\") pod \"ovnkube-node-l2lg5\" (UID: \"f9823062-c35f-4aca-b363-912ec43ecb40\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:25.981949 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.981723    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5ad6cab9-b922-456e-ad28-95f40ce6692c-multus-socket-dir-parent\") pod \"multus-wk2zv\" (UID: \"5ad6cab9-b922-456e-ad28-95f40ce6692c\") " pod="openshift-multus/multus-wk2zv"
May 26 16:01:25.981949 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.981745    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5ad6cab9-b922-456e-ad28-95f40ce6692c-host-run-netns\") pod \"multus-wk2zv\" (UID: \"5ad6cab9-b922-456e-ad28-95f40ce6692c\") " pod="openshift-multus/multus-wk2zv"
May 26 16:01:25.981949 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.981767    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/10ea1129-78a4-4d29-ab50-50b69b6d30c1-serviceca\") pod \"node-ca-qdk28\" (UID: \"10ea1129-78a4-4d29-ab50-50b69b6d30c1\") " pod="openshift-image-registry/node-ca-qdk28"
May 26 16:01:25.981949 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.981790    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/23cb5db1-ddc3-46a0-bc9d-3133631ffc58-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mjznd\" (UID: \"23cb5db1-ddc3-46a0-bc9d-3133631ffc58\") " pod="openshift-multus/multus-additional-cni-plugins-mjznd"
May 26 16:01:25.981949 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.981827    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/23cb5db1-ddc3-46a0-bc9d-3133631ffc58-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mjznd\" (UID: \"23cb5db1-ddc3-46a0-bc9d-3133631ffc58\") " pod="openshift-multus/multus-additional-cni-plugins-mjznd"
May 26 16:01:25.981949 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.981851    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/744c275a-8d87-4882-8e2e-73254829ca4a-etc-systemd\") pod \"tuned-8hhhx\" (UID: \"744c275a-8d87-4882-8e2e-73254829ca4a\") " pod="openshift-cluster-node-tuning-operator/tuned-8hhhx"
May 26 16:01:25.981949 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.981883    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f9823062-c35f-4aca-b363-912ec43ecb40-host-cni-bin\") pod \"ovnkube-node-l2lg5\" (UID: \"f9823062-c35f-4aca-b363-912ec43ecb40\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:25.981949 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.981905    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/02c134e5-0f87-48c2-80ed-427d242f1435-hosts-file\") pod \"node-resolver-lvfvd\" (UID: \"02c134e5-0f87-48c2-80ed-427d242f1435\") " pod="openshift-dns/node-resolver-lvfvd"
May 26 16:01:25.981949 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.981923    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/744c275a-8d87-4882-8e2e-73254829ca4a-sys\") pod \"tuned-8hhhx\" (UID: \"744c275a-8d87-4882-8e2e-73254829ca4a\") " pod="openshift-cluster-node-tuning-operator/tuned-8hhhx"
May 26 16:01:25.981949 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.981937    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8828138d-e7ed-4d5c-ba76-26429679c7b1-metrics-certs\") pod \"network-metrics-daemon-qpzqg\" (UID: \"8828138d-e7ed-4d5c-ba76-26429679c7b1\") " pod="openshift-multus/network-metrics-daemon-qpzqg"
May 26 16:01:25.982504 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.981966    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f9823062-c35f-4aca-b363-912ec43ecb40-host-cni-netd\") pod \"ovnkube-node-l2lg5\" (UID: \"f9823062-c35f-4aca-b363-912ec43ecb40\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:25.982504 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.982004    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9mxv\" (UniqueName: \"kubernetes.io/projected/5ad6cab9-b922-456e-ad28-95f40ce6692c-kube-api-access-x9mxv\") pod \"multus-wk2zv\" (UID: \"5ad6cab9-b922-456e-ad28-95f40ce6692c\") " pod="openshift-multus/multus-wk2zv"
May 26 16:01:25.982504 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.982030    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/744c275a-8d87-4882-8e2e-73254829ca4a-lib-modules\") pod \"tuned-8hhhx\" (UID: \"744c275a-8d87-4882-8e2e-73254829ca4a\") " pod="openshift-cluster-node-tuning-operator/tuned-8hhhx"
May 26 16:01:25.982504 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.982049    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5ad6cab9-b922-456e-ad28-95f40ce6692c-cni-binary-copy\") pod \"multus-wk2zv\" (UID: \"5ad6cab9-b922-456e-ad28-95f40ce6692c\") " pod="openshift-multus/multus-wk2zv"
May 26 16:01:25.982504 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.982069    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ad6cab9-b922-456e-ad28-95f40ce6692c-etc-kubernetes\") pod \"multus-wk2zv\" (UID: \"5ad6cab9-b922-456e-ad28-95f40ce6692c\") " pod="openshift-multus/multus-wk2zv"
May 26 16:01:25.982504 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.982095    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/23cb5db1-ddc3-46a0-bc9d-3133631ffc58-cnibin\") pod \"multus-additional-cni-plugins-mjznd\" (UID: \"23cb5db1-ddc3-46a0-bc9d-3133631ffc58\") " pod="openshift-multus/multus-additional-cni-plugins-mjznd"
May 26 16:01:25.982504 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.982112    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdchq\" (UniqueName: \"kubernetes.io/projected/23cb5db1-ddc3-46a0-bc9d-3133631ffc58-kube-api-access-fdchq\") pod \"multus-additional-cni-plugins-mjznd\" (UID: \"23cb5db1-ddc3-46a0-bc9d-3133631ffc58\") " pod="openshift-multus/multus-additional-cni-plugins-mjznd"
May 26 16:01:25.982504 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.982127    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f9823062-c35f-4aca-b363-912ec43ecb40-host-run-netns\") pod \"ovnkube-node-l2lg5\" (UID: \"f9823062-c35f-4aca-b363-912ec43ecb40\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:25.982504 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.982148    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9823062-c35f-4aca-b363-912ec43ecb40-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l2lg5\" (UID: \"f9823062-c35f-4aca-b363-912ec43ecb40\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:25.982504 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.982173    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/744c275a-8d87-4882-8e2e-73254829ca4a-run\") pod \"tuned-8hhhx\" (UID: \"744c275a-8d87-4882-8e2e-73254829ca4a\") " pod="openshift-cluster-node-tuning-operator/tuned-8hhhx"
May 26 16:01:25.982504 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.982189    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-877m4\" (UniqueName: \"kubernetes.io/projected/8828138d-e7ed-4d5c-ba76-26429679c7b1-kube-api-access-877m4\") pod \"network-metrics-daemon-qpzqg\" (UID: \"8828138d-e7ed-4d5c-ba76-26429679c7b1\") " pod="openshift-multus/network-metrics-daemon-qpzqg"
May 26 16:01:25.982504 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.982203    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/23cb5db1-ddc3-46a0-bc9d-3133631ffc58-cni-binary-copy\") pod \"multus-additional-cni-plugins-mjznd\" (UID: \"23cb5db1-ddc3-46a0-bc9d-3133631ffc58\") " pod="openshift-multus/multus-additional-cni-plugins-mjznd"
May 26 16:01:25.982504 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.982219    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f9823062-c35f-4aca-b363-912ec43ecb40-log-socket\") pod \"ovnkube-node-l2lg5\" (UID: \"f9823062-c35f-4aca-b363-912ec43ecb40\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:25.982504 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.982242    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5ad6cab9-b922-456e-ad28-95f40ce6692c-host-var-lib-cni-multus\") pod \"multus-wk2zv\" (UID: \"5ad6cab9-b922-456e-ad28-95f40ce6692c\") " pod="openshift-multus/multus-wk2zv"
May 26 16:01:25.982504 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.982278    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ef455917-b870-4d4a-b021-45a76d22734d-registration-dir\") pod \"aws-ebs-csi-driver-node-64xz7\" (UID: \"ef455917-b870-4d4a-b021-45a76d22734d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64xz7"
May 26 16:01:25.982504 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.982305    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ef455917-b870-4d4a-b021-45a76d22734d-etc-selinux\") pod \"aws-ebs-csi-driver-node-64xz7\" (UID: \"ef455917-b870-4d4a-b021-45a76d22734d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64xz7"
May 26 16:01:25.983298 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.982328    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9823062-c35f-4aca-b363-912ec43ecb40-run-openvswitch\") pod \"ovnkube-node-l2lg5\" (UID: \"f9823062-c35f-4aca-b363-912ec43ecb40\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:25.983298 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.982343    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5ad6cab9-b922-456e-ad28-95f40ce6692c-hostroot\") pod \"multus-wk2zv\" (UID: \"5ad6cab9-b922-456e-ad28-95f40ce6692c\") " pod="openshift-multus/multus-wk2zv"
May 26 16:01:25.983298 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:25.982382    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82dsh\" (UniqueName: \"kubernetes.io/projected/c54d2f93-a58c-4f73-966d-2584dc7015e8-kube-api-access-82dsh\") pod \"iptables-alerter-jlndg\" (UID: \"c54d2f93-a58c-4f73-966d-2584dc7015e8\") " pod="openshift-network-operator/iptables-alerter-jlndg"
May 26 16:01:26.015257 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.015164    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-25 15:56:25 +0000 UTC" deadline="2028-02-17 06:17:20.724281108 +0000 UTC"
May 26 16:01:26.015257 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.015192    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15158h15m54.709092324s"
May 26 16:01:26.070773 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.070739    2572 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
May 26 16:01:26.082786 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.082758    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/744c275a-8d87-4882-8e2e-73254829ca4a-lib-modules\") pod \"tuned-8hhhx\" (UID: \"744c275a-8d87-4882-8e2e-73254829ca4a\") " pod="openshift-cluster-node-tuning-operator/tuned-8hhhx"
May 26 16:01:26.082786 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.082789    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5ad6cab9-b922-456e-ad28-95f40ce6692c-cni-binary-copy\") pod \"multus-wk2zv\" (UID: \"5ad6cab9-b922-456e-ad28-95f40ce6692c\") " pod="openshift-multus/multus-wk2zv"
May 26 16:01:26.083018 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.082804    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ad6cab9-b922-456e-ad28-95f40ce6692c-etc-kubernetes\") pod \"multus-wk2zv\" (UID: \"5ad6cab9-b922-456e-ad28-95f40ce6692c\") " pod="openshift-multus/multus-wk2zv"
May 26 16:01:26.083018 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.082820    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/23cb5db1-ddc3-46a0-bc9d-3133631ffc58-cnibin\") pod \"multus-additional-cni-plugins-mjznd\" (UID: \"23cb5db1-ddc3-46a0-bc9d-3133631ffc58\") " pod="openshift-multus/multus-additional-cni-plugins-mjznd"
May 26 16:01:26.083018 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.082857    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/23cb5db1-ddc3-46a0-bc9d-3133631ffc58-cnibin\") pod \"multus-additional-cni-plugins-mjznd\" (UID: \"23cb5db1-ddc3-46a0-bc9d-3133631ffc58\") " pod="openshift-multus/multus-additional-cni-plugins-mjznd"
May 26 16:01:26.083018 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.082892    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ad6cab9-b922-456e-ad28-95f40ce6692c-etc-kubernetes\") pod \"multus-wk2zv\" (UID: \"5ad6cab9-b922-456e-ad28-95f40ce6692c\") " pod="openshift-multus/multus-wk2zv"
May 26 16:01:26.083018 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.082914    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fdchq\" (UniqueName: \"kubernetes.io/projected/23cb5db1-ddc3-46a0-bc9d-3133631ffc58-kube-api-access-fdchq\") pod \"multus-additional-cni-plugins-mjznd\" (UID: \"23cb5db1-ddc3-46a0-bc9d-3133631ffc58\") " pod="openshift-multus/multus-additional-cni-plugins-mjznd"
May 26 16:01:26.083018 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.082963    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/744c275a-8d87-4882-8e2e-73254829ca4a-lib-modules\") pod \"tuned-8hhhx\" (UID: \"744c275a-8d87-4882-8e2e-73254829ca4a\") " pod="openshift-cluster-node-tuning-operator/tuned-8hhhx"
May 26 16:01:26.083018 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.082968    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f9823062-c35f-4aca-b363-912ec43ecb40-host-run-netns\") pod \"ovnkube-node-l2lg5\" (UID: \"f9823062-c35f-4aca-b363-912ec43ecb40\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:26.083018 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.083010    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9823062-c35f-4aca-b363-912ec43ecb40-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l2lg5\" (UID: \"f9823062-c35f-4aca-b363-912ec43ecb40\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:26.083018 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.083011    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f9823062-c35f-4aca-b363-912ec43ecb40-host-run-netns\") pod \"ovnkube-node-l2lg5\" (UID: \"f9823062-c35f-4aca-b363-912ec43ecb40\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:26.083493 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.083047    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9823062-c35f-4aca-b363-912ec43ecb40-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l2lg5\" (UID: \"f9823062-c35f-4aca-b363-912ec43ecb40\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:26.083493 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.083083    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/744c275a-8d87-4882-8e2e-73254829ca4a-run\") pod \"tuned-8hhhx\" (UID: \"744c275a-8d87-4882-8e2e-73254829ca4a\") " pod="openshift-cluster-node-tuning-operator/tuned-8hhhx"
May 26 16:01:26.083493 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.083110    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-877m4\" (UniqueName: \"kubernetes.io/projected/8828138d-e7ed-4d5c-ba76-26429679c7b1-kube-api-access-877m4\") pod \"network-metrics-daemon-qpzqg\" (UID: \"8828138d-e7ed-4d5c-ba76-26429679c7b1\") " pod="openshift-multus/network-metrics-daemon-qpzqg"
May 26 16:01:26.083493 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.083129    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/23cb5db1-ddc3-46a0-bc9d-3133631ffc58-cni-binary-copy\") pod \"multus-additional-cni-plugins-mjznd\" (UID: \"23cb5db1-ddc3-46a0-bc9d-3133631ffc58\") " pod="openshift-multus/multus-additional-cni-plugins-mjznd"
May 26 16:01:26.083493 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.083145    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f9823062-c35f-4aca-b363-912ec43ecb40-log-socket\") pod \"ovnkube-node-l2lg5\" (UID: \"f9823062-c35f-4aca-b363-912ec43ecb40\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:26.083493 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.083153    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/744c275a-8d87-4882-8e2e-73254829ca4a-run\") pod \"tuned-8hhhx\" (UID: \"744c275a-8d87-4882-8e2e-73254829ca4a\") " pod="openshift-cluster-node-tuning-operator/tuned-8hhhx"
May 26 16:01:26.083493 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.083165    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5ad6cab9-b922-456e-ad28-95f40ce6692c-host-var-lib-cni-multus\") pod \"multus-wk2zv\" (UID: \"5ad6cab9-b922-456e-ad28-95f40ce6692c\") " pod="openshift-multus/multus-wk2zv"
May 26 16:01:26.083493 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.083197    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5ad6cab9-b922-456e-ad28-95f40ce6692c-host-var-lib-cni-multus\") pod \"multus-wk2zv\" (UID: \"5ad6cab9-b922-456e-ad28-95f40ce6692c\") " pod="openshift-multus/multus-wk2zv"
May 26 16:01:26.083493 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.083201    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f9823062-c35f-4aca-b363-912ec43ecb40-log-socket\") pod \"ovnkube-node-l2lg5\" (UID: \"f9823062-c35f-4aca-b363-912ec43ecb40\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:26.083493 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.083273    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ef455917-b870-4d4a-b021-45a76d22734d-registration-dir\") pod \"aws-ebs-csi-driver-node-64xz7\" (UID: \"ef455917-b870-4d4a-b021-45a76d22734d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64xz7"
May 26 16:01:26.083493 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.083302    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ef455917-b870-4d4a-b021-45a76d22734d-etc-selinux\") pod \"aws-ebs-csi-driver-node-64xz7\" (UID: \"ef455917-b870-4d4a-b021-45a76d22734d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64xz7"
May 26 16:01:26.083493 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.083328    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9823062-c35f-4aca-b363-912ec43ecb40-run-openvswitch\") pod \"ovnkube-node-l2lg5\" (UID: \"f9823062-c35f-4aca-b363-912ec43ecb40\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:26.083493 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.083374    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ef455917-b870-4d4a-b021-45a76d22734d-registration-dir\") pod \"aws-ebs-csi-driver-node-64xz7\" (UID: \"ef455917-b870-4d4a-b021-45a76d22734d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64xz7"
May 26 16:01:26.083493 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.083379    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5ad6cab9-b922-456e-ad28-95f40ce6692c-hostroot\") pod \"multus-wk2zv\" (UID: \"5ad6cab9-b922-456e-ad28-95f40ce6692c\") " pod="openshift-multus/multus-wk2zv"
May 26 16:01:26.083493 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.083405    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5ad6cab9-b922-456e-ad28-95f40ce6692c-cni-binary-copy\") pod \"multus-wk2zv\" (UID: \"5ad6cab9-b922-456e-ad28-95f40ce6692c\") " pod="openshift-multus/multus-wk2zv"
May 26 16:01:26.083493 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.083418    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5ad6cab9-b922-456e-ad28-95f40ce6692c-hostroot\") pod \"multus-wk2zv\" (UID: \"5ad6cab9-b922-456e-ad28-95f40ce6692c\") " pod="openshift-multus/multus-wk2zv"
May 26 16:01:26.083493 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.083423    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-82dsh\" (UniqueName: \"kubernetes.io/projected/c54d2f93-a58c-4f73-966d-2584dc7015e8-kube-api-access-82dsh\") pod \"iptables-alerter-jlndg\" (UID: \"c54d2f93-a58c-4f73-966d-2584dc7015e8\") " pod="openshift-network-operator/iptables-alerter-jlndg"
May 26 16:01:26.084006 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.083448    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ef455917-b870-4d4a-b021-45a76d22734d-etc-selinux\") pod \"aws-ebs-csi-driver-node-64xz7\" (UID: \"ef455917-b870-4d4a-b021-45a76d22734d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64xz7"
May 26 16:01:26.084006 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.083456    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/32dd34b2-8588-46f9-b744-7138c2ac7025-konnectivity-ca\") pod \"konnectivity-agent-tbphl\" (UID: \"32dd34b2-8588-46f9-b744-7138c2ac7025\") " pod="kube-system/konnectivity-agent-tbphl"
May 26 16:01:26.084006 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.083477    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9823062-c35f-4aca-b363-912ec43ecb40-run-openvswitch\") pod \"ovnkube-node-l2lg5\" (UID: \"f9823062-c35f-4aca-b363-912ec43ecb40\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:26.084006 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.083487    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9823062-c35f-4aca-b363-912ec43ecb40-host-kubelet\") pod \"ovnkube-node-l2lg5\" (UID: \"f9823062-c35f-4aca-b363-912ec43ecb40\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:26.084006 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.083537    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9823062-c35f-4aca-b363-912ec43ecb40-host-kubelet\") pod \"ovnkube-node-l2lg5\" (UID: \"f9823062-c35f-4aca-b363-912ec43ecb40\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:26.084006 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.083562    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9823062-c35f-4aca-b363-912ec43ecb40-var-lib-openvswitch\") pod \"ovnkube-node-l2lg5\" (UID: \"f9823062-c35f-4aca-b363-912ec43ecb40\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:26.084006 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.083594    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9823062-c35f-4aca-b363-912ec43ecb40-host-run-ovn-kubernetes\") pod \"ovnkube-node-l2lg5\" (UID: \"f9823062-c35f-4aca-b363-912ec43ecb40\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:26.084006 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.083635    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5ad6cab9-b922-456e-ad28-95f40ce6692c-cnibin\") pod \"multus-wk2zv\" (UID: \"5ad6cab9-b922-456e-ad28-95f40ce6692c\") " pod="openshift-multus/multus-wk2zv"
May 26 16:01:26.084006 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.083640    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9823062-c35f-4aca-b363-912ec43ecb40-var-lib-openvswitch\") pod \"ovnkube-node-l2lg5\" (UID: \"f9823062-c35f-4aca-b363-912ec43ecb40\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:26.084006 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.083660    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/744c275a-8d87-4882-8e2e-73254829ca4a-tmp\") pod \"tuned-8hhhx\" (UID: \"744c275a-8d87-4882-8e2e-73254829ca4a\") " pod="openshift-cluster-node-tuning-operator/tuned-8hhhx"
May 26 16:01:26.084006 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.083663    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9823062-c35f-4aca-b363-912ec43ecb40-host-run-ovn-kubernetes\") pod \"ovnkube-node-l2lg5\" (UID: \"f9823062-c35f-4aca-b363-912ec43ecb40\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:26.084006 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.083684    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-f8vcn\" (UniqueName: \"kubernetes.io/projected/744c275a-8d87-4882-8e2e-73254829ca4a-kube-api-access-f8vcn\") pod \"tuned-8hhhx\" (UID: \"744c275a-8d87-4882-8e2e-73254829ca4a\") " pod="openshift-cluster-node-tuning-operator/tuned-8hhhx"
May 26 16:01:26.084006 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.083697    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5ad6cab9-b922-456e-ad28-95f40ce6692c-cnibin\") pod \"multus-wk2zv\" (UID: \"5ad6cab9-b922-456e-ad28-95f40ce6692c\") " pod="openshift-multus/multus-wk2zv"
May 26 16:01:26.084006 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.083712    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lbgsb\" (UniqueName: \"kubernetes.io/projected/8f54a9e9-f1b3-4073-bf4e-fef6f31dc622-kube-api-access-lbgsb\") pod \"network-check-target-rvlkc\" (UID: \"8f54a9e9-f1b3-4073-bf4e-fef6f31dc622\") " pod="openshift-network-diagnostics/network-check-target-rvlkc"
May 26 16:01:26.084006 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.083737    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ef455917-b870-4d4a-b021-45a76d22734d-socket-dir\") pod \"aws-ebs-csi-driver-node-64xz7\" (UID: \"ef455917-b870-4d4a-b021-45a76d22734d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64xz7"
May 26 16:01:26.084006 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.083790    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9823062-c35f-4aca-b363-912ec43ecb40-etc-openvswitch\") pod \"ovnkube-node-l2lg5\" (UID: \"f9823062-c35f-4aca-b363-912ec43ecb40\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:26.084006 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.083806    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/23cb5db1-ddc3-46a0-bc9d-3133631ffc58-cni-binary-copy\") pod \"multus-additional-cni-plugins-mjznd\" (UID: \"23cb5db1-ddc3-46a0-bc9d-3133631ffc58\") " pod="openshift-multus/multus-additional-cni-plugins-mjznd"
May 26 16:01:26.084714 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.083815    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ad6cab9-b922-456e-ad28-95f40ce6692c-system-cni-dir\") pod \"multus-wk2zv\" (UID: \"5ad6cab9-b922-456e-ad28-95f40ce6692c\") " pod="openshift-multus/multus-wk2zv"
May 26 16:01:26.084714 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.083840    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ad6cab9-b922-456e-ad28-95f40ce6692c-host-var-lib-kubelet\") pod \"multus-wk2zv\" (UID: \"5ad6cab9-b922-456e-ad28-95f40ce6692c\") " pod="openshift-multus/multus-wk2zv"
May 26 16:01:26.084714 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.083862    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5ad6cab9-b922-456e-ad28-95f40ce6692c-multus-daemon-config\") pod \"multus-wk2zv\" (UID: \"5ad6cab9-b922-456e-ad28-95f40ce6692c\") " pod="openshift-multus/multus-wk2zv"
May 26 16:01:26.084714 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.083884    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/744c275a-8d87-4882-8e2e-73254829ca4a-etc-modprobe-d\") pod \"tuned-8hhhx\" (UID: \"744c275a-8d87-4882-8e2e-73254829ca4a\") " pod="openshift-cluster-node-tuning-operator/tuned-8hhhx"
May 26 16:01:26.084714 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.083908    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ef455917-b870-4d4a-b021-45a76d22734d-socket-dir\") pod \"aws-ebs-csi-driver-node-64xz7\" (UID: \"ef455917-b870-4d4a-b021-45a76d22734d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64xz7"
May 26 16:01:26.084714 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.083907    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/744c275a-8d87-4882-8e2e-73254829ca4a-host\") pod \"tuned-8hhhx\" (UID: \"744c275a-8d87-4882-8e2e-73254829ca4a\") " pod="openshift-cluster-node-tuning-operator/tuned-8hhhx"
May 26 16:01:26.084714 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.083957    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ad6cab9-b922-456e-ad28-95f40ce6692c-host-var-lib-kubelet\") pod \"multus-wk2zv\" (UID: \"5ad6cab9-b922-456e-ad28-95f40ce6692c\") " pod="openshift-multus/multus-wk2zv"
May 26 16:01:26.084714 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.083981    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/744c275a-8d87-4882-8e2e-73254829ca4a-host\") pod \"tuned-8hhhx\" (UID: \"744c275a-8d87-4882-8e2e-73254829ca4a\") " pod="openshift-cluster-node-tuning-operator/tuned-8hhhx"
May 26 16:01:26.084714 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.083989    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/744c275a-8d87-4882-8e2e-73254829ca4a-etc-tuned\") pod \"tuned-8hhhx\" (UID: \"744c275a-8d87-4882-8e2e-73254829ca4a\") " pod="openshift-cluster-node-tuning-operator/tuned-8hhhx"
May 26 16:01:26.084714 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.083999    2572 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
May 26 16:01:26.084714 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.084025    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ef455917-b870-4d4a-b021-45a76d22734d-sys-fs\") pod \"aws-ebs-csi-driver-node-64xz7\" (UID: \"ef455917-b870-4d4a-b021-45a76d22734d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64xz7"
May 26 16:01:26.084714 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.084023    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/32dd34b2-8588-46f9-b744-7138c2ac7025-konnectivity-ca\") pod \"konnectivity-agent-tbphl\" (UID: \"32dd34b2-8588-46f9-b744-7138c2ac7025\") " pod="kube-system/konnectivity-agent-tbphl"
May 26 16:01:26.084714 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.084081    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ad6cab9-b922-456e-ad28-95f40ce6692c-system-cni-dir\") pod \"multus-wk2zv\" (UID: \"5ad6cab9-b922-456e-ad28-95f40ce6692c\") " pod="openshift-multus/multus-wk2zv"
May 26 16:01:26.084714 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.084123    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ef455917-b870-4d4a-b021-45a76d22734d-sys-fs\") pod \"aws-ebs-csi-driver-node-64xz7\" (UID: \"ef455917-b870-4d4a-b021-45a76d22734d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64xz7"
May 26 16:01:26.084714 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.084119    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f9823062-c35f-4aca-b363-912ec43ecb40-host-slash\") pod \"ovnkube-node-l2lg5\" (UID: \"f9823062-c35f-4aca-b363-912ec43ecb40\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:26.084714 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.084169    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f9823062-c35f-4aca-b363-912ec43ecb40-run-ovn\") pod \"ovnkube-node-l2lg5\" (UID: \"f9823062-c35f-4aca-b363-912ec43ecb40\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:26.084714 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.084196    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-84fpx\" (UniqueName: \"kubernetes.io/projected/f9823062-c35f-4aca-b363-912ec43ecb40-kube-api-access-84fpx\") pod \"ovnkube-node-l2lg5\" (UID: \"f9823062-c35f-4aca-b363-912ec43ecb40\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:26.084714 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.084200    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f9823062-c35f-4aca-b363-912ec43ecb40-host-slash\") pod \"ovnkube-node-l2lg5\" (UID: \"f9823062-c35f-4aca-b363-912ec43ecb40\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:26.085521 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.084170    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9823062-c35f-4aca-b363-912ec43ecb40-etc-openvswitch\") pod \"ovnkube-node-l2lg5\" (UID: \"f9823062-c35f-4aca-b363-912ec43ecb40\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:26.085521 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.084222    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5ad6cab9-b922-456e-ad28-95f40ce6692c-os-release\") pod \"multus-wk2zv\" (UID: \"5ad6cab9-b922-456e-ad28-95f40ce6692c\") " pod="openshift-multus/multus-wk2zv"
May 26 16:01:26.085521 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.084250    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/23cb5db1-ddc3-46a0-bc9d-3133631ffc58-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mjznd\" (UID: \"23cb5db1-ddc3-46a0-bc9d-3133631ffc58\") " pod="openshift-multus/multus-additional-cni-plugins-mjznd"
May 26 16:01:26.085521 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.084277    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02c134e5-0f87-48c2-80ed-427d242f1435-tmp-dir\") pod \"node-resolver-lvfvd\" (UID: \"02c134e5-0f87-48c2-80ed-427d242f1435\") " pod="openshift-dns/node-resolver-lvfvd"
May 26 16:01:26.085521 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.084308    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5ad6cab9-b922-456e-ad28-95f40ce6692c-os-release\") pod \"multus-wk2zv\" (UID: \"5ad6cab9-b922-456e-ad28-95f40ce6692c\") " pod="openshift-multus/multus-wk2zv"
May 26 16:01:26.085521 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.084310    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5ad6cab9-b922-456e-ad28-95f40ce6692c-host-run-multus-certs\") pod \"multus-wk2zv\" (UID: \"5ad6cab9-b922-456e-ad28-95f40ce6692c\") " pod="openshift-multus/multus-wk2zv"
May 26 16:01:26.085521 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.084350    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/10ea1129-78a4-4d29-ab50-50b69b6d30c1-host\") pod \"node-ca-qdk28\" (UID: \"10ea1129-78a4-4d29-ab50-50b69b6d30c1\") " pod="openshift-image-registry/node-ca-qdk28"
May 26 16:01:26.085521 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.084377    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/23cb5db1-ddc3-46a0-bc9d-3133631ffc58-system-cni-dir\") pod \"multus-additional-cni-plugins-mjznd\" (UID: \"23cb5db1-ddc3-46a0-bc9d-3133631ffc58\") " pod="openshift-multus/multus-additional-cni-plugins-mjznd"
May 26 16:01:26.085521 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.084350    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5ad6cab9-b922-456e-ad28-95f40ce6692c-host-run-multus-certs\") pod \"multus-wk2zv\" (UID: \"5ad6cab9-b922-456e-ad28-95f40ce6692c\") " pod="openshift-multus/multus-wk2zv"
May 26 16:01:26.085521 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.084429    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f9823062-c35f-4aca-b363-912ec43ecb40-ovn-node-metrics-cert\") pod \"ovnkube-node-l2lg5\" (UID: \"f9823062-c35f-4aca-b363-912ec43ecb40\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:26.085521 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.084436    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5ad6cab9-b922-456e-ad28-95f40ce6692c-multus-daemon-config\") pod \"multus-wk2zv\" (UID: \"5ad6cab9-b922-456e-ad28-95f40ce6692c\") " pod="openshift-multus/multus-wk2zv"
May 26 16:01:26.085521 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.084251    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f9823062-c35f-4aca-b363-912ec43ecb40-run-ovn\") pod \"ovnkube-node-l2lg5\" (UID: \"f9823062-c35f-4aca-b363-912ec43ecb40\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:26.085521 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.084456    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5ad6cab9-b922-456e-ad28-95f40ce6692c-host-var-lib-cni-bin\") pod \"multus-wk2zv\" (UID: \"5ad6cab9-b922-456e-ad28-95f40ce6692c\") " pod="openshift-multus/multus-wk2zv"
May 26 16:01:26.085521 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.084468    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/23cb5db1-ddc3-46a0-bc9d-3133631ffc58-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mjznd\" (UID: \"23cb5db1-ddc3-46a0-bc9d-3133631ffc58\") " pod="openshift-multus/multus-additional-cni-plugins-mjznd"
May 26 16:01:26.085521 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.084220    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/744c275a-8d87-4882-8e2e-73254829ca4a-etc-modprobe-d\") pod \"tuned-8hhhx\" (UID: \"744c275a-8d87-4882-8e2e-73254829ca4a\") " pod="openshift-cluster-node-tuning-operator/tuned-8hhhx"
May 26 16:01:26.085521 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.084487    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/744c275a-8d87-4882-8e2e-73254829ca4a-etc-sysctl-d\") pod \"tuned-8hhhx\" (UID: \"744c275a-8d87-4882-8e2e-73254829ca4a\") " pod="openshift-cluster-node-tuning-operator/tuned-8hhhx"
May 26 16:01:26.085521 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.084508    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/23cb5db1-ddc3-46a0-bc9d-3133631ffc58-system-cni-dir\") pod \"multus-additional-cni-plugins-mjznd\" (UID: \"23cb5db1-ddc3-46a0-bc9d-3133631ffc58\") " pod="openshift-multus/multus-additional-cni-plugins-mjznd"
May 26 16:01:26.085521 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.084513    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/744c275a-8d87-4882-8e2e-73254829ca4a-etc-sysctl-conf\") pod \"tuned-8hhhx\" (UID: \"744c275a-8d87-4882-8e2e-73254829ca4a\") " pod="openshift-cluster-node-tuning-operator/tuned-8hhhx"
May 26 16:01:26.086374 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.084538    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/23cb5db1-ddc3-46a0-bc9d-3133631ffc58-os-release\") pod \"multus-additional-cni-plugins-mjznd\" (UID: \"23cb5db1-ddc3-46a0-bc9d-3133631ffc58\") " pod="openshift-multus/multus-additional-cni-plugins-mjznd"
May 26 16:01:26.086374 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.084563    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f9823062-c35f-4aca-b363-912ec43ecb40-systemd-units\") pod \"ovnkube-node-l2lg5\" (UID: \"f9823062-c35f-4aca-b363-912ec43ecb40\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:26.086374 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.084592    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f9823062-c35f-4aca-b363-912ec43ecb40-run-systemd\") pod \"ovnkube-node-l2lg5\" (UID: \"f9823062-c35f-4aca-b363-912ec43ecb40\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:26.086374 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.084632    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ad6cab9-b922-456e-ad28-95f40ce6692c-multus-cni-dir\") pod \"multus-wk2zv\" (UID: \"5ad6cab9-b922-456e-ad28-95f40ce6692c\") " pod="openshift-multus/multus-wk2zv"
May 26 16:01:26.086374 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.084650    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/744c275a-8d87-4882-8e2e-73254829ca4a-etc-sysctl-d\") pod \"tuned-8hhhx\" (UID: \"744c275a-8d87-4882-8e2e-73254829ca4a\") " pod="openshift-cluster-node-tuning-operator/tuned-8hhhx"
May 26 16:01:26.086374 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.084657    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5ad6cab9-b922-456e-ad28-95f40ce6692c-multus-conf-dir\") pod \"multus-wk2zv\" (UID: \"5ad6cab9-b922-456e-ad28-95f40ce6692c\") " pod="openshift-multus/multus-wk2zv"
May 26 16:01:26.086374 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.084682    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/744c275a-8d87-4882-8e2e-73254829ca4a-var-lib-kubelet\") pod \"tuned-8hhhx\" (UID: \"744c275a-8d87-4882-8e2e-73254829ca4a\") " pod="openshift-cluster-node-tuning-operator/tuned-8hhhx"
May 26 16:01:26.086374 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.084697    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5ad6cab9-b922-456e-ad28-95f40ce6692c-host-var-lib-cni-bin\") pod \"multus-wk2zv\" (UID: \"5ad6cab9-b922-456e-ad28-95f40ce6692c\") " pod="openshift-multus/multus-wk2zv"
May 26 16:01:26.086374 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.084708    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c54d2f93-a58c-4f73-966d-2584dc7015e8-host-slash\") pod \"iptables-alerter-jlndg\" (UID: \"c54d2f93-a58c-4f73-966d-2584dc7015e8\") " pod="openshift-network-operator/iptables-alerter-jlndg"
May 26 16:01:26.086374 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.084718    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02c134e5-0f87-48c2-80ed-427d242f1435-tmp-dir\") pod \"node-resolver-lvfvd\" (UID: \"02c134e5-0f87-48c2-80ed-427d242f1435\") " pod="openshift-dns/node-resolver-lvfvd"
May 26 16:01:26.086374 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.084734    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ltn4t\" (UniqueName: \"kubernetes.io/projected/10ea1129-78a4-4d29-ab50-50b69b6d30c1-kube-api-access-ltn4t\") pod \"node-ca-qdk28\" (UID: \"10ea1129-78a4-4d29-ab50-50b69b6d30c1\") " pod="openshift-image-registry/node-ca-qdk28"
May 26 16:01:26.086374 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.084472    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/10ea1129-78a4-4d29-ab50-50b69b6d30c1-host\") pod \"node-ca-qdk28\" (UID: \"10ea1129-78a4-4d29-ab50-50b69b6d30c1\") " pod="openshift-image-registry/node-ca-qdk28"
May 26 16:01:26.086374 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.084763    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f9823062-c35f-4aca-b363-912ec43ecb40-env-overrides\") pod \"ovnkube-node-l2lg5\" (UID: \"f9823062-c35f-4aca-b363-912ec43ecb40\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:26.086374 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.084817    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/23cb5db1-ddc3-46a0-bc9d-3133631ffc58-os-release\") pod \"multus-additional-cni-plugins-mjznd\" (UID: \"23cb5db1-ddc3-46a0-bc9d-3133631ffc58\") " pod="openshift-multus/multus-additional-cni-plugins-mjznd"
May 26 16:01:26.086374 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.084847    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-frmct\" (UniqueName: \"kubernetes.io/projected/ef455917-b870-4d4a-b021-45a76d22734d-kube-api-access-frmct\") pod \"aws-ebs-csi-driver-node-64xz7\" (UID: \"ef455917-b870-4d4a-b021-45a76d22734d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64xz7"
May 26 16:01:26.086374 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.084858    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/744c275a-8d87-4882-8e2e-73254829ca4a-etc-sysctl-conf\") pod \"tuned-8hhhx\" (UID: \"744c275a-8d87-4882-8e2e-73254829ca4a\") " pod="openshift-cluster-node-tuning-operator/tuned-8hhhx"
May 26 16:01:26.086374 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.084871    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f9823062-c35f-4aca-b363-912ec43ecb40-node-log\") pod \"ovnkube-node-l2lg5\" (UID: \"f9823062-c35f-4aca-b363-912ec43ecb40\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:26.086374 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.084899    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/32dd34b2-8588-46f9-b744-7138c2ac7025-agent-certs\") pod \"konnectivity-agent-tbphl\" (UID: \"32dd34b2-8588-46f9-b744-7138c2ac7025\") " pod="kube-system/konnectivity-agent-tbphl"
May 26 16:01:26.087224 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.084929    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ef455917-b870-4d4a-b021-45a76d22734d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-64xz7\" (UID: \"ef455917-b870-4d4a-b021-45a76d22734d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64xz7"
May 26 16:01:26.087224 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.084928    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f9823062-c35f-4aca-b363-912ec43ecb40-run-systemd\") pod \"ovnkube-node-l2lg5\" (UID: \"f9823062-c35f-4aca-b363-912ec43ecb40\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:26.087224 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.084955    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ef455917-b870-4d4a-b021-45a76d22734d-device-dir\") pod \"aws-ebs-csi-driver-node-64xz7\" (UID: \"ef455917-b870-4d4a-b021-45a76d22734d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64xz7"
May 26 16:01:26.087224 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.084965    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f9823062-c35f-4aca-b363-912ec43ecb40-node-log\") pod \"ovnkube-node-l2lg5\" (UID: \"f9823062-c35f-4aca-b363-912ec43ecb40\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:26.087224 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.084982    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-x6rlr\" (UniqueName: \"kubernetes.io/projected/02c134e5-0f87-48c2-80ed-427d242f1435-kube-api-access-x6rlr\") pod \"node-resolver-lvfvd\" (UID: \"02c134e5-0f87-48c2-80ed-427d242f1435\") " pod="openshift-dns/node-resolver-lvfvd"
May 26 16:01:26.087224 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.085003    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ad6cab9-b922-456e-ad28-95f40ce6692c-multus-cni-dir\") pod \"multus-wk2zv\" (UID: \"5ad6cab9-b922-456e-ad28-95f40ce6692c\") " pod="openshift-multus/multus-wk2zv"
May 26 16:01:26.087224 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.085009    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5ad6cab9-b922-456e-ad28-95f40ce6692c-host-run-k8s-cni-cncf-io\") pod \"multus-wk2zv\" (UID: \"5ad6cab9-b922-456e-ad28-95f40ce6692c\") " pod="openshift-multus/multus-wk2zv"
May 26 16:01:26.087224 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.085035    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/744c275a-8d87-4882-8e2e-73254829ca4a-etc-kubernetes\") pod \"tuned-8hhhx\" (UID: \"744c275a-8d87-4882-8e2e-73254829ca4a\") " pod="openshift-cluster-node-tuning-operator/tuned-8hhhx"
May 26 16:01:26.087224 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.085045    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5ad6cab9-b922-456e-ad28-95f40ce6692c-multus-conf-dir\") pod \"multus-wk2zv\" (UID: \"5ad6cab9-b922-456e-ad28-95f40ce6692c\") " pod="openshift-multus/multus-wk2zv"
May 26 16:01:26.087224 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.085059    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c54d2f93-a58c-4f73-966d-2584dc7015e8-iptables-alerter-script\") pod \"iptables-alerter-jlndg\" (UID: \"c54d2f93-a58c-4f73-966d-2584dc7015e8\") " pod="openshift-network-operator/iptables-alerter-jlndg"
May 26 16:01:26.087224 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.085085    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/744c275a-8d87-4882-8e2e-73254829ca4a-etc-sysconfig\") pod \"tuned-8hhhx\" (UID: \"744c275a-8d87-4882-8e2e-73254829ca4a\") " pod="openshift-cluster-node-tuning-operator/tuned-8hhhx"
May 26 16:01:26.087224 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.085100    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ef455917-b870-4d4a-b021-45a76d22734d-device-dir\") pod \"aws-ebs-csi-driver-node-64xz7\" (UID: \"ef455917-b870-4d4a-b021-45a76d22734d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64xz7"
May 26 16:01:26.087224 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.085119    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f9823062-c35f-4aca-b363-912ec43ecb40-ovnkube-config\") pod \"ovnkube-node-l2lg5\" (UID: \"f9823062-c35f-4aca-b363-912ec43ecb40\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:26.087224 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.085147    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f9823062-c35f-4aca-b363-912ec43ecb40-ovnkube-script-lib\") pod \"ovnkube-node-l2lg5\" (UID: \"f9823062-c35f-4aca-b363-912ec43ecb40\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:26.087224 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.085171    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5ad6cab9-b922-456e-ad28-95f40ce6692c-multus-socket-dir-parent\") pod \"multus-wk2zv\" (UID: \"5ad6cab9-b922-456e-ad28-95f40ce6692c\") " pod="openshift-multus/multus-wk2zv"
May 26 16:01:26.087224 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.085190    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c54d2f93-a58c-4f73-966d-2584dc7015e8-host-slash\") pod \"iptables-alerter-jlndg\" (UID: \"c54d2f93-a58c-4f73-966d-2584dc7015e8\") " pod="openshift-network-operator/iptables-alerter-jlndg"
May 26 16:01:26.087224 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.085198    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5ad6cab9-b922-456e-ad28-95f40ce6692c-host-run-netns\") pod \"multus-wk2zv\" (UID: \"5ad6cab9-b922-456e-ad28-95f40ce6692c\") " pod="openshift-multus/multus-wk2zv"
May 26 16:01:26.088000 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.085223    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/10ea1129-78a4-4d29-ab50-50b69b6d30c1-serviceca\") pod \"node-ca-qdk28\" (UID: \"10ea1129-78a4-4d29-ab50-50b69b6d30c1\") " pod="openshift-image-registry/node-ca-qdk28"
May 26 16:01:26.088000 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.085248    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/23cb5db1-ddc3-46a0-bc9d-3133631ffc58-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mjznd\" (UID: \"23cb5db1-ddc3-46a0-bc9d-3133631ffc58\") " pod="openshift-multus/multus-additional-cni-plugins-mjznd"
May 26 16:01:26.088000 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.084735    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f9823062-c35f-4aca-b363-912ec43ecb40-systemd-units\") pod \"ovnkube-node-l2lg5\" (UID: \"f9823062-c35f-4aca-b363-912ec43ecb40\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:26.088000 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.085146    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f9823062-c35f-4aca-b363-912ec43ecb40-env-overrides\") pod \"ovnkube-node-l2lg5\" (UID: \"f9823062-c35f-4aca-b363-912ec43ecb40\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:26.088000 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.085398    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5ad6cab9-b922-456e-ad28-95f40ce6692c-multus-socket-dir-parent\") pod \"multus-wk2zv\" (UID: \"5ad6cab9-b922-456e-ad28-95f40ce6692c\") " pod="openshift-multus/multus-wk2zv"
May 26 16:01:26.088000 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.085398    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/744c275a-8d87-4882-8e2e-73254829ca4a-etc-sysconfig\") pod \"tuned-8hhhx\" (UID: \"744c275a-8d87-4882-8e2e-73254829ca4a\") " pod="openshift-cluster-node-tuning-operator/tuned-8hhhx"
May 26 16:01:26.088000 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.085432    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ef455917-b870-4d4a-b021-45a76d22734d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-64xz7\" (UID: \"ef455917-b870-4d4a-b021-45a76d22734d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64xz7"
May 26 16:01:26.088000 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.085496    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/744c275a-8d87-4882-8e2e-73254829ca4a-etc-kubernetes\") pod \"tuned-8hhhx\" (UID: \"744c275a-8d87-4882-8e2e-73254829ca4a\") " pod="openshift-cluster-node-tuning-operator/tuned-8hhhx"
May 26 16:01:26.088000 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.084907    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/744c275a-8d87-4882-8e2e-73254829ca4a-var-lib-kubelet\") pod \"tuned-8hhhx\" (UID: \"744c275a-8d87-4882-8e2e-73254829ca4a\") " pod="openshift-cluster-node-tuning-operator/tuned-8hhhx"
May 26 16:01:26.088000 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.085682    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c54d2f93-a58c-4f73-966d-2584dc7015e8-iptables-alerter-script\") pod \"iptables-alerter-jlndg\" (UID: \"c54d2f93-a58c-4f73-966d-2584dc7015e8\") " pod="openshift-network-operator/iptables-alerter-jlndg"
May 26 16:01:26.088000 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.085730    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5ad6cab9-b922-456e-ad28-95f40ce6692c-host-run-k8s-cni-cncf-io\") pod \"multus-wk2zv\" (UID: \"5ad6cab9-b922-456e-ad28-95f40ce6692c\") " pod="openshift-multus/multus-wk2zv"
May 26 16:01:26.088000 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.085895    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/23cb5db1-ddc3-46a0-bc9d-3133631ffc58-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mjznd\" (UID: \"23cb5db1-ddc3-46a0-bc9d-3133631ffc58\") " pod="openshift-multus/multus-additional-cni-plugins-mjznd"
May 26 16:01:26.088000 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.085899    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f9823062-c35f-4aca-b363-912ec43ecb40-ovnkube-config\") pod \"ovnkube-node-l2lg5\" (UID: \"f9823062-c35f-4aca-b363-912ec43ecb40\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:26.088000 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.085320    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/23cb5db1-ddc3-46a0-bc9d-3133631ffc58-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mjznd\" (UID: \"23cb5db1-ddc3-46a0-bc9d-3133631ffc58\") " pod="openshift-multus/multus-additional-cni-plugins-mjznd"
May 26 16:01:26.088000 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.085956    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/744c275a-8d87-4882-8e2e-73254829ca4a-etc-systemd\") pod \"tuned-8hhhx\" (UID: \"744c275a-8d87-4882-8e2e-73254829ca4a\") " pod="openshift-cluster-node-tuning-operator/tuned-8hhhx"
May 26 16:01:26.088000 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.085982    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f9823062-c35f-4aca-b363-912ec43ecb40-host-cni-bin\") pod \"ovnkube-node-l2lg5\" (UID: \"f9823062-c35f-4aca-b363-912ec43ecb40\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:26.088000 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.086005    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/02c134e5-0f87-48c2-80ed-427d242f1435-hosts-file\") pod \"node-resolver-lvfvd\" (UID: \"02c134e5-0f87-48c2-80ed-427d242f1435\") " pod="openshift-dns/node-resolver-lvfvd"
May 26 16:01:26.088715 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.086024    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f9823062-c35f-4aca-b363-912ec43ecb40-ovnkube-script-lib\") pod \"ovnkube-node-l2lg5\" (UID: \"f9823062-c35f-4aca-b363-912ec43ecb40\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:26.088715 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.086074    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/744c275a-8d87-4882-8e2e-73254829ca4a-etc-systemd\") pod \"tuned-8hhhx\" (UID: \"744c275a-8d87-4882-8e2e-73254829ca4a\") " pod="openshift-cluster-node-tuning-operator/tuned-8hhhx"
May 26 16:01:26.088715 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.086083    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/744c275a-8d87-4882-8e2e-73254829ca4a-sys\") pod \"tuned-8hhhx\" (UID: \"744c275a-8d87-4882-8e2e-73254829ca4a\") " pod="openshift-cluster-node-tuning-operator/tuned-8hhhx"
May 26 16:01:26.088715 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.086029    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/744c275a-8d87-4882-8e2e-73254829ca4a-sys\") pod \"tuned-8hhhx\" (UID: \"744c275a-8d87-4882-8e2e-73254829ca4a\") " pod="openshift-cluster-node-tuning-operator/tuned-8hhhx"
May 26 16:01:26.088715 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.086117    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f9823062-c35f-4aca-b363-912ec43ecb40-host-cni-bin\") pod \"ovnkube-node-l2lg5\" (UID: \"f9823062-c35f-4aca-b363-912ec43ecb40\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:26.088715 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.086121    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8828138d-e7ed-4d5c-ba76-26429679c7b1-metrics-certs\") pod \"network-metrics-daemon-qpzqg\" (UID: \"8828138d-e7ed-4d5c-ba76-26429679c7b1\") " pod="openshift-multus/network-metrics-daemon-qpzqg"
May 26 16:01:26.088715 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.086127    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/10ea1129-78a4-4d29-ab50-50b69b6d30c1-serviceca\") pod \"node-ca-qdk28\" (UID: \"10ea1129-78a4-4d29-ab50-50b69b6d30c1\") " pod="openshift-image-registry/node-ca-qdk28"
May 26 16:01:26.088715 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.086151    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f9823062-c35f-4aca-b363-912ec43ecb40-host-cni-netd\") pod \"ovnkube-node-l2lg5\" (UID: \"f9823062-c35f-4aca-b363-912ec43ecb40\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:26.088715 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.086173    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/02c134e5-0f87-48c2-80ed-427d242f1435-hosts-file\") pod \"node-resolver-lvfvd\" (UID: \"02c134e5-0f87-48c2-80ed-427d242f1435\") " pod="openshift-dns/node-resolver-lvfvd"
May 26 16:01:26.088715 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:26.086208    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:01:26.088715 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.086213    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f9823062-c35f-4aca-b363-912ec43ecb40-host-cni-netd\") pod \"ovnkube-node-l2lg5\" (UID: \"f9823062-c35f-4aca-b363-912ec43ecb40\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:26.088715 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.086234    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5ad6cab9-b922-456e-ad28-95f40ce6692c-host-run-netns\") pod \"multus-wk2zv\" (UID: \"5ad6cab9-b922-456e-ad28-95f40ce6692c\") " pod="openshift-multus/multus-wk2zv"
May 26 16:01:26.088715 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.086244    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-x9mxv\" (UniqueName: \"kubernetes.io/projected/5ad6cab9-b922-456e-ad28-95f40ce6692c-kube-api-access-x9mxv\") pod \"multus-wk2zv\" (UID: \"5ad6cab9-b922-456e-ad28-95f40ce6692c\") " pod="openshift-multus/multus-wk2zv"
May 26 16:01:26.088715 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:26.086293    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8828138d-e7ed-4d5c-ba76-26429679c7b1-metrics-certs podName:8828138d-e7ed-4d5c-ba76-26429679c7b1 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:26.586251277 +0000 UTC m=+3.061978622 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8828138d-e7ed-4d5c-ba76-26429679c7b1-metrics-certs") pod "network-metrics-daemon-qpzqg" (UID: "8828138d-e7ed-4d5c-ba76-26429679c7b1") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:01:26.088715 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.086306    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/23cb5db1-ddc3-46a0-bc9d-3133631ffc58-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mjznd\" (UID: \"23cb5db1-ddc3-46a0-bc9d-3133631ffc58\") " pod="openshift-multus/multus-additional-cni-plugins-mjznd"
May 26 16:01:26.088715 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.087693    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/744c275a-8d87-4882-8e2e-73254829ca4a-etc-tuned\") pod \"tuned-8hhhx\" (UID: \"744c275a-8d87-4882-8e2e-73254829ca4a\") " pod="openshift-cluster-node-tuning-operator/tuned-8hhhx"
May 26 16:01:26.088715 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.088178    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/744c275a-8d87-4882-8e2e-73254829ca4a-tmp\") pod \"tuned-8hhhx\" (UID: \"744c275a-8d87-4882-8e2e-73254829ca4a\") " pod="openshift-cluster-node-tuning-operator/tuned-8hhhx"
May 26 16:01:26.088715 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.088449    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/32dd34b2-8588-46f9-b744-7138c2ac7025-agent-certs\") pod \"konnectivity-agent-tbphl\" (UID: \"32dd34b2-8588-46f9-b744-7138c2ac7025\") " pod="kube-system/konnectivity-agent-tbphl"
May 26 16:01:26.089587 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.088585    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f9823062-c35f-4aca-b363-912ec43ecb40-ovn-node-metrics-cert\") pod \"ovnkube-node-l2lg5\" (UID: \"f9823062-c35f-4aca-b363-912ec43ecb40\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:26.095719 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.094177    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8vcn\" (UniqueName: \"kubernetes.io/projected/744c275a-8d87-4882-8e2e-73254829ca4a-kube-api-access-f8vcn\") pod \"tuned-8hhhx\" (UID: \"744c275a-8d87-4882-8e2e-73254829ca4a\") " pod="openshift-cluster-node-tuning-operator/tuned-8hhhx"
May 26 16:01:26.095719 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.094521    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-877m4\" (UniqueName: \"kubernetes.io/projected/8828138d-e7ed-4d5c-ba76-26429679c7b1-kube-api-access-877m4\") pod \"network-metrics-daemon-qpzqg\" (UID: \"8828138d-e7ed-4d5c-ba76-26429679c7b1\") " pod="openshift-multus/network-metrics-daemon-qpzqg"
May 26 16:01:26.095719 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.094757    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82dsh\" (UniqueName: \"kubernetes.io/projected/c54d2f93-a58c-4f73-966d-2584dc7015e8-kube-api-access-82dsh\") pod \"iptables-alerter-jlndg\" (UID: \"c54d2f93-a58c-4f73-966d-2584dc7015e8\") " pod="openshift-network-operator/iptables-alerter-jlndg"
May 26 16:01:26.095719 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:26.094897    2572 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 16:01:26.095719 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:26.094914    2572 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 16:01:26.095719 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:26.094934    2572 projected.go:194] Error preparing data for projected volume kube-api-access-lbgsb for pod openshift-network-diagnostics/network-check-target-rvlkc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:01:26.095719 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:26.095044    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f54a9e9-f1b3-4073-bf4e-fef6f31dc622-kube-api-access-lbgsb podName:8f54a9e9-f1b3-4073-bf4e-fef6f31dc622 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:26.595016536 +0000 UTC m=+3.070743878 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lbgsb" (UniqueName: "kubernetes.io/projected/8f54a9e9-f1b3-4073-bf4e-fef6f31dc622-kube-api-access-lbgsb") pod "network-check-target-rvlkc" (UID: "8f54a9e9-f1b3-4073-bf4e-fef6f31dc622") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:01:26.095719 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.095183    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84fpx\" (UniqueName: \"kubernetes.io/projected/f9823062-c35f-4aca-b363-912ec43ecb40-kube-api-access-84fpx\") pod \"ovnkube-node-l2lg5\" (UID: \"f9823062-c35f-4aca-b363-912ec43ecb40\") " pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:26.095719 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.095338    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdchq\" (UniqueName: \"kubernetes.io/projected/23cb5db1-ddc3-46a0-bc9d-3133631ffc58-kube-api-access-fdchq\") pod \"multus-additional-cni-plugins-mjznd\" (UID: \"23cb5db1-ddc3-46a0-bc9d-3133631ffc58\") " pod="openshift-multus/multus-additional-cni-plugins-mjznd"
May 26 16:01:26.096283 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.096172    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frmct\" (UniqueName: \"kubernetes.io/projected/ef455917-b870-4d4a-b021-45a76d22734d-kube-api-access-frmct\") pod \"aws-ebs-csi-driver-node-64xz7\" (UID: \"ef455917-b870-4d4a-b021-45a76d22734d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64xz7"
May 26 16:01:26.099338 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.099275    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6rlr\" (UniqueName: \"kubernetes.io/projected/02c134e5-0f87-48c2-80ed-427d242f1435-kube-api-access-x6rlr\") pod \"node-resolver-lvfvd\" (UID: \"02c134e5-0f87-48c2-80ed-427d242f1435\") " pod="openshift-dns/node-resolver-lvfvd"
May 26 16:01:26.100681 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.100636    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltn4t\" (UniqueName: \"kubernetes.io/projected/10ea1129-78a4-4d29-ab50-50b69b6d30c1-kube-api-access-ltn4t\") pod \"node-ca-qdk28\" (UID: \"10ea1129-78a4-4d29-ab50-50b69b6d30c1\") " pod="openshift-image-registry/node-ca-qdk28"
May 26 16:01:26.100791 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.100776    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9mxv\" (UniqueName: \"kubernetes.io/projected/5ad6cab9-b922-456e-ad28-95f40ce6692c-kube-api-access-x9mxv\") pod \"multus-wk2zv\" (UID: \"5ad6cab9-b922-456e-ad28-95f40ce6692c\") " pod="openshift-multus/multus-wk2zv"
May 26 16:01:26.268253 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.268169    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qdk28"
May 26 16:01:26.274213 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.274184    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mjznd"
May 26 16:01:26.281821 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.281793    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wk2zv"
May 26 16:01:26.287415 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.287396    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tbphl"
May 26 16:01:26.293097 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.293066    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64xz7"
May 26 16:01:26.298760 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.298735    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8hhhx"
May 26 16:01:26.307404 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.307379    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jlndg"
May 26 16:01:26.312049 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.312022    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:26.317694 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.317676    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lvfvd"
May 26 16:01:26.492015 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:26.491988    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef455917_b870_4d4a_b021_45a76d22734d.slice/crio-c073c56af2b6cc0cd381ef1104c3e7ee9f9b31cc1859a89f4ffd0bf5c15022a8 WatchSource:0}: Error finding container c073c56af2b6cc0cd381ef1104c3e7ee9f9b31cc1859a89f4ffd0bf5c15022a8: Status 404 returned error can't find the container with id c073c56af2b6cc0cd381ef1104c3e7ee9f9b31cc1859a89f4ffd0bf5c15022a8
May 26 16:01:26.493450 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:26.493365    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod744c275a_8d87_4882_8e2e_73254829ca4a.slice/crio-817d1f5c0fd3a09664b7fccdf850b2cc4e2999a756ac2d8d84386016649fc087 WatchSource:0}: Error finding container 817d1f5c0fd3a09664b7fccdf850b2cc4e2999a756ac2d8d84386016649fc087: Status 404 returned error can't find the container with id 817d1f5c0fd3a09664b7fccdf850b2cc4e2999a756ac2d8d84386016649fc087
May 26 16:01:26.494092 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:26.494070    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02c134e5_0f87_48c2_80ed_427d242f1435.slice/crio-52984daf8a066da7f99b13d0adb77e9d82e9f7c5bb8064f55223d8ef55467698 WatchSource:0}: Error finding container 52984daf8a066da7f99b13d0adb77e9d82e9f7c5bb8064f55223d8ef55467698: Status 404 returned error can't find the container with id 52984daf8a066da7f99b13d0adb77e9d82e9f7c5bb8064f55223d8ef55467698
May 26 16:01:26.496016 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:26.495923    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ad6cab9_b922_456e_ad28_95f40ce6692c.slice/crio-30e58e85cd96842014da8daea0f8f506dd7df192694772aefd6db620e5d91a10 WatchSource:0}: Error finding container 30e58e85cd96842014da8daea0f8f506dd7df192694772aefd6db620e5d91a10: Status 404 returned error can't find the container with id 30e58e85cd96842014da8daea0f8f506dd7df192694772aefd6db620e5d91a10
May 26 16:01:26.500211 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:26.500073    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc54d2f93_a58c_4f73_966d_2584dc7015e8.slice/crio-bc8dbd6306a7e403ece5e140fca51b4f4ea8b1e6bcc8dd6635bf6a2618e9820d WatchSource:0}: Error finding container bc8dbd6306a7e403ece5e140fca51b4f4ea8b1e6bcc8dd6635bf6a2618e9820d: Status 404 returned error can't find the container with id bc8dbd6306a7e403ece5e140fca51b4f4ea8b1e6bcc8dd6635bf6a2618e9820d
May 26 16:01:26.501453 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:26.501065    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10ea1129_78a4_4d29_ab50_50b69b6d30c1.slice/crio-0b54cb8d9712927573b57ad7a5dda45c3249f8ee489ff3f818471427c86d94c5 WatchSource:0}: Error finding container 0b54cb8d9712927573b57ad7a5dda45c3249f8ee489ff3f818471427c86d94c5: Status 404 returned error can't find the container with id 0b54cb8d9712927573b57ad7a5dda45c3249f8ee489ff3f818471427c86d94c5
May 26 16:01:26.502134 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:26.502066    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9823062_c35f_4aca_b363_912ec43ecb40.slice/crio-7f0be8b0552770d5c0e8a55334065caa1da3bb5e6557bf35d97d41e6642e0acf WatchSource:0}: Error finding container 7f0be8b0552770d5c0e8a55334065caa1da3bb5e6557bf35d97d41e6642e0acf: Status 404 returned error can't find the container with id 7f0be8b0552770d5c0e8a55334065caa1da3bb5e6557bf35d97d41e6642e0acf
May 26 16:01:26.503631 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:26.502775    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32dd34b2_8588_46f9_b744_7138c2ac7025.slice/crio-28137f3d548a9ada6ef8fa37dff90febe6f9d3e5d7add0a9364ac785657c8371 WatchSource:0}: Error finding container 28137f3d548a9ada6ef8fa37dff90febe6f9d3e5d7add0a9364ac785657c8371: Status 404 returned error can't find the container with id 28137f3d548a9ada6ef8fa37dff90febe6f9d3e5d7add0a9364ac785657c8371
May 26 16:01:26.504346 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:26.504323    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23cb5db1_ddc3_46a0_bc9d_3133631ffc58.slice/crio-79b870f440c70b1e3292a13d839d704ee1225ddf38bc8e9efe3ff9f8a8014e69 WatchSource:0}: Error finding container 79b870f440c70b1e3292a13d839d704ee1225ddf38bc8e9efe3ff9f8a8014e69: Status 404 returned error can't find the container with id 79b870f440c70b1e3292a13d839d704ee1225ddf38bc8e9efe3ff9f8a8014e69
May 26 16:01:26.589172 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.588972    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8828138d-e7ed-4d5c-ba76-26429679c7b1-metrics-certs\") pod \"network-metrics-daemon-qpzqg\" (UID: \"8828138d-e7ed-4d5c-ba76-26429679c7b1\") " pod="openshift-multus/network-metrics-daemon-qpzqg"
May 26 16:01:26.589365 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:26.589116    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:01:26.589365 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:26.589302    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8828138d-e7ed-4d5c-ba76-26429679c7b1-metrics-certs podName:8828138d-e7ed-4d5c-ba76-26429679c7b1 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:27.589286122 +0000 UTC m=+4.065013450 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8828138d-e7ed-4d5c-ba76-26429679c7b1-metrics-certs") pod "network-metrics-daemon-qpzqg" (UID: "8828138d-e7ed-4d5c-ba76-26429679c7b1") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:01:26.690270 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.690233    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lbgsb\" (UniqueName: \"kubernetes.io/projected/8f54a9e9-f1b3-4073-bf4e-fef6f31dc622-kube-api-access-lbgsb\") pod \"network-check-target-rvlkc\" (UID: \"8f54a9e9-f1b3-4073-bf4e-fef6f31dc622\") " pod="openshift-network-diagnostics/network-check-target-rvlkc"
May 26 16:01:26.690442 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:26.690399    2572 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 16:01:26.690442 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:26.690422    2572 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 16:01:26.690442 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:26.690434    2572 projected.go:194] Error preparing data for projected volume kube-api-access-lbgsb for pod openshift-network-diagnostics/network-check-target-rvlkc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:01:26.690619 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:26.690494    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f54a9e9-f1b3-4073-bf4e-fef6f31dc622-kube-api-access-lbgsb podName:8f54a9e9-f1b3-4073-bf4e-fef6f31dc622 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:27.690476322 +0000 UTC m=+4.166203664 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lbgsb" (UniqueName: "kubernetes.io/projected/8f54a9e9-f1b3-4073-bf4e-fef6f31dc622-kube-api-access-lbgsb") pod "network-check-target-rvlkc" (UID: "8f54a9e9-f1b3-4073-bf4e-fef6f31dc622") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:01:26.828282 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.828250    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-2fdgm"]
May 26 16:01:26.830039 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.830009    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2fdgm"
May 26 16:01:26.830155 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:26.830081    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2fdgm" podUID="81fbb108-f79a-4018-82a4-0b0e29541320"
May 26 16:01:26.891566 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.891532    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/81fbb108-f79a-4018-82a4-0b0e29541320-kubelet-config\") pod \"global-pull-secret-syncer-2fdgm\" (UID: \"81fbb108-f79a-4018-82a4-0b0e29541320\") " pod="kube-system/global-pull-secret-syncer-2fdgm"
May 26 16:01:26.891758 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.891581    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81fbb108-f79a-4018-82a4-0b0e29541320-original-pull-secret\") pod \"global-pull-secret-syncer-2fdgm\" (UID: \"81fbb108-f79a-4018-82a4-0b0e29541320\") " pod="kube-system/global-pull-secret-syncer-2fdgm"
May 26 16:01:26.891758 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.891647    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/81fbb108-f79a-4018-82a4-0b0e29541320-dbus\") pod \"global-pull-secret-syncer-2fdgm\" (UID: \"81fbb108-f79a-4018-82a4-0b0e29541320\") " pod="kube-system/global-pull-secret-syncer-2fdgm"
May 26 16:01:26.992246 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.992211    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/81fbb108-f79a-4018-82a4-0b0e29541320-dbus\") pod \"global-pull-secret-syncer-2fdgm\" (UID: \"81fbb108-f79a-4018-82a4-0b0e29541320\") " pod="kube-system/global-pull-secret-syncer-2fdgm"
May 26 16:01:26.992826 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.992295    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/81fbb108-f79a-4018-82a4-0b0e29541320-kubelet-config\") pod \"global-pull-secret-syncer-2fdgm\" (UID: \"81fbb108-f79a-4018-82a4-0b0e29541320\") " pod="kube-system/global-pull-secret-syncer-2fdgm"
May 26 16:01:26.992826 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.992332    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81fbb108-f79a-4018-82a4-0b0e29541320-original-pull-secret\") pod \"global-pull-secret-syncer-2fdgm\" (UID: \"81fbb108-f79a-4018-82a4-0b0e29541320\") " pod="kube-system/global-pull-secret-syncer-2fdgm"
May 26 16:01:26.992826 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:26.992453    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 26 16:01:26.992826 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:26.992517    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81fbb108-f79a-4018-82a4-0b0e29541320-original-pull-secret podName:81fbb108-f79a-4018-82a4-0b0e29541320 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:27.492496942 +0000 UTC m=+3.968224283 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/81fbb108-f79a-4018-82a4-0b0e29541320-original-pull-secret") pod "global-pull-secret-syncer-2fdgm" (UID: "81fbb108-f79a-4018-82a4-0b0e29541320") : object "kube-system"/"original-pull-secret" not registered
May 26 16:01:26.992826 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.992695    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/81fbb108-f79a-4018-82a4-0b0e29541320-dbus\") pod \"global-pull-secret-syncer-2fdgm\" (UID: \"81fbb108-f79a-4018-82a4-0b0e29541320\") " pod="kube-system/global-pull-secret-syncer-2fdgm"
May 26 16:01:26.992826 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:26.992756    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/81fbb108-f79a-4018-82a4-0b0e29541320-kubelet-config\") pod \"global-pull-secret-syncer-2fdgm\" (UID: \"81fbb108-f79a-4018-82a4-0b0e29541320\") " pod="kube-system/global-pull-secret-syncer-2fdgm"
May 26 16:01:27.016403 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:27.016337    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-25 15:56:25 +0000 UTC" deadline="2028-02-12 14:35:31.47086776 +0000 UTC"
May 26 16:01:27.016403 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:27.016373    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15046h34m4.454498207s"
May 26 16:01:27.075347 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:27.074778    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qpzqg"
May 26 16:01:27.075347 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:27.074928    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qpzqg" podUID="8828138d-e7ed-4d5c-ba76-26429679c7b1"
May 26 16:01:27.095743 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:27.095483    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lvfvd" event={"ID":"02c134e5-0f87-48c2-80ed-427d242f1435","Type":"ContainerStarted","Data":"52984daf8a066da7f99b13d0adb77e9d82e9f7c5bb8064f55223d8ef55467698"}
May 26 16:01:27.097496 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:27.097427    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8hhhx" event={"ID":"744c275a-8d87-4882-8e2e-73254829ca4a","Type":"ContainerStarted","Data":"817d1f5c0fd3a09664b7fccdf850b2cc4e2999a756ac2d8d84386016649fc087"}
May 26 16:01:27.113226 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:27.113075    2572 generic.go:358] "Generic (PLEG): container finished" podID="75eda3ff71f275b7dd3449d695a0e806" containerID="c4773c9284d2d5567acf9d40261a2a253e4537564a178ed4e8b01a9e5b6fed26" exitCode=0
May 26 16:01:27.113226 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:27.113165    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-83.ec2.internal" event={"ID":"75eda3ff71f275b7dd3449d695a0e806","Type":"ContainerDied","Data":"c4773c9284d2d5567acf9d40261a2a253e4537564a178ed4e8b01a9e5b6fed26"}
May 26 16:01:27.134074 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:27.134038    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-83.ec2.internal" event={"ID":"79549bd5c5e41ee1a69fee2a0b6c1223","Type":"ContainerStarted","Data":"c40c13f15221905f56baeb231548c6d133ccc372ad41161a5735c9be08f26862"}
May 26 16:01:27.144794 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:27.144743    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mjznd" event={"ID":"23cb5db1-ddc3-46a0-bc9d-3133631ffc58","Type":"ContainerStarted","Data":"79b870f440c70b1e3292a13d839d704ee1225ddf38bc8e9efe3ff9f8a8014e69"}
May 26 16:01:27.158242 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:27.158149    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qdk28" event={"ID":"10ea1129-78a4-4d29-ab50-50b69b6d30c1","Type":"ContainerStarted","Data":"0b54cb8d9712927573b57ad7a5dda45c3249f8ee489ff3f818471427c86d94c5"}
May 26 16:01:27.163164 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:27.163107    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jlndg" event={"ID":"c54d2f93-a58c-4f73-966d-2584dc7015e8","Type":"ContainerStarted","Data":"bc8dbd6306a7e403ece5e140fca51b4f4ea8b1e6bcc8dd6635bf6a2618e9820d"}
May 26 16:01:27.169412 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:27.169335    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wk2zv" event={"ID":"5ad6cab9-b922-456e-ad28-95f40ce6692c","Type":"ContainerStarted","Data":"30e58e85cd96842014da8daea0f8f506dd7df192694772aefd6db620e5d91a10"}
May 26 16:01:27.173729 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:27.173667    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64xz7" event={"ID":"ef455917-b870-4d4a-b021-45a76d22734d","Type":"ContainerStarted","Data":"c073c56af2b6cc0cd381ef1104c3e7ee9f9b31cc1859a89f4ffd0bf5c15022a8"}
May 26 16:01:27.182584 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:27.182550    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tbphl" event={"ID":"32dd34b2-8588-46f9-b744-7138c2ac7025","Type":"ContainerStarted","Data":"28137f3d548a9ada6ef8fa37dff90febe6f9d3e5d7add0a9364ac785657c8371"}
May 26 16:01:27.191290 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:27.191250    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5" event={"ID":"f9823062-c35f-4aca-b363-912ec43ecb40","Type":"ContainerStarted","Data":"7f0be8b0552770d5c0e8a55334065caa1da3bb5e6557bf35d97d41e6642e0acf"}
May 26 16:01:27.497988 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:27.497372    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81fbb108-f79a-4018-82a4-0b0e29541320-original-pull-secret\") pod \"global-pull-secret-syncer-2fdgm\" (UID: \"81fbb108-f79a-4018-82a4-0b0e29541320\") " pod="kube-system/global-pull-secret-syncer-2fdgm"
May 26 16:01:27.497988 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:27.497530    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 26 16:01:27.497988 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:27.497598    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81fbb108-f79a-4018-82a4-0b0e29541320-original-pull-secret podName:81fbb108-f79a-4018-82a4-0b0e29541320 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:28.497575717 +0000 UTC m=+4.973303051 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/81fbb108-f79a-4018-82a4-0b0e29541320-original-pull-secret") pod "global-pull-secret-syncer-2fdgm" (UID: "81fbb108-f79a-4018-82a4-0b0e29541320") : object "kube-system"/"original-pull-secret" not registered
May 26 16:01:27.598704 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:27.598125    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8828138d-e7ed-4d5c-ba76-26429679c7b1-metrics-certs\") pod \"network-metrics-daemon-qpzqg\" (UID: \"8828138d-e7ed-4d5c-ba76-26429679c7b1\") " pod="openshift-multus/network-metrics-daemon-qpzqg"
May 26 16:01:27.598704 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:27.598380    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:01:27.598704 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:27.598461    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8828138d-e7ed-4d5c-ba76-26429679c7b1-metrics-certs podName:8828138d-e7ed-4d5c-ba76-26429679c7b1 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:29.598440082 +0000 UTC m=+6.074167436 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8828138d-e7ed-4d5c-ba76-26429679c7b1-metrics-certs") pod "network-metrics-daemon-qpzqg" (UID: "8828138d-e7ed-4d5c-ba76-26429679c7b1") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:01:27.699124 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:27.699084    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lbgsb\" (UniqueName: \"kubernetes.io/projected/8f54a9e9-f1b3-4073-bf4e-fef6f31dc622-kube-api-access-lbgsb\") pod \"network-check-target-rvlkc\" (UID: \"8f54a9e9-f1b3-4073-bf4e-fef6f31dc622\") " pod="openshift-network-diagnostics/network-check-target-rvlkc"
May 26 16:01:27.699310 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:27.699263    2572 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 16:01:27.699310 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:27.699282    2572 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 16:01:27.699310 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:27.699294    2572 projected.go:194] Error preparing data for projected volume kube-api-access-lbgsb for pod openshift-network-diagnostics/network-check-target-rvlkc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:01:27.699467 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:27.699355    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f54a9e9-f1b3-4073-bf4e-fef6f31dc622-kube-api-access-lbgsb podName:8f54a9e9-f1b3-4073-bf4e-fef6f31dc622 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:29.699337118 +0000 UTC m=+6.175064449 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lbgsb" (UniqueName: "kubernetes.io/projected/8f54a9e9-f1b3-4073-bf4e-fef6f31dc622-kube-api-access-lbgsb") pod "network-check-target-rvlkc" (UID: "8f54a9e9-f1b3-4073-bf4e-fef6f31dc622") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:01:28.076890 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:28.076801    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rvlkc"
May 26 16:01:28.077325 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:28.076927    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rvlkc" podUID="8f54a9e9-f1b3-4073-bf4e-fef6f31dc622"
May 26 16:01:28.077709 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:28.077452    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2fdgm"
May 26 16:01:28.077709 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:28.077551    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2fdgm" podUID="81fbb108-f79a-4018-82a4-0b0e29541320"
May 26 16:01:28.205992 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:28.205869    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-83.ec2.internal" event={"ID":"75eda3ff71f275b7dd3449d695a0e806","Type":"ContainerStarted","Data":"2ee2053f02011503ea4ee3807e7265097d234c1c5682d69e782ec76bf595c83a"}
May 26 16:01:28.217825 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:28.217756    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-83.ec2.internal" podStartSLOduration=3.217737023 podStartE2EDuration="3.217737023s" podCreationTimestamp="2026-05-26 16:01:25 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:01:27.146373864 +0000 UTC m=+3.622101215" watchObservedRunningTime="2026-05-26 16:01:28.217737023 +0000 UTC m=+4.693464383"
May 26 16:01:28.507617 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:28.507007    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81fbb108-f79a-4018-82a4-0b0e29541320-original-pull-secret\") pod \"global-pull-secret-syncer-2fdgm\" (UID: \"81fbb108-f79a-4018-82a4-0b0e29541320\") " pod="kube-system/global-pull-secret-syncer-2fdgm"
May 26 16:01:28.507617 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:28.507149    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 26 16:01:28.507617 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:28.507205    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81fbb108-f79a-4018-82a4-0b0e29541320-original-pull-secret podName:81fbb108-f79a-4018-82a4-0b0e29541320 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:30.507186958 +0000 UTC m=+6.982914289 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/81fbb108-f79a-4018-82a4-0b0e29541320-original-pull-secret") pod "global-pull-secret-syncer-2fdgm" (UID: "81fbb108-f79a-4018-82a4-0b0e29541320") : object "kube-system"/"original-pull-secret" not registered
May 26 16:01:29.075475 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:29.075441    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qpzqg"
May 26 16:01:29.075766 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:29.075723    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qpzqg" podUID="8828138d-e7ed-4d5c-ba76-26429679c7b1"
May 26 16:01:29.617250 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:29.617201    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8828138d-e7ed-4d5c-ba76-26429679c7b1-metrics-certs\") pod \"network-metrics-daemon-qpzqg\" (UID: \"8828138d-e7ed-4d5c-ba76-26429679c7b1\") " pod="openshift-multus/network-metrics-daemon-qpzqg"
May 26 16:01:29.617761 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:29.617386    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:01:29.617761 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:29.617459    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8828138d-e7ed-4d5c-ba76-26429679c7b1-metrics-certs podName:8828138d-e7ed-4d5c-ba76-26429679c7b1 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:33.617439806 +0000 UTC m=+10.093167137 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8828138d-e7ed-4d5c-ba76-26429679c7b1-metrics-certs") pod "network-metrics-daemon-qpzqg" (UID: "8828138d-e7ed-4d5c-ba76-26429679c7b1") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:01:29.718046 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:29.718009    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lbgsb\" (UniqueName: \"kubernetes.io/projected/8f54a9e9-f1b3-4073-bf4e-fef6f31dc622-kube-api-access-lbgsb\") pod \"network-check-target-rvlkc\" (UID: \"8f54a9e9-f1b3-4073-bf4e-fef6f31dc622\") " pod="openshift-network-diagnostics/network-check-target-rvlkc"
May 26 16:01:29.718206 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:29.718194    2572 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 16:01:29.718434 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:29.718215    2572 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 16:01:29.718434 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:29.718228    2572 projected.go:194] Error preparing data for projected volume kube-api-access-lbgsb for pod openshift-network-diagnostics/network-check-target-rvlkc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:01:29.718434 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:29.718284    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f54a9e9-f1b3-4073-bf4e-fef6f31dc622-kube-api-access-lbgsb podName:8f54a9e9-f1b3-4073-bf4e-fef6f31dc622 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:33.718267684 +0000 UTC m=+10.193995018 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lbgsb" (UniqueName: "kubernetes.io/projected/8f54a9e9-f1b3-4073-bf4e-fef6f31dc622-kube-api-access-lbgsb") pod "network-check-target-rvlkc" (UID: "8f54a9e9-f1b3-4073-bf4e-fef6f31dc622") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:01:30.076289 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:30.075518    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2fdgm"
May 26 16:01:30.076289 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:30.075671    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2fdgm" podUID="81fbb108-f79a-4018-82a4-0b0e29541320"
May 26 16:01:30.076289 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:30.076105    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rvlkc"
May 26 16:01:30.076289 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:30.076204    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rvlkc" podUID="8f54a9e9-f1b3-4073-bf4e-fef6f31dc622"
May 26 16:01:30.526421 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:30.525747    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81fbb108-f79a-4018-82a4-0b0e29541320-original-pull-secret\") pod \"global-pull-secret-syncer-2fdgm\" (UID: \"81fbb108-f79a-4018-82a4-0b0e29541320\") " pod="kube-system/global-pull-secret-syncer-2fdgm"
May 26 16:01:30.526421 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:30.525974    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 26 16:01:30.526421 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:30.526043    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81fbb108-f79a-4018-82a4-0b0e29541320-original-pull-secret podName:81fbb108-f79a-4018-82a4-0b0e29541320 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:34.526022935 +0000 UTC m=+11.001750269 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/81fbb108-f79a-4018-82a4-0b0e29541320-original-pull-secret") pod "global-pull-secret-syncer-2fdgm" (UID: "81fbb108-f79a-4018-82a4-0b0e29541320") : object "kube-system"/"original-pull-secret" not registered
May 26 16:01:31.075103 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:31.075068    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qpzqg"
May 26 16:01:31.075638 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:31.075230    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qpzqg" podUID="8828138d-e7ed-4d5c-ba76-26429679c7b1"
May 26 16:01:32.075502 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:32.075265    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2fdgm"
May 26 16:01:32.075502 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:32.075336    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rvlkc"
May 26 16:01:32.075502 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:32.075419    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2fdgm" podUID="81fbb108-f79a-4018-82a4-0b0e29541320"
May 26 16:01:32.076065 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:32.075593    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rvlkc" podUID="8f54a9e9-f1b3-4073-bf4e-fef6f31dc622"
May 26 16:01:33.075074 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:33.075033    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qpzqg"
May 26 16:01:33.075271 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:33.075197    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qpzqg" podUID="8828138d-e7ed-4d5c-ba76-26429679c7b1"
May 26 16:01:33.655246 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:33.655134    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8828138d-e7ed-4d5c-ba76-26429679c7b1-metrics-certs\") pod \"network-metrics-daemon-qpzqg\" (UID: \"8828138d-e7ed-4d5c-ba76-26429679c7b1\") " pod="openshift-multus/network-metrics-daemon-qpzqg"
May 26 16:01:33.655714 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:33.655301    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:01:33.655714 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:33.655386    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8828138d-e7ed-4d5c-ba76-26429679c7b1-metrics-certs podName:8828138d-e7ed-4d5c-ba76-26429679c7b1 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:41.655366231 +0000 UTC m=+18.131093584 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8828138d-e7ed-4d5c-ba76-26429679c7b1-metrics-certs") pod "network-metrics-daemon-qpzqg" (UID: "8828138d-e7ed-4d5c-ba76-26429679c7b1") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:01:33.756555 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:33.756496    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lbgsb\" (UniqueName: \"kubernetes.io/projected/8f54a9e9-f1b3-4073-bf4e-fef6f31dc622-kube-api-access-lbgsb\") pod \"network-check-target-rvlkc\" (UID: \"8f54a9e9-f1b3-4073-bf4e-fef6f31dc622\") " pod="openshift-network-diagnostics/network-check-target-rvlkc"
May 26 16:01:33.756726 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:33.756704    2572 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 16:01:33.756726 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:33.756722    2572 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 16:01:33.756854 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:33.756735    2572 projected.go:194] Error preparing data for projected volume kube-api-access-lbgsb for pod openshift-network-diagnostics/network-check-target-rvlkc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:01:33.756854 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:33.756783    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f54a9e9-f1b3-4073-bf4e-fef6f31dc622-kube-api-access-lbgsb podName:8f54a9e9-f1b3-4073-bf4e-fef6f31dc622 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:41.756769858 +0000 UTC m=+18.232497185 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lbgsb" (UniqueName: "kubernetes.io/projected/8f54a9e9-f1b3-4073-bf4e-fef6f31dc622-kube-api-access-lbgsb") pod "network-check-target-rvlkc" (UID: "8f54a9e9-f1b3-4073-bf4e-fef6f31dc622") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:01:34.075618 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:34.075526    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rvlkc"
May 26 16:01:34.075788 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:34.075679    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rvlkc" podUID="8f54a9e9-f1b3-4073-bf4e-fef6f31dc622"
May 26 16:01:34.076971 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:34.076784    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2fdgm"
May 26 16:01:34.076971 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:34.076923    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2fdgm" podUID="81fbb108-f79a-4018-82a4-0b0e29541320"
May 26 16:01:34.562365 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:34.562274    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81fbb108-f79a-4018-82a4-0b0e29541320-original-pull-secret\") pod \"global-pull-secret-syncer-2fdgm\" (UID: \"81fbb108-f79a-4018-82a4-0b0e29541320\") " pod="kube-system/global-pull-secret-syncer-2fdgm"
May 26 16:01:34.562526 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:34.562423    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 26 16:01:34.562526 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:34.562503    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81fbb108-f79a-4018-82a4-0b0e29541320-original-pull-secret podName:81fbb108-f79a-4018-82a4-0b0e29541320 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:42.562482934 +0000 UTC m=+19.038210276 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/81fbb108-f79a-4018-82a4-0b0e29541320-original-pull-secret") pod "global-pull-secret-syncer-2fdgm" (UID: "81fbb108-f79a-4018-82a4-0b0e29541320") : object "kube-system"/"original-pull-secret" not registered
May 26 16:01:35.075633 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:35.075590    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qpzqg"
May 26 16:01:35.076072 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:35.075731    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qpzqg" podUID="8828138d-e7ed-4d5c-ba76-26429679c7b1"
May 26 16:01:36.074866 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:36.074829    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rvlkc"
May 26 16:01:36.075051 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:36.074956    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rvlkc" podUID="8f54a9e9-f1b3-4073-bf4e-fef6f31dc622"
May 26 16:01:36.075051 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:36.075021    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2fdgm"
May 26 16:01:36.075170 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:36.075134    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2fdgm" podUID="81fbb108-f79a-4018-82a4-0b0e29541320"
May 26 16:01:37.074966 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:37.074923    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qpzqg"
May 26 16:01:37.075438 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:37.075050    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qpzqg" podUID="8828138d-e7ed-4d5c-ba76-26429679c7b1"
May 26 16:01:38.074984 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:38.074944    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rvlkc"
May 26 16:01:38.075476 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:38.074992    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2fdgm"
May 26 16:01:38.075476 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:38.075075    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rvlkc" podUID="8f54a9e9-f1b3-4073-bf4e-fef6f31dc622"
May 26 16:01:38.075476 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:38.075179    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2fdgm" podUID="81fbb108-f79a-4018-82a4-0b0e29541320"
May 26 16:01:39.075532 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:39.075496    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qpzqg"
May 26 16:01:39.076002 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:39.075657    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qpzqg" podUID="8828138d-e7ed-4d5c-ba76-26429679c7b1"
May 26 16:01:40.075433 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:40.075397    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rvlkc"
May 26 16:01:40.075635 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:40.075534    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rvlkc" podUID="8f54a9e9-f1b3-4073-bf4e-fef6f31dc622"
May 26 16:01:40.075635 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:40.075612    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2fdgm"
May 26 16:01:40.076026 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:40.075732    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2fdgm" podUID="81fbb108-f79a-4018-82a4-0b0e29541320"
May 26 16:01:41.075530 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:41.075487    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qpzqg"
May 26 16:01:41.075777 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:41.075622    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qpzqg" podUID="8828138d-e7ed-4d5c-ba76-26429679c7b1"
May 26 16:01:41.715576 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:41.715535    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8828138d-e7ed-4d5c-ba76-26429679c7b1-metrics-certs\") pod \"network-metrics-daemon-qpzqg\" (UID: \"8828138d-e7ed-4d5c-ba76-26429679c7b1\") " pod="openshift-multus/network-metrics-daemon-qpzqg"
May 26 16:01:41.715755 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:41.715682    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:01:41.715755 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:41.715748    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8828138d-e7ed-4d5c-ba76-26429679c7b1-metrics-certs podName:8828138d-e7ed-4d5c-ba76-26429679c7b1 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:57.715728376 +0000 UTC m=+34.191455706 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8828138d-e7ed-4d5c-ba76-26429679c7b1-metrics-certs") pod "network-metrics-daemon-qpzqg" (UID: "8828138d-e7ed-4d5c-ba76-26429679c7b1") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:01:41.816523 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:41.816481    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lbgsb\" (UniqueName: \"kubernetes.io/projected/8f54a9e9-f1b3-4073-bf4e-fef6f31dc622-kube-api-access-lbgsb\") pod \"network-check-target-rvlkc\" (UID: \"8f54a9e9-f1b3-4073-bf4e-fef6f31dc622\") " pod="openshift-network-diagnostics/network-check-target-rvlkc"
May 26 16:01:41.816726 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:41.816705    2572 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 16:01:41.816790 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:41.816735    2572 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 16:01:41.816790 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:41.816749    2572 projected.go:194] Error preparing data for projected volume kube-api-access-lbgsb for pod openshift-network-diagnostics/network-check-target-rvlkc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:01:41.816868 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:41.816817    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f54a9e9-f1b3-4073-bf4e-fef6f31dc622-kube-api-access-lbgsb podName:8f54a9e9-f1b3-4073-bf4e-fef6f31dc622 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:57.816798295 +0000 UTC m=+34.292525638 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lbgsb" (UniqueName: "kubernetes.io/projected/8f54a9e9-f1b3-4073-bf4e-fef6f31dc622-kube-api-access-lbgsb") pod "network-check-target-rvlkc" (UID: "8f54a9e9-f1b3-4073-bf4e-fef6f31dc622") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:01:42.075522 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:42.075442    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rvlkc"
May 26 16:01:42.075691 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:42.075442    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2fdgm"
May 26 16:01:42.075691 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:42.075571    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rvlkc" podUID="8f54a9e9-f1b3-4073-bf4e-fef6f31dc622"
May 26 16:01:42.075815 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:42.075749    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2fdgm" podUID="81fbb108-f79a-4018-82a4-0b0e29541320"
May 26 16:01:42.622295 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:42.622250    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81fbb108-f79a-4018-82a4-0b0e29541320-original-pull-secret\") pod \"global-pull-secret-syncer-2fdgm\" (UID: \"81fbb108-f79a-4018-82a4-0b0e29541320\") " pod="kube-system/global-pull-secret-syncer-2fdgm"
May 26 16:01:42.622474 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:42.622415    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 26 16:01:42.622551 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:42.622490    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81fbb108-f79a-4018-82a4-0b0e29541320-original-pull-secret podName:81fbb108-f79a-4018-82a4-0b0e29541320 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:58.622469627 +0000 UTC m=+35.098196970 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/81fbb108-f79a-4018-82a4-0b0e29541320-original-pull-secret") pod "global-pull-secret-syncer-2fdgm" (UID: "81fbb108-f79a-4018-82a4-0b0e29541320") : object "kube-system"/"original-pull-secret" not registered
May 26 16:01:43.075255 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:43.075178    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qpzqg"
May 26 16:01:43.075402 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:43.075314    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qpzqg" podUID="8828138d-e7ed-4d5c-ba76-26429679c7b1"
May 26 16:01:44.076091 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:44.076060    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rvlkc"
May 26 16:01:44.076525 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:44.076147    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rvlkc" podUID="8f54a9e9-f1b3-4073-bf4e-fef6f31dc622"
May 26 16:01:44.076525 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:44.076159    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2fdgm"
May 26 16:01:44.076525 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:44.076277    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2fdgm" podUID="81fbb108-f79a-4018-82a4-0b0e29541320"
May 26 16:01:45.075392 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:45.075071    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qpzqg"
May 26 16:01:45.075517 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:45.075484    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qpzqg" podUID="8828138d-e7ed-4d5c-ba76-26429679c7b1"
May 26 16:01:45.236134 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:45.236099    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qdk28" event={"ID":"10ea1129-78a4-4d29-ab50-50b69b6d30c1","Type":"ContainerStarted","Data":"5f5db0cc5244a122f41227c8efef137f0e81f4e0d4652cb0a4bf1b619799d53a"}
May 26 16:01:45.237867 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:45.237838    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wk2zv" event={"ID":"5ad6cab9-b922-456e-ad28-95f40ce6692c","Type":"ContainerStarted","Data":"b10153100a1f93fbfbb258a87a88d3b897a5942e98f7d8ddd10e11d0b13b1683"}
May 26 16:01:45.239585 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:45.239545    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64xz7" event={"ID":"ef455917-b870-4d4a-b021-45a76d22734d","Type":"ContainerStarted","Data":"86ee26175efa7f5548ffff12d275a650ddec2e4722050815ca140201d7480b07"}
May 26 16:01:45.241172 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:45.241145    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tbphl" event={"ID":"32dd34b2-8588-46f9-b744-7138c2ac7025","Type":"ContainerStarted","Data":"e4ed1115f0e291ce526638fb85942065cfa5d6fd63bac05ef9cb54f50feb9366"}
May 26 16:01:45.248222 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:45.248097    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-83.ec2.internal" podStartSLOduration=20.248078565 podStartE2EDuration="20.248078565s" podCreationTimestamp="2026-05-26 16:01:25 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:01:28.218764258 +0000 UTC m=+4.694491609" watchObservedRunningTime="2026-05-26 16:01:45.248078565 +0000 UTC m=+21.723805915"
May 26 16:01:45.248832 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:45.248803    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5" event={"ID":"f9823062-c35f-4aca-b363-912ec43ecb40","Type":"ContainerStarted","Data":"71bde7132e519b089a26f4f1e589fa4feb9acbb3de2defe820d8be4a3568a171"}
May 26 16:01:45.248963 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:45.248836    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5" event={"ID":"f9823062-c35f-4aca-b363-912ec43ecb40","Type":"ContainerStarted","Data":"8b0b48f409080304d4b382f900cc0a0580b0658b3b3abc9f8441901bb8ae0a43"}
May 26 16:01:45.248963 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:45.248849    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5" event={"ID":"f9823062-c35f-4aca-b363-912ec43ecb40","Type":"ContainerStarted","Data":"cd3bd23c8f1a8a4416b59e53a28b73c7c2ff798c7246897e29c29c14daf58bdf"}
May 26 16:01:45.248963 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:45.248859    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5" event={"ID":"f9823062-c35f-4aca-b363-912ec43ecb40","Type":"ContainerStarted","Data":"1ee7583344ac5af1dfc794f38ee464099b6bde83fdc4e8adc0d8e043339c1d25"}
May 26 16:01:45.248963 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:45.248871    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5" event={"ID":"f9823062-c35f-4aca-b363-912ec43ecb40","Type":"ContainerStarted","Data":"ac4faabf23a0ca87d61f822a1187e3665fe55f902ef30b46a5f65e8df7768fc4"}
May 26 16:01:45.248963 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:45.248882    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5" event={"ID":"f9823062-c35f-4aca-b363-912ec43ecb40","Type":"ContainerStarted","Data":"e32a6d1bf2a4d16bbbc58a2560fdb62858290e3319c50689c4c8c70228fd2e82"}
May 26 16:01:45.249208 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:45.249137    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qdk28" podStartSLOduration=3.230184558 podStartE2EDuration="21.249118393s" podCreationTimestamp="2026-05-26 16:01:24 +0000 UTC" firstStartedPulling="2026-05-26 16:01:26.503256207 +0000 UTC m=+2.978983540" lastFinishedPulling="2026-05-26 16:01:44.522190042 +0000 UTC m=+20.997917375" observedRunningTime="2026-05-26 16:01:45.247998578 +0000 UTC m=+21.723725931" watchObservedRunningTime="2026-05-26 16:01:45.249118393 +0000 UTC m=+21.724845745"
May 26 16:01:45.250815 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:45.250788    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lvfvd" event={"ID":"02c134e5-0f87-48c2-80ed-427d242f1435","Type":"ContainerStarted","Data":"b542d7e63d8f2ef44fbc2850fc410fc13b23a22d42acf0370bee0ddf0fc7ad6c"}
May 26 16:01:45.252194 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:45.252172    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8hhhx" event={"ID":"744c275a-8d87-4882-8e2e-73254829ca4a","Type":"ContainerStarted","Data":"3ef0146ced199a26e1d099f40187f6a949aedf175f0d93e6cd72a9f626524d63"}
May 26 16:01:45.253868 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:45.253845    2572 generic.go:358] "Generic (PLEG): container finished" podID="23cb5db1-ddc3-46a0-bc9d-3133631ffc58" containerID="16281ce80256ebc14a52941c8f1cec7f29fdd59ae9bd0f1bdd658e047f28f2fd" exitCode=0
May 26 16:01:45.253971 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:45.253884    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mjznd" event={"ID":"23cb5db1-ddc3-46a0-bc9d-3133631ffc58","Type":"ContainerDied","Data":"16281ce80256ebc14a52941c8f1cec7f29fdd59ae9bd0f1bdd658e047f28f2fd"}
May 26 16:01:45.259297 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:45.259250    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-tbphl" podStartSLOduration=11.248166868 podStartE2EDuration="21.259234467s" podCreationTimestamp="2026-05-26 16:01:24 +0000 UTC" firstStartedPulling="2026-05-26 16:01:26.505345271 +0000 UTC m=+2.981072599" lastFinishedPulling="2026-05-26 16:01:36.516412863 +0000 UTC m=+12.992140198" observedRunningTime="2026-05-26 16:01:45.259152105 +0000 UTC m=+21.734879447" watchObservedRunningTime="2026-05-26 16:01:45.259234467 +0000 UTC m=+21.734961818"
May 26 16:01:45.273988 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:45.273938    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-wk2zv" podStartSLOduration=3.1929606 podStartE2EDuration="21.273921842s" podCreationTimestamp="2026-05-26 16:01:24 +0000 UTC" firstStartedPulling="2026-05-26 16:01:26.498871684 +0000 UTC m=+2.974599026" lastFinishedPulling="2026-05-26 16:01:44.579832926 +0000 UTC m=+21.055560268" observedRunningTime="2026-05-26 16:01:45.273331289 +0000 UTC m=+21.749058645" watchObservedRunningTime="2026-05-26 16:01:45.273921842 +0000 UTC m=+21.749649192"
May 26 16:01:45.298701 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:45.298641    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-8hhhx" podStartSLOduration=3.270317994 podStartE2EDuration="21.298625248s" podCreationTimestamp="2026-05-26 16:01:24 +0000 UTC" firstStartedPulling="2026-05-26 16:01:26.495061949 +0000 UTC m=+2.970789295" lastFinishedPulling="2026-05-26 16:01:44.523369204 +0000 UTC m=+20.999096549" observedRunningTime="2026-05-26 16:01:45.286313647 +0000 UTC m=+21.762040998" watchObservedRunningTime="2026-05-26 16:01:45.298625248 +0000 UTC m=+21.774352595"
May 26 16:01:45.315407 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:45.315361    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-lvfvd" podStartSLOduration=3.290582032 podStartE2EDuration="21.315347466s" podCreationTimestamp="2026-05-26 16:01:24 +0000 UTC" firstStartedPulling="2026-05-26 16:01:26.496980112 +0000 UTC m=+2.972707441" lastFinishedPulling="2026-05-26 16:01:44.521745544 +0000 UTC m=+20.997472875" observedRunningTime="2026-05-26 16:01:45.298371936 +0000 UTC m=+21.774099286" watchObservedRunningTime="2026-05-26 16:01:45.315347466 +0000 UTC m=+21.791074871"
May 26 16:01:46.028623 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:46.028582    2572 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
May 26 16:01:46.048010 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:46.047912    2572 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-26T16:01:46.028621011Z","UUID":"8c00fe2e-aee9-4628-908a-92da7c988d1a","Handler":null,"Name":"","Endpoint":""}
May 26 16:01:46.049557 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:46.049535    2572 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
May 26 16:01:46.049691 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:46.049565    2572 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
May 26 16:01:46.075031 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:46.075000    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rvlkc"
May 26 16:01:46.075196 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:46.075108    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rvlkc" podUID="8f54a9e9-f1b3-4073-bf4e-fef6f31dc622"
May 26 16:01:46.075196 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:46.075166    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2fdgm"
May 26 16:01:46.075298 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:46.075267    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2fdgm" podUID="81fbb108-f79a-4018-82a4-0b0e29541320"
May 26 16:01:46.214653 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:46.214557    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-tbphl"
May 26 16:01:46.226678 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:46.226650    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-tbphl"
May 26 16:01:46.227339 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:46.227319    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-tbphl"
May 26 16:01:46.257777 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:46.257743    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jlndg" event={"ID":"c54d2f93-a58c-4f73-966d-2584dc7015e8","Type":"ContainerStarted","Data":"7269cb6d3d4aae62a4bb825ba2205ec8bcbc37e944d0dfa35ae4a46864947512"}
May 26 16:01:46.259648 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:46.259618    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64xz7" event={"ID":"ef455917-b870-4d4a-b021-45a76d22734d","Type":"ContainerStarted","Data":"0a3c0404a7d32bae7b42a45e3f1c4ec5fb593e2ef058dcab1f03b32aaef07e85"}
May 26 16:01:46.260850 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:46.260831    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-tbphl"
May 26 16:01:46.272024 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:46.271977    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-jlndg" podStartSLOduration=4.252069428 podStartE2EDuration="22.271958355s" podCreationTimestamp="2026-05-26 16:01:24 +0000 UTC" firstStartedPulling="2026-05-26 16:01:26.50234376 +0000 UTC m=+2.978071091" lastFinishedPulling="2026-05-26 16:01:44.522232686 +0000 UTC m=+20.997960018" observedRunningTime="2026-05-26 16:01:46.271043547 +0000 UTC m=+22.746770899" watchObservedRunningTime="2026-05-26 16:01:46.271958355 +0000 UTC m=+22.747685712"
May 26 16:01:47.074810 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:47.074771    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qpzqg"
May 26 16:01:47.074982 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:47.074886    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qpzqg" podUID="8828138d-e7ed-4d5c-ba76-26429679c7b1"
May 26 16:01:47.263914 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:47.263830    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64xz7" event={"ID":"ef455917-b870-4d4a-b021-45a76d22734d","Type":"ContainerStarted","Data":"f2a233853bbd39d5a6ec4c2928276c2a569e9185f702cdd2292433f8b2729b83"}
May 26 16:01:47.281037 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:47.280808    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64xz7" podStartSLOduration=2.830745723 podStartE2EDuration="23.280793193s" podCreationTimestamp="2026-05-26 16:01:24 +0000 UTC" firstStartedPulling="2026-05-26 16:01:26.493658278 +0000 UTC m=+2.969385615" lastFinishedPulling="2026-05-26 16:01:46.943705753 +0000 UTC m=+23.419433085" observedRunningTime="2026-05-26 16:01:47.280239419 +0000 UTC m=+23.755966769" watchObservedRunningTime="2026-05-26 16:01:47.280793193 +0000 UTC m=+23.756520544"
May 26 16:01:48.075127 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:48.075094    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rvlkc"
May 26 16:01:48.075127 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:48.075118    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2fdgm"
May 26 16:01:48.075368 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:48.075220    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2fdgm" podUID="81fbb108-f79a-4018-82a4-0b0e29541320"
May 26 16:01:48.075438 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:48.075359    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rvlkc" podUID="8f54a9e9-f1b3-4073-bf4e-fef6f31dc622"
May 26 16:01:48.270279 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:48.270240    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5" event={"ID":"f9823062-c35f-4aca-b363-912ec43ecb40","Type":"ContainerStarted","Data":"c0c8ee1b4e32876826307e3bcd3cde9c022047283ae5d30d64be0636464a4023"}
May 26 16:01:49.075245 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:49.075210    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qpzqg"
May 26 16:01:49.075432 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:49.075352    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qpzqg" podUID="8828138d-e7ed-4d5c-ba76-26429679c7b1"
May 26 16:01:50.075267 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:50.075003    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rvlkc"
May 26 16:01:50.075267 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:50.075040    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2fdgm"
May 26 16:01:50.078934 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:50.075621    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rvlkc" podUID="8f54a9e9-f1b3-4073-bf4e-fef6f31dc622"
May 26 16:01:50.078934 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:50.075856    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2fdgm" podUID="81fbb108-f79a-4018-82a4-0b0e29541320"
May 26 16:01:50.275214 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:50.275175    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mjznd" event={"ID":"23cb5db1-ddc3-46a0-bc9d-3133631ffc58","Type":"ContainerStarted","Data":"f2aa0e8a508e95d996bc37d2ef031c58c56b0940cfac5770d9a8ae381c62131c"}
May 26 16:01:50.277957 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:50.277923    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5" event={"ID":"f9823062-c35f-4aca-b363-912ec43ecb40","Type":"ContainerStarted","Data":"9577cc40ba8945b762e13e836f7bd8fe29ac10321593b7b27e41966526fc110a"}
May 26 16:01:50.278265 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:50.278248    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:50.293764 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:50.293739    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:50.320316 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:50.320267    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5" podStartSLOduration=8.225008923 podStartE2EDuration="26.320252507s" podCreationTimestamp="2026-05-26 16:01:24 +0000 UTC" firstStartedPulling="2026-05-26 16:01:26.503758214 +0000 UTC m=+2.979485552" lastFinishedPulling="2026-05-26 16:01:44.599001793 +0000 UTC m=+21.074729136" observedRunningTime="2026-05-26 16:01:50.319080148 +0000 UTC m=+26.794807498" watchObservedRunningTime="2026-05-26 16:01:50.320252507 +0000 UTC m=+26.795979857"
May 26 16:01:51.075019 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:51.074983    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qpzqg"
May 26 16:01:51.075318 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:51.075127    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qpzqg" podUID="8828138d-e7ed-4d5c-ba76-26429679c7b1"
May 26 16:01:51.281082 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:51.281051    2572 generic.go:358] "Generic (PLEG): container finished" podID="23cb5db1-ddc3-46a0-bc9d-3133631ffc58" containerID="f2aa0e8a508e95d996bc37d2ef031c58c56b0940cfac5770d9a8ae381c62131c" exitCode=0
May 26 16:01:51.281223 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:51.281126    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mjznd" event={"ID":"23cb5db1-ddc3-46a0-bc9d-3133631ffc58","Type":"ContainerDied","Data":"f2aa0e8a508e95d996bc37d2ef031c58c56b0940cfac5770d9a8ae381c62131c"}
May 26 16:01:51.281351 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:51.281335    2572 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
May 26 16:01:51.282374 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:51.281837    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:51.297764 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:51.297739    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:51.805278 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:51.805248    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2fdgm"]
May 26 16:01:51.805431 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:51.805366    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2fdgm"
May 26 16:01:51.805474 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:51.805445    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2fdgm" podUID="81fbb108-f79a-4018-82a4-0b0e29541320"
May 26 16:01:51.808535 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:51.808503    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rvlkc"]
May 26 16:01:51.808713 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:51.808653    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rvlkc"
May 26 16:01:51.808783 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:51.808762    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rvlkc" podUID="8f54a9e9-f1b3-4073-bf4e-fef6f31dc622"
May 26 16:01:51.820167 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:51.820139    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qpzqg"]
May 26 16:01:51.820314 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:51.820255    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qpzqg"
May 26 16:01:51.820364 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:51.820346    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qpzqg" podUID="8828138d-e7ed-4d5c-ba76-26429679c7b1"
May 26 16:01:52.284656 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:52.284623    2572 generic.go:358] "Generic (PLEG): container finished" podID="23cb5db1-ddc3-46a0-bc9d-3133631ffc58" containerID="2f6c3ec3fe3f4be68d0a04d719898f873dc1312bd3d2b201841374d002a27dd2" exitCode=0
May 26 16:01:52.285129 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:52.284722    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mjznd" event={"ID":"23cb5db1-ddc3-46a0-bc9d-3133631ffc58","Type":"ContainerDied","Data":"2f6c3ec3fe3f4be68d0a04d719898f873dc1312bd3d2b201841374d002a27dd2"}
May 26 16:01:52.285129 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:52.284936    2572 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
May 26 16:01:52.607884 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:52.607797    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:01:53.075045 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:53.075009    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2fdgm"
May 26 16:01:53.075212 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:53.075125    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2fdgm" podUID="81fbb108-f79a-4018-82a4-0b0e29541320"
May 26 16:01:53.075212 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:53.075177    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rvlkc"
May 26 16:01:53.075291 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:53.075265    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rvlkc" podUID="8f54a9e9-f1b3-4073-bf4e-fef6f31dc622"
May 26 16:01:53.288677 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:53.288573    2572 generic.go:358] "Generic (PLEG): container finished" podID="23cb5db1-ddc3-46a0-bc9d-3133631ffc58" containerID="76b5d04538133045dde9c429e18686ff5e77d7c27b575d8f6cb7df8bde320467" exitCode=0
May 26 16:01:53.288677 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:53.288662    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mjznd" event={"ID":"23cb5db1-ddc3-46a0-bc9d-3133631ffc58","Type":"ContainerDied","Data":"76b5d04538133045dde9c429e18686ff5e77d7c27b575d8f6cb7df8bde320467"}
May 26 16:01:54.076031 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:54.075828    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qpzqg"
May 26 16:01:54.076401 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:54.076139    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qpzqg" podUID="8828138d-e7ed-4d5c-ba76-26429679c7b1"
May 26 16:01:54.302931 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:54.302842    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5" podUID="f9823062-c35f-4aca-b363-912ec43ecb40" containerName="ovnkube-controller" probeResult="failure" output=""
May 26 16:01:55.075139 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:55.075104    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2fdgm"
May 26 16:01:55.075327 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:55.075104    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rvlkc"
May 26 16:01:55.075327 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:55.075237    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2fdgm" podUID="81fbb108-f79a-4018-82a4-0b0e29541320"
May 26 16:01:55.075444 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:55.075327    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rvlkc" podUID="8f54a9e9-f1b3-4073-bf4e-fef6f31dc622"
May 26 16:01:56.075838 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:56.075806    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qpzqg"
May 26 16:01:56.076563 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:56.075946    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qpzqg" podUID="8828138d-e7ed-4d5c-ba76-26429679c7b1"
May 26 16:01:57.075207 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.075168    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2fdgm"
May 26 16:01:57.075407 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:57.075300    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2fdgm" podUID="81fbb108-f79a-4018-82a4-0b0e29541320"
May 26 16:01:57.075407 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.075356    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rvlkc"
May 26 16:01:57.075507 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:57.075451    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rvlkc" podUID="8f54a9e9-f1b3-4073-bf4e-fef6f31dc622"
May 26 16:01:57.396611 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.396579    2572 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-83.ec2.internal" event="NodeReady"
May 26 16:01:57.397201 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.396762    2572 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
May 26 16:01:57.431706 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.431676    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-d4d8f94f9-bj22q"]
May 26 16:01:57.466065 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.465971    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-546498794-d7sw5"]
May 26 16:01:57.466729 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.466451    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-bj22q"
May 26 16:01:57.468912 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.468892    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-q4j2j\""
May 26 16:01:57.469019 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.468919    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
May 26 16:01:57.469158 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.469019    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
May 26 16:01:57.488595 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.488569    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cf978fc94-qtgk9"]
May 26 16:01:57.488749 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.488692    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-546498794-d7sw5"
May 26 16:01:57.490836 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.490798    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
May 26 16:01:57.491045 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.491021    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
May 26 16:01:57.491145 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.491023    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
May 26 16:01:57.491389 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.491367    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-q98vj\""
May 26 16:01:57.497242 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.497219    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
May 26 16:01:57.514500 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.514470    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdfbb578c-xqj5q"]
May 26 16:01:57.514702 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.514661    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cf978fc94-qtgk9"
May 26 16:01:57.517002 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.516975    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
May 26 16:01:57.517131 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.517088    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
May 26 16:01:57.517298 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.517280    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
May 26 16:01:57.517369 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.517336    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
May 26 16:01:57.517486 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.517469    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-b2rzg\""
May 26 16:01:57.530406 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.530384    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c66b498d4-wltk2"]
May 26 16:01:57.530547 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.530531    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdfbb578c-xqj5q"
May 26 16:01:57.532808 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.532780    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
May 26 16:01:57.532970 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.532946    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
May 26 16:01:57.533076 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.532984    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
May 26 16:01:57.533076 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.532956    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
May 26 16:01:57.548556 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.548529    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-d4d8f94f9-bj22q"]
May 26 16:01:57.548718 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.548564    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hqnkg"]
May 26 16:01:57.548718 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.548695    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c66b498d4-wltk2"
May 26 16:01:57.550862 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.550808    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
May 26 16:01:57.572785 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.572757    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdfbb578c-xqj5q"]
May 26 16:01:57.572785 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.572789    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-546498794-d7sw5"]
May 26 16:01:57.572961 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.572804    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cf978fc94-qtgk9"]
May 26 16:01:57.572961 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.572818    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hqnkg"]
May 26 16:01:57.572961 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.572828    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c66b498d4-wltk2"]
May 26 16:01:57.572961 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.572854    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-fvwnc"]
May 26 16:01:57.572961 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.572863    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hqnkg"
May 26 16:01:57.575163 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.575141    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
May 26 16:01:57.575163 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.575154    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
May 26 16:01:57.575336 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.575238    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
May 26 16:01:57.575336 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.575296    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vkcq2\""
May 26 16:01:57.596436 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.596407    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fvwnc"]
May 26 16:01:57.596633 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.596532    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fvwnc"
May 26 16:01:57.598869 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.598847    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
May 26 16:01:57.598994 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.598878    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
May 26 16:01:57.598994 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.598963    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-t9x9h\""
May 26 16:01:57.636303 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.636264    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvlxj\" (UniqueName: \"kubernetes.io/projected/3a61b0a0-aa24-4a24-8e10-8a23103bf3bf-kube-api-access-cvlxj\") pod \"managed-serviceaccount-addon-agent-7cf978fc94-qtgk9\" (UID: \"3a61b0a0-aa24-4a24-8e10-8a23103bf3bf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cf978fc94-qtgk9"
May 26 16:01:57.636476 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.636345    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6610092d-3e19-47ae-8270-ba37dd62232c-installation-pull-secrets\") pod \"image-registry-546498794-d7sw5\" (UID: \"6610092d-3e19-47ae-8270-ba37dd62232c\") " pod="openshift-image-registry/image-registry-546498794-d7sw5"
May 26 16:01:57.636476 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.636378    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6610092d-3e19-47ae-8270-ba37dd62232c-bound-sa-token\") pod \"image-registry-546498794-d7sw5\" (UID: \"6610092d-3e19-47ae-8270-ba37dd62232c\") " pod="openshift-image-registry/image-registry-546498794-d7sw5"
May 26 16:01:57.636476 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.636411    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6610092d-3e19-47ae-8270-ba37dd62232c-image-registry-private-configuration\") pod \"image-registry-546498794-d7sw5\" (UID: \"6610092d-3e19-47ae-8270-ba37dd62232c\") " pod="openshift-image-registry/image-registry-546498794-d7sw5"
May 26 16:01:57.636476 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.636435    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6610092d-3e19-47ae-8270-ba37dd62232c-ca-trust-extracted\") pod \"image-registry-546498794-d7sw5\" (UID: \"6610092d-3e19-47ae-8270-ba37dd62232c\") " pod="openshift-image-registry/image-registry-546498794-d7sw5"
May 26 16:01:57.636476 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.636458    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6610092d-3e19-47ae-8270-ba37dd62232c-registry-certificates\") pod \"image-registry-546498794-d7sw5\" (UID: \"6610092d-3e19-47ae-8270-ba37dd62232c\") " pod="openshift-image-registry/image-registry-546498794-d7sw5"
May 26 16:01:57.636745 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.636480    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/cadbc4ee-438a-4e73-b55a-95d246057641-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-cdfbb578c-xqj5q\" (UID: \"cadbc4ee-438a-4e73-b55a-95d246057641\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdfbb578c-xqj5q"
May 26 16:01:57.636745 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.636501    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs2f9\" (UniqueName: \"kubernetes.io/projected/6610092d-3e19-47ae-8270-ba37dd62232c-kube-api-access-fs2f9\") pod \"image-registry-546498794-d7sw5\" (UID: \"6610092d-3e19-47ae-8270-ba37dd62232c\") " pod="openshift-image-registry/image-registry-546498794-d7sw5"
May 26 16:01:57.636745 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.636516    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3a61b0a0-aa24-4a24-8e10-8a23103bf3bf-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7cf978fc94-qtgk9\" (UID: \"3a61b0a0-aa24-4a24-8e10-8a23103bf3bf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cf978fc94-qtgk9"
May 26 16:01:57.636745 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.636589    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6610092d-3e19-47ae-8270-ba37dd62232c-trusted-ca\") pod \"image-registry-546498794-d7sw5\" (UID: \"6610092d-3e19-47ae-8270-ba37dd62232c\") " pod="openshift-image-registry/image-registry-546498794-d7sw5"
May 26 16:01:57.636745 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.636632    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/cadbc4ee-438a-4e73-b55a-95d246057641-hub\") pod \"cluster-proxy-proxy-agent-cdfbb578c-xqj5q\" (UID: \"cadbc4ee-438a-4e73-b55a-95d246057641\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdfbb578c-xqj5q"
May 26 16:01:57.636745 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.636650    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29j2b\" (UniqueName: \"kubernetes.io/projected/cadbc4ee-438a-4e73-b55a-95d246057641-kube-api-access-29j2b\") pod \"cluster-proxy-proxy-agent-cdfbb578c-xqj5q\" (UID: \"cadbc4ee-438a-4e73-b55a-95d246057641\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdfbb578c-xqj5q"
May 26 16:01:57.636745 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.636688    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/be441f80-19f8-48c2-b1b2-624b3bddcb27-nginx-conf\") pod \"networking-console-plugin-d4d8f94f9-bj22q\" (UID: \"be441f80-19f8-48c2-b1b2-624b3bddcb27\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-bj22q"
May 26 16:01:57.636745 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.636713    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/cadbc4ee-438a-4e73-b55a-95d246057641-ca\") pod \"cluster-proxy-proxy-agent-cdfbb578c-xqj5q\" (UID: \"cadbc4ee-438a-4e73-b55a-95d246057641\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdfbb578c-xqj5q"
May 26 16:01:57.637077 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.636778    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cadbc4ee-438a-4e73-b55a-95d246057641-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-cdfbb578c-xqj5q\" (UID: \"cadbc4ee-438a-4e73-b55a-95d246057641\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdfbb578c-xqj5q"
May 26 16:01:57.637077 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.636813    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/cadbc4ee-438a-4e73-b55a-95d246057641-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-cdfbb578c-xqj5q\" (UID: \"cadbc4ee-438a-4e73-b55a-95d246057641\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdfbb578c-xqj5q"
May 26 16:01:57.637077 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.636831    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6610092d-3e19-47ae-8270-ba37dd62232c-registry-tls\") pod \"image-registry-546498794-d7sw5\" (UID: \"6610092d-3e19-47ae-8270-ba37dd62232c\") " pod="openshift-image-registry/image-registry-546498794-d7sw5"
May 26 16:01:57.637077 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.636882    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d6f02a68-6e6d-4a7e-850e-3172a9f2bd55-klusterlet-config\") pod \"klusterlet-addon-workmgr-6c66b498d4-wltk2\" (UID: \"d6f02a68-6e6d-4a7e-850e-3172a9f2bd55\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c66b498d4-wltk2"
May 26 16:01:57.637077 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.636940    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9qt5\" (UniqueName: \"kubernetes.io/projected/d6f02a68-6e6d-4a7e-850e-3172a9f2bd55-kube-api-access-h9qt5\") pod \"klusterlet-addon-workmgr-6c66b498d4-wltk2\" (UID: \"d6f02a68-6e6d-4a7e-850e-3172a9f2bd55\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c66b498d4-wltk2"
May 26 16:01:57.637077 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.636970    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/be441f80-19f8-48c2-b1b2-624b3bddcb27-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-bj22q\" (UID: \"be441f80-19f8-48c2-b1b2-624b3bddcb27\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-bj22q"
May 26 16:01:57.637077 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.636998    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6f02a68-6e6d-4a7e-850e-3172a9f2bd55-tmp\") pod \"klusterlet-addon-workmgr-6c66b498d4-wltk2\" (UID: \"d6f02a68-6e6d-4a7e-850e-3172a9f2bd55\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c66b498d4-wltk2"
May 26 16:01:57.738211 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.738115    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3a61b0a0-aa24-4a24-8e10-8a23103bf3bf-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7cf978fc94-qtgk9\" (UID: \"3a61b0a0-aa24-4a24-8e10-8a23103bf3bf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cf978fc94-qtgk9"
May 26 16:01:57.738211 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.738172    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6610092d-3e19-47ae-8270-ba37dd62232c-trusted-ca\") pod \"image-registry-546498794-d7sw5\" (UID: \"6610092d-3e19-47ae-8270-ba37dd62232c\") " pod="openshift-image-registry/image-registry-546498794-d7sw5"
May 26 16:01:57.738211 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.738195    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/cadbc4ee-438a-4e73-b55a-95d246057641-hub\") pod \"cluster-proxy-proxy-agent-cdfbb578c-xqj5q\" (UID: \"cadbc4ee-438a-4e73-b55a-95d246057641\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdfbb578c-xqj5q"
May 26 16:01:57.738472 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.738231    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw45z\" (UniqueName: \"kubernetes.io/projected/7b87e383-816e-4130-80ba-843dd20a187d-kube-api-access-mw45z\") pod \"dns-default-fvwnc\" (UID: \"7b87e383-816e-4130-80ba-843dd20a187d\") " pod="openshift-dns/dns-default-fvwnc"
May 26 16:01:57.738472 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.738262    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/be441f80-19f8-48c2-b1b2-624b3bddcb27-nginx-conf\") pod \"networking-console-plugin-d4d8f94f9-bj22q\" (UID: \"be441f80-19f8-48c2-b1b2-624b3bddcb27\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-bj22q"
May 26 16:01:57.738472 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.738435    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8828138d-e7ed-4d5c-ba76-26429679c7b1-metrics-certs\") pod \"network-metrics-daemon-qpzqg\" (UID: \"8828138d-e7ed-4d5c-ba76-26429679c7b1\") " pod="openshift-multus/network-metrics-daemon-qpzqg"
May 26 16:01:57.738594 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.738477    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5e434be1-fcca-488a-aeff-38268e150853-cert\") pod \"ingress-canary-hqnkg\" (UID: \"5e434be1-fcca-488a-aeff-38268e150853\") " pod="openshift-ingress-canary/ingress-canary-hqnkg"
May 26 16:01:57.738594 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.738511    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6610092d-3e19-47ae-8270-ba37dd62232c-registry-tls\") pod \"image-registry-546498794-d7sw5\" (UID: \"6610092d-3e19-47ae-8270-ba37dd62232c\") " pod="openshift-image-registry/image-registry-546498794-d7sw5"
May 26 16:01:57.738594 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.738547    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-h9qt5\" (UniqueName: \"kubernetes.io/projected/d6f02a68-6e6d-4a7e-850e-3172a9f2bd55-kube-api-access-h9qt5\") pod \"klusterlet-addon-workmgr-6c66b498d4-wltk2\" (UID: \"d6f02a68-6e6d-4a7e-850e-3172a9f2bd55\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c66b498d4-wltk2"
May 26 16:01:57.738594 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.738574    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b87e383-816e-4130-80ba-843dd20a187d-tmp-dir\") pod \"dns-default-fvwnc\" (UID: \"7b87e383-816e-4130-80ba-843dd20a187d\") " pod="openshift-dns/dns-default-fvwnc"
May 26 16:01:57.738594 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.738613    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6f02a68-6e6d-4a7e-850e-3172a9f2bd55-tmp\") pod \"klusterlet-addon-workmgr-6c66b498d4-wltk2\" (UID: \"d6f02a68-6e6d-4a7e-850e-3172a9f2bd55\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c66b498d4-wltk2"
May 26 16:01:57.738836 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.738645    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cvlxj\" (UniqueName: \"kubernetes.io/projected/3a61b0a0-aa24-4a24-8e10-8a23103bf3bf-kube-api-access-cvlxj\") pod \"managed-serviceaccount-addon-agent-7cf978fc94-qtgk9\" (UID: \"3a61b0a0-aa24-4a24-8e10-8a23103bf3bf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cf978fc94-qtgk9"
May 26 16:01:57.738836 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:57.738593    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:01:57.738836 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.738685    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6610092d-3e19-47ae-8270-ba37dd62232c-image-registry-private-configuration\") pod \"image-registry-546498794-d7sw5\" (UID: \"6610092d-3e19-47ae-8270-ba37dd62232c\") " pod="openshift-image-registry/image-registry-546498794-d7sw5"
May 26 16:01:57.738836 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.738711    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6610092d-3e19-47ae-8270-ba37dd62232c-registry-certificates\") pod \"image-registry-546498794-d7sw5\" (UID: \"6610092d-3e19-47ae-8270-ba37dd62232c\") " pod="openshift-image-registry/image-registry-546498794-d7sw5"
May 26 16:01:57.738836 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:57.738744    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8828138d-e7ed-4d5c-ba76-26429679c7b1-metrics-certs podName:8828138d-e7ed-4d5c-ba76-26429679c7b1 nodeName:}" failed. No retries permitted until 2026-05-26 16:02:29.738724458 +0000 UTC m=+66.214451785 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8828138d-e7ed-4d5c-ba76-26429679c7b1-metrics-certs") pod "network-metrics-daemon-qpzqg" (UID: "8828138d-e7ed-4d5c-ba76-26429679c7b1") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:01:57.738836 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.738780    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/cadbc4ee-438a-4e73-b55a-95d246057641-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-cdfbb578c-xqj5q\" (UID: \"cadbc4ee-438a-4e73-b55a-95d246057641\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdfbb578c-xqj5q"
May 26 16:01:57.738836 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.738823    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/cadbc4ee-438a-4e73-b55a-95d246057641-ca\") pod \"cluster-proxy-proxy-agent-cdfbb578c-xqj5q\" (UID: \"cadbc4ee-438a-4e73-b55a-95d246057641\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdfbb578c-xqj5q"
May 26 16:01:57.739187 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.738880    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsp7x\" (UniqueName: \"kubernetes.io/projected/5e434be1-fcca-488a-aeff-38268e150853-kube-api-access-bsp7x\") pod \"ingress-canary-hqnkg\" (UID: \"5e434be1-fcca-488a-aeff-38268e150853\") " pod="openshift-ingress-canary/ingress-canary-hqnkg"
May 26 16:01:57.739187 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.738905    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7b87e383-816e-4130-80ba-843dd20a187d-config-volume\") pod \"dns-default-fvwnc\" (UID: \"7b87e383-816e-4130-80ba-843dd20a187d\") " pod="openshift-dns/dns-default-fvwnc"
May 26 16:01:57.739187 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.738936    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cadbc4ee-438a-4e73-b55a-95d246057641-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-cdfbb578c-xqj5q\" (UID: \"cadbc4ee-438a-4e73-b55a-95d246057641\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdfbb578c-xqj5q"
May 26 16:01:57.739187 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:57.738670    2572 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 26 16:01:57.739187 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:57.738959    2572 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-546498794-d7sw5: secret "image-registry-tls" not found
May 26 16:01:57.739187 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.738960    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/cadbc4ee-438a-4e73-b55a-95d246057641-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-cdfbb578c-xqj5q\" (UID: \"cadbc4ee-438a-4e73-b55a-95d246057641\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdfbb578c-xqj5q"
May 26 16:01:57.739187 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.738986    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d6f02a68-6e6d-4a7e-850e-3172a9f2bd55-klusterlet-config\") pod \"klusterlet-addon-workmgr-6c66b498d4-wltk2\" (UID: \"d6f02a68-6e6d-4a7e-850e-3172a9f2bd55\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c66b498d4-wltk2"
May 26 16:01:57.739187 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:57.739005    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6610092d-3e19-47ae-8270-ba37dd62232c-registry-tls podName:6610092d-3e19-47ae-8270-ba37dd62232c nodeName:}" failed. No retries permitted until 2026-05-26 16:01:58.238989848 +0000 UTC m=+34.714717375 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6610092d-3e19-47ae-8270-ba37dd62232c-registry-tls") pod "image-registry-546498794-d7sw5" (UID: "6610092d-3e19-47ae-8270-ba37dd62232c") : secret "image-registry-tls" not found
May 26 16:01:57.739187 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.739006    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/be441f80-19f8-48c2-b1b2-624b3bddcb27-nginx-conf\") pod \"networking-console-plugin-d4d8f94f9-bj22q\" (UID: \"be441f80-19f8-48c2-b1b2-624b3bddcb27\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-bj22q"
May 26 16:01:57.739187 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.739044    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/be441f80-19f8-48c2-b1b2-624b3bddcb27-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-bj22q\" (UID: \"be441f80-19f8-48c2-b1b2-624b3bddcb27\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-bj22q"
May 26 16:01:57.739187 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.738933    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d6f02a68-6e6d-4a7e-850e-3172a9f2bd55-tmp\") pod \"klusterlet-addon-workmgr-6c66b498d4-wltk2\" (UID: \"d6f02a68-6e6d-4a7e-850e-3172a9f2bd55\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c66b498d4-wltk2"
May 26 16:01:57.739187 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.739183    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6610092d-3e19-47ae-8270-ba37dd62232c-registry-certificates\") pod \"image-registry-546498794-d7sw5\" (UID: \"6610092d-3e19-47ae-8270-ba37dd62232c\") " pod="openshift-image-registry/image-registry-546498794-d7sw5"
May 26 16:01:57.739676 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.739654    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6610092d-3e19-47ae-8270-ba37dd62232c-trusted-ca\") pod \"image-registry-546498794-d7sw5\" (UID: \"6610092d-3e19-47ae-8270-ba37dd62232c\") " pod="openshift-image-registry/image-registry-546498794-d7sw5"
May 26 16:01:57.740098 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.740000    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/cadbc4ee-438a-4e73-b55a-95d246057641-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-cdfbb578c-xqj5q\" (UID: \"cadbc4ee-438a-4e73-b55a-95d246057641\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdfbb578c-xqj5q"
May 26 16:01:57.740098 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.740072    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b87e383-816e-4130-80ba-843dd20a187d-metrics-tls\") pod \"dns-default-fvwnc\" (UID: \"7b87e383-816e-4130-80ba-843dd20a187d\") " pod="openshift-dns/dns-default-fvwnc"
May 26 16:01:57.740252 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:57.740223    2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 26 16:01:57.740297 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:57.740280    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be441f80-19f8-48c2-b1b2-624b3bddcb27-networking-console-plugin-cert podName:be441f80-19f8-48c2-b1b2-624b3bddcb27 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:58.240264183 +0000 UTC m=+34.715991525 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/be441f80-19f8-48c2-b1b2-624b3bddcb27-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-bj22q" (UID: "be441f80-19f8-48c2-b1b2-624b3bddcb27") : secret "networking-console-plugin-cert" not found
May 26 16:01:57.740363 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.740344    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6610092d-3e19-47ae-8270-ba37dd62232c-ca-trust-extracted\") pod \"image-registry-546498794-d7sw5\" (UID: \"6610092d-3e19-47ae-8270-ba37dd62232c\") " pod="openshift-image-registry/image-registry-546498794-d7sw5"
May 26 16:01:57.740688 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.740666    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6610092d-3e19-47ae-8270-ba37dd62232c-ca-trust-extracted\") pod \"image-registry-546498794-d7sw5\" (UID: \"6610092d-3e19-47ae-8270-ba37dd62232c\") " pod="openshift-image-registry/image-registry-546498794-d7sw5"
May 26 16:01:57.740784 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.740721    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-29j2b\" (UniqueName: \"kubernetes.io/projected/cadbc4ee-438a-4e73-b55a-95d246057641-kube-api-access-29j2b\") pod \"cluster-proxy-proxy-agent-cdfbb578c-xqj5q\" (UID: \"cadbc4ee-438a-4e73-b55a-95d246057641\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdfbb578c-xqj5q"
May 26 16:01:57.740838 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.740784    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6610092d-3e19-47ae-8270-ba37dd62232c-installation-pull-secrets\") pod \"image-registry-546498794-d7sw5\" (UID: \"6610092d-3e19-47ae-8270-ba37dd62232c\") " pod="openshift-image-registry/image-registry-546498794-d7sw5"
May 26 16:01:57.740838 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.740816    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6610092d-3e19-47ae-8270-ba37dd62232c-bound-sa-token\") pod \"image-registry-546498794-d7sw5\" (UID: \"6610092d-3e19-47ae-8270-ba37dd62232c\") " pod="openshift-image-registry/image-registry-546498794-d7sw5"
May 26 16:01:57.740932 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.740846    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fs2f9\" (UniqueName: \"kubernetes.io/projected/6610092d-3e19-47ae-8270-ba37dd62232c-kube-api-access-fs2f9\") pod \"image-registry-546498794-d7sw5\" (UID: \"6610092d-3e19-47ae-8270-ba37dd62232c\") " pod="openshift-image-registry/image-registry-546498794-d7sw5"
May 26 16:01:57.744073 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.744031    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/cadbc4ee-438a-4e73-b55a-95d246057641-ca\") pod \"cluster-proxy-proxy-agent-cdfbb578c-xqj5q\" (UID: \"cadbc4ee-438a-4e73-b55a-95d246057641\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdfbb578c-xqj5q"
May 26 16:01:57.744182 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.744120    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6610092d-3e19-47ae-8270-ba37dd62232c-installation-pull-secrets\") pod \"image-registry-546498794-d7sw5\" (UID: \"6610092d-3e19-47ae-8270-ba37dd62232c\") " pod="openshift-image-registry/image-registry-546498794-d7sw5"
May 26 16:01:57.744244 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.744201    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/cadbc4ee-438a-4e73-b55a-95d246057641-hub\") pod \"cluster-proxy-proxy-agent-cdfbb578c-xqj5q\" (UID: \"cadbc4ee-438a-4e73-b55a-95d246057641\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdfbb578c-xqj5q"
May 26 16:01:57.744297 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.744249    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d6f02a68-6e6d-4a7e-850e-3172a9f2bd55-klusterlet-config\") pod \"klusterlet-addon-workmgr-6c66b498d4-wltk2\" (UID: \"d6f02a68-6e6d-4a7e-850e-3172a9f2bd55\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c66b498d4-wltk2"
May 26 16:01:57.744354 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.744335    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cadbc4ee-438a-4e73-b55a-95d246057641-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-cdfbb578c-xqj5q\" (UID: \"cadbc4ee-438a-4e73-b55a-95d246057641\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdfbb578c-xqj5q"
May 26 16:01:57.744769 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.744745    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3a61b0a0-aa24-4a24-8e10-8a23103bf3bf-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7cf978fc94-qtgk9\" (UID: \"3a61b0a0-aa24-4a24-8e10-8a23103bf3bf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cf978fc94-qtgk9"
May 26 16:01:57.745008 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.744972    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6610092d-3e19-47ae-8270-ba37dd62232c-image-registry-private-configuration\") pod \"image-registry-546498794-d7sw5\" (UID: \"6610092d-3e19-47ae-8270-ba37dd62232c\") " pod="openshift-image-registry/image-registry-546498794-d7sw5"
May 26 16:01:57.745590 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.745565    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/cadbc4ee-438a-4e73-b55a-95d246057641-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-cdfbb578c-xqj5q\" (UID: \"cadbc4ee-438a-4e73-b55a-95d246057641\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdfbb578c-xqj5q"
May 26 16:01:57.748489 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.748465    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvlxj\" (UniqueName: \"kubernetes.io/projected/3a61b0a0-aa24-4a24-8e10-8a23103bf3bf-kube-api-access-cvlxj\") pod \"managed-serviceaccount-addon-agent-7cf978fc94-qtgk9\" (UID: \"3a61b0a0-aa24-4a24-8e10-8a23103bf3bf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cf978fc94-qtgk9"
May 26 16:01:57.748595 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.748550    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9qt5\" (UniqueName: \"kubernetes.io/projected/d6f02a68-6e6d-4a7e-850e-3172a9f2bd55-kube-api-access-h9qt5\") pod \"klusterlet-addon-workmgr-6c66b498d4-wltk2\" (UID: \"d6f02a68-6e6d-4a7e-850e-3172a9f2bd55\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c66b498d4-wltk2"
May 26 16:01:57.750506 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.750482    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29j2b\" (UniqueName: \"kubernetes.io/projected/cadbc4ee-438a-4e73-b55a-95d246057641-kube-api-access-29j2b\") pod \"cluster-proxy-proxy-agent-cdfbb578c-xqj5q\" (UID: \"cadbc4ee-438a-4e73-b55a-95d246057641\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdfbb578c-xqj5q"
May 26 16:01:57.750832 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.750812    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6610092d-3e19-47ae-8270-ba37dd62232c-bound-sa-token\") pod \"image-registry-546498794-d7sw5\" (UID: \"6610092d-3e19-47ae-8270-ba37dd62232c\") " pod="openshift-image-registry/image-registry-546498794-d7sw5"
May 26 16:01:57.750972 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.750956    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs2f9\" (UniqueName: \"kubernetes.io/projected/6610092d-3e19-47ae-8270-ba37dd62232c-kube-api-access-fs2f9\") pod \"image-registry-546498794-d7sw5\" (UID: \"6610092d-3e19-47ae-8270-ba37dd62232c\") " pod="openshift-image-registry/image-registry-546498794-d7sw5"
May 26 16:01:57.835974 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.835929    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cf978fc94-qtgk9"
May 26 16:01:57.841369 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.841336    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bsp7x\" (UniqueName: \"kubernetes.io/projected/5e434be1-fcca-488a-aeff-38268e150853-kube-api-access-bsp7x\") pod \"ingress-canary-hqnkg\" (UID: \"5e434be1-fcca-488a-aeff-38268e150853\") " pod="openshift-ingress-canary/ingress-canary-hqnkg"
May 26 16:01:57.841491 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.841384    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7b87e383-816e-4130-80ba-843dd20a187d-config-volume\") pod \"dns-default-fvwnc\" (UID: \"7b87e383-816e-4130-80ba-843dd20a187d\") " pod="openshift-dns/dns-default-fvwnc"
May 26 16:01:57.841491 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.841435    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b87e383-816e-4130-80ba-843dd20a187d-metrics-tls\") pod \"dns-default-fvwnc\" (UID: \"7b87e383-816e-4130-80ba-843dd20a187d\") " pod="openshift-dns/dns-default-fvwnc"
May 26 16:01:57.841620 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:57.841537    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 16:01:57.841620 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.841563    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mw45z\" (UniqueName: \"kubernetes.io/projected/7b87e383-816e-4130-80ba-843dd20a187d-kube-api-access-mw45z\") pod \"dns-default-fvwnc\" (UID: \"7b87e383-816e-4130-80ba-843dd20a187d\") " pod="openshift-dns/dns-default-fvwnc"
May 26 16:01:57.841620 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:57.841595    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b87e383-816e-4130-80ba-843dd20a187d-metrics-tls podName:7b87e383-816e-4130-80ba-843dd20a187d nodeName:}" failed. No retries permitted until 2026-05-26 16:01:58.341576592 +0000 UTC m=+34.817303937 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7b87e383-816e-4130-80ba-843dd20a187d-metrics-tls") pod "dns-default-fvwnc" (UID: "7b87e383-816e-4130-80ba-843dd20a187d") : secret "dns-default-metrics-tls" not found
May 26 16:01:57.841762 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.841655    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5e434be1-fcca-488a-aeff-38268e150853-cert\") pod \"ingress-canary-hqnkg\" (UID: \"5e434be1-fcca-488a-aeff-38268e150853\") " pod="openshift-ingress-canary/ingress-canary-hqnkg"
May 26 16:01:57.841762 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.841700    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b87e383-816e-4130-80ba-843dd20a187d-tmp-dir\") pod \"dns-default-fvwnc\" (UID: \"7b87e383-816e-4130-80ba-843dd20a187d\") " pod="openshift-dns/dns-default-fvwnc"
May 26 16:01:57.841762 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.841737    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lbgsb\" (UniqueName: \"kubernetes.io/projected/8f54a9e9-f1b3-4073-bf4e-fef6f31dc622-kube-api-access-lbgsb\") pod \"network-check-target-rvlkc\" (UID: \"8f54a9e9-f1b3-4073-bf4e-fef6f31dc622\") " pod="openshift-network-diagnostics/network-check-target-rvlkc"
May 26 16:01:57.841762 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:57.841759    2572 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 16:01:57.841950 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:57.841833    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e434be1-fcca-488a-aeff-38268e150853-cert podName:5e434be1-fcca-488a-aeff-38268e150853 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:58.34181225 +0000 UTC m=+34.817539592 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5e434be1-fcca-488a-aeff-38268e150853-cert") pod "ingress-canary-hqnkg" (UID: "5e434be1-fcca-488a-aeff-38268e150853") : secret "canary-serving-cert" not found
May 26 16:01:57.841950 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:57.841910    2572 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 16:01:57.841950 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:57.841924    2572 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 16:01:57.841950 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:57.841944    2572 projected.go:194] Error preparing data for projected volume kube-api-access-lbgsb for pod openshift-network-diagnostics/network-check-target-rvlkc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:01:57.842108 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:57.841978    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f54a9e9-f1b3-4073-bf4e-fef6f31dc622-kube-api-access-lbgsb podName:8f54a9e9-f1b3-4073-bf4e-fef6f31dc622 nodeName:}" failed. No retries permitted until 2026-05-26 16:02:29.841969543 +0000 UTC m=+66.317696870 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-lbgsb" (UniqueName: "kubernetes.io/projected/8f54a9e9-f1b3-4073-bf4e-fef6f31dc622-kube-api-access-lbgsb") pod "network-check-target-rvlkc" (UID: "8f54a9e9-f1b3-4073-bf4e-fef6f31dc622") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:01:57.842108 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.842014    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b87e383-816e-4130-80ba-843dd20a187d-tmp-dir\") pod \"dns-default-fvwnc\" (UID: \"7b87e383-816e-4130-80ba-843dd20a187d\") " pod="openshift-dns/dns-default-fvwnc"
May 26 16:01:57.842108 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.842049    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7b87e383-816e-4130-80ba-843dd20a187d-config-volume\") pod \"dns-default-fvwnc\" (UID: \"7b87e383-816e-4130-80ba-843dd20a187d\") " pod="openshift-dns/dns-default-fvwnc"
May 26 16:01:57.844300 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.844276    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdfbb578c-xqj5q"
May 26 16:01:57.850837 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.850805    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw45z\" (UniqueName: \"kubernetes.io/projected/7b87e383-816e-4130-80ba-843dd20a187d-kube-api-access-mw45z\") pod \"dns-default-fvwnc\" (UID: \"7b87e383-816e-4130-80ba-843dd20a187d\") " pod="openshift-dns/dns-default-fvwnc"
May 26 16:01:57.850949 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.850892    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsp7x\" (UniqueName: \"kubernetes.io/projected/5e434be1-fcca-488a-aeff-38268e150853-kube-api-access-bsp7x\") pod \"ingress-canary-hqnkg\" (UID: \"5e434be1-fcca-488a-aeff-38268e150853\") " pod="openshift-ingress-canary/ingress-canary-hqnkg"
May 26 16:01:57.873287 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:57.873254    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c66b498d4-wltk2"
May 26 16:01:58.075473 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:58.075385    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qpzqg"
May 26 16:01:58.077794 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:58.077750    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 26 16:01:58.077946 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:58.077879    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fjs6m\""
May 26 16:01:58.244511 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:58.244469    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/be441f80-19f8-48c2-b1b2-624b3bddcb27-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-bj22q\" (UID: \"be441f80-19f8-48c2-b1b2-624b3bddcb27\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-bj22q"
May 26 16:01:58.244703 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:58.244597    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6610092d-3e19-47ae-8270-ba37dd62232c-registry-tls\") pod \"image-registry-546498794-d7sw5\" (UID: \"6610092d-3e19-47ae-8270-ba37dd62232c\") " pod="openshift-image-registry/image-registry-546498794-d7sw5"
May 26 16:01:58.244703 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:58.244628    2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 26 16:01:58.244703 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:58.244685    2572 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 26 16:01:58.244703 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:58.244704    2572 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-546498794-d7sw5: secret "image-registry-tls" not found
May 26 16:01:58.244925 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:58.244711    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be441f80-19f8-48c2-b1b2-624b3bddcb27-networking-console-plugin-cert podName:be441f80-19f8-48c2-b1b2-624b3bddcb27 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:59.244689171 +0000 UTC m=+35.720416512 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/be441f80-19f8-48c2-b1b2-624b3bddcb27-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-bj22q" (UID: "be441f80-19f8-48c2-b1b2-624b3bddcb27") : secret "networking-console-plugin-cert" not found
May 26 16:01:58.244925 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:58.244746    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6610092d-3e19-47ae-8270-ba37dd62232c-registry-tls podName:6610092d-3e19-47ae-8270-ba37dd62232c nodeName:}" failed. No retries permitted until 2026-05-26 16:01:59.24473309 +0000 UTC m=+35.720460419 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6610092d-3e19-47ae-8270-ba37dd62232c-registry-tls") pod "image-registry-546498794-d7sw5" (UID: "6610092d-3e19-47ae-8270-ba37dd62232c") : secret "image-registry-tls" not found
May 26 16:01:58.345363 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:58.345283    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b87e383-816e-4130-80ba-843dd20a187d-metrics-tls\") pod \"dns-default-fvwnc\" (UID: \"7b87e383-816e-4130-80ba-843dd20a187d\") " pod="openshift-dns/dns-default-fvwnc"
May 26 16:01:58.345525 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:58.345382    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5e434be1-fcca-488a-aeff-38268e150853-cert\") pod \"ingress-canary-hqnkg\" (UID: \"5e434be1-fcca-488a-aeff-38268e150853\") " pod="openshift-ingress-canary/ingress-canary-hqnkg"
May 26 16:01:58.345525 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:58.345463    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 16:01:58.345636 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:58.345532    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b87e383-816e-4130-80ba-843dd20a187d-metrics-tls podName:7b87e383-816e-4130-80ba-843dd20a187d nodeName:}" failed. No retries permitted until 2026-05-26 16:01:59.345515565 +0000 UTC m=+35.821242916 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7b87e383-816e-4130-80ba-843dd20a187d-metrics-tls") pod "dns-default-fvwnc" (UID: "7b87e383-816e-4130-80ba-843dd20a187d") : secret "dns-default-metrics-tls" not found
May 26 16:01:58.345636 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:58.345547    2572 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 16:01:58.345636 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:58.345596    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e434be1-fcca-488a-aeff-38268e150853-cert podName:5e434be1-fcca-488a-aeff-38268e150853 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:59.345581276 +0000 UTC m=+35.821308603 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5e434be1-fcca-488a-aeff-38268e150853-cert") pod "ingress-canary-hqnkg" (UID: "5e434be1-fcca-488a-aeff-38268e150853") : secret "canary-serving-cert" not found
May 26 16:01:58.648656 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:58.648590    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81fbb108-f79a-4018-82a4-0b0e29541320-original-pull-secret\") pod \"global-pull-secret-syncer-2fdgm\" (UID: \"81fbb108-f79a-4018-82a4-0b0e29541320\") " pod="kube-system/global-pull-secret-syncer-2fdgm"
May 26 16:01:58.649301 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:58.648777    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 26 16:01:58.649301 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:58.648865    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81fbb108-f79a-4018-82a4-0b0e29541320-original-pull-secret podName:81fbb108-f79a-4018-82a4-0b0e29541320 nodeName:}" failed. No retries permitted until 2026-05-26 16:02:30.648844599 +0000 UTC m=+67.124571938 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/81fbb108-f79a-4018-82a4-0b0e29541320-original-pull-secret") pod "global-pull-secret-syncer-2fdgm" (UID: "81fbb108-f79a-4018-82a4-0b0e29541320") : object "kube-system"/"original-pull-secret" not registered
May 26 16:01:59.048921 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:59.048745    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cf978fc94-qtgk9"]
May 26 16:01:59.051524 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:59.051498    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c66b498d4-wltk2"]
May 26 16:01:59.054789 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:59.054766    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdfbb578c-xqj5q"]
May 26 16:01:59.075285 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:59.075262    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rvlkc"
May 26 16:01:59.075410 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:59.075262    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2fdgm"
May 26 16:01:59.077553 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:59.077530    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 26 16:01:59.077688 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:59.077658    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bvvx5\""
May 26 16:01:59.077688 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:59.077668    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 26 16:01:59.077832 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:59.077695    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 26 16:01:59.081839 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:59.081811    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a61b0a0_aa24_4a24_8e10_8a23103bf3bf.slice/crio-100c0a2e4d050e1785e3c6d5f545466744324c8d0f93c0f1ce9550d75b201139 WatchSource:0}: Error finding container 100c0a2e4d050e1785e3c6d5f545466744324c8d0f93c0f1ce9550d75b201139: Status 404 returned error can't find the container with id 100c0a2e4d050e1785e3c6d5f545466744324c8d0f93c0f1ce9550d75b201139
May 26 16:01:59.082539 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:59.082514    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6f02a68_6e6d_4a7e_850e_3172a9f2bd55.slice/crio-11d82c020b3243f4ed7c254a347a9701db3d2064f785497bcb11ddb72a14592b WatchSource:0}: Error finding container 11d82c020b3243f4ed7c254a347a9701db3d2064f785497bcb11ddb72a14592b: Status 404 returned error can't find the container with id 11d82c020b3243f4ed7c254a347a9701db3d2064f785497bcb11ddb72a14592b
May 26 16:01:59.083389 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:01:59.083371    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcadbc4ee_438a_4e73_b55a_95d246057641.slice/crio-460555a973757eb9daed6ba6834a260b0dc7991f1af933dc12734a4088d999aa WatchSource:0}: Error finding container 460555a973757eb9daed6ba6834a260b0dc7991f1af933dc12734a4088d999aa: Status 404 returned error can't find the container with id 460555a973757eb9daed6ba6834a260b0dc7991f1af933dc12734a4088d999aa
May 26 16:01:59.253528 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:59.253500    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/be441f80-19f8-48c2-b1b2-624b3bddcb27-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-bj22q\" (UID: \"be441f80-19f8-48c2-b1b2-624b3bddcb27\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-bj22q"
May 26 16:01:59.253675 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:59.253659    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6610092d-3e19-47ae-8270-ba37dd62232c-registry-tls\") pod \"image-registry-546498794-d7sw5\" (UID: \"6610092d-3e19-47ae-8270-ba37dd62232c\") " pod="openshift-image-registry/image-registry-546498794-d7sw5"
May 26 16:01:59.253746 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:59.253670    2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 26 16:01:59.253746 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:59.253730    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be441f80-19f8-48c2-b1b2-624b3bddcb27-networking-console-plugin-cert podName:be441f80-19f8-48c2-b1b2-624b3bddcb27 nodeName:}" failed. No retries permitted until 2026-05-26 16:02:01.25371084 +0000 UTC m=+37.729438174 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/be441f80-19f8-48c2-b1b2-624b3bddcb27-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-bj22q" (UID: "be441f80-19f8-48c2-b1b2-624b3bddcb27") : secret "networking-console-plugin-cert" not found
May 26 16:01:59.253837 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:59.253777    2572 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 26 16:01:59.253837 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:59.253790    2572 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-546498794-d7sw5: secret "image-registry-tls" not found
May 26 16:01:59.253901 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:59.253853    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6610092d-3e19-47ae-8270-ba37dd62232c-registry-tls podName:6610092d-3e19-47ae-8270-ba37dd62232c nodeName:}" failed. No retries permitted until 2026-05-26 16:02:01.253840943 +0000 UTC m=+37.729568270 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6610092d-3e19-47ae-8270-ba37dd62232c-registry-tls") pod "image-registry-546498794-d7sw5" (UID: "6610092d-3e19-47ae-8270-ba37dd62232c") : secret "image-registry-tls" not found
May 26 16:01:59.301101 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:59.301039    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cf978fc94-qtgk9" event={"ID":"3a61b0a0-aa24-4a24-8e10-8a23103bf3bf","Type":"ContainerStarted","Data":"100c0a2e4d050e1785e3c6d5f545466744324c8d0f93c0f1ce9550d75b201139"}
May 26 16:01:59.302134 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:59.302098    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c66b498d4-wltk2" event={"ID":"d6f02a68-6e6d-4a7e-850e-3172a9f2bd55","Type":"ContainerStarted","Data":"11d82c020b3243f4ed7c254a347a9701db3d2064f785497bcb11ddb72a14592b"}
May 26 16:01:59.304729 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:59.304703    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mjznd" event={"ID":"23cb5db1-ddc3-46a0-bc9d-3133631ffc58","Type":"ContainerStarted","Data":"1404d024551f9dc71e6362f2a956937bfa286534c7e3c81e7a487252902ae632"}
May 26 16:01:59.305736 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:59.305705    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdfbb578c-xqj5q" event={"ID":"cadbc4ee-438a-4e73-b55a-95d246057641","Type":"ContainerStarted","Data":"460555a973757eb9daed6ba6834a260b0dc7991f1af933dc12734a4088d999aa"}
May 26 16:01:59.354115 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:59.354086    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5e434be1-fcca-488a-aeff-38268e150853-cert\") pod \"ingress-canary-hqnkg\" (UID: \"5e434be1-fcca-488a-aeff-38268e150853\") " pod="openshift-ingress-canary/ingress-canary-hqnkg"
May 26 16:01:59.354255 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:01:59.354178    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b87e383-816e-4130-80ba-843dd20a187d-metrics-tls\") pod \"dns-default-fvwnc\" (UID: \"7b87e383-816e-4130-80ba-843dd20a187d\") " pod="openshift-dns/dns-default-fvwnc"
May 26 16:01:59.354255 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:59.354235    2572 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 16:01:59.354453 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:59.354278    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 16:01:59.354453 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:59.354309    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e434be1-fcca-488a-aeff-38268e150853-cert podName:5e434be1-fcca-488a-aeff-38268e150853 nodeName:}" failed. No retries permitted until 2026-05-26 16:02:01.354293618 +0000 UTC m=+37.830020945 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5e434be1-fcca-488a-aeff-38268e150853-cert") pod "ingress-canary-hqnkg" (UID: "5e434be1-fcca-488a-aeff-38268e150853") : secret "canary-serving-cert" not found
May 26 16:01:59.354453 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:01:59.354323    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b87e383-816e-4130-80ba-843dd20a187d-metrics-tls podName:7b87e383-816e-4130-80ba-843dd20a187d nodeName:}" failed. No retries permitted until 2026-05-26 16:02:01.354317063 +0000 UTC m=+37.830044390 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7b87e383-816e-4130-80ba-843dd20a187d-metrics-tls") pod "dns-default-fvwnc" (UID: "7b87e383-816e-4130-80ba-843dd20a187d") : secret "dns-default-metrics-tls" not found
May 26 16:02:00.311765 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:02:00.311227    2572 generic.go:358] "Generic (PLEG): container finished" podID="23cb5db1-ddc3-46a0-bc9d-3133631ffc58" containerID="1404d024551f9dc71e6362f2a956937bfa286534c7e3c81e7a487252902ae632" exitCode=0
May 26 16:02:00.311765 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:02:00.311346    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mjznd" event={"ID":"23cb5db1-ddc3-46a0-bc9d-3133631ffc58","Type":"ContainerDied","Data":"1404d024551f9dc71e6362f2a956937bfa286534c7e3c81e7a487252902ae632"}
May 26 16:02:01.271797 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:02:01.271755    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6610092d-3e19-47ae-8270-ba37dd62232c-registry-tls\") pod \"image-registry-546498794-d7sw5\" (UID: \"6610092d-3e19-47ae-8270-ba37dd62232c\") " pod="openshift-image-registry/image-registry-546498794-d7sw5"
May 26 16:02:01.271995 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:02:01.271861    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/be441f80-19f8-48c2-b1b2-624b3bddcb27-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-bj22q\" (UID: \"be441f80-19f8-48c2-b1b2-624b3bddcb27\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-bj22q"
May 26 16:02:01.271995 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:02:01.271906    2572 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 26 16:02:01.271995 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:02:01.271924    2572 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-546498794-d7sw5: secret "image-registry-tls" not found
May 26 16:02:01.271995 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:02:01.271954    2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 26 16:02:01.271995 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:02:01.271986    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6610092d-3e19-47ae-8270-ba37dd62232c-registry-tls podName:6610092d-3e19-47ae-8270-ba37dd62232c nodeName:}" failed. No retries permitted until 2026-05-26 16:02:05.27196728 +0000 UTC m=+41.747694623 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6610092d-3e19-47ae-8270-ba37dd62232c-registry-tls") pod "image-registry-546498794-d7sw5" (UID: "6610092d-3e19-47ae-8270-ba37dd62232c") : secret "image-registry-tls" not found
May 26 16:02:01.272266 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:02:01.272005    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be441f80-19f8-48c2-b1b2-624b3bddcb27-networking-console-plugin-cert podName:be441f80-19f8-48c2-b1b2-624b3bddcb27 nodeName:}" failed. No retries permitted until 2026-05-26 16:02:05.271995974 +0000 UTC m=+41.747723301 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/be441f80-19f8-48c2-b1b2-624b3bddcb27-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-bj22q" (UID: "be441f80-19f8-48c2-b1b2-624b3bddcb27") : secret "networking-console-plugin-cert" not found
May 26 16:02:01.320636 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:02:01.320121    2572 generic.go:358] "Generic (PLEG): container finished" podID="23cb5db1-ddc3-46a0-bc9d-3133631ffc58" containerID="41f88b7858d770c550bdde94b6f7452c9417cca3c018404f0143f30d074a4985" exitCode=0
May 26 16:02:01.320636 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:02:01.320203    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mjznd" event={"ID":"23cb5db1-ddc3-46a0-bc9d-3133631ffc58","Type":"ContainerDied","Data":"41f88b7858d770c550bdde94b6f7452c9417cca3c018404f0143f30d074a4985"}
May 26 16:02:01.372969 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:02:01.372931    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5e434be1-fcca-488a-aeff-38268e150853-cert\") pod \"ingress-canary-hqnkg\" (UID: \"5e434be1-fcca-488a-aeff-38268e150853\") " pod="openshift-ingress-canary/ingress-canary-hqnkg"
May 26 16:02:01.373150 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:02:01.373052    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b87e383-816e-4130-80ba-843dd20a187d-metrics-tls\") pod \"dns-default-fvwnc\" (UID: \"7b87e383-816e-4130-80ba-843dd20a187d\") " pod="openshift-dns/dns-default-fvwnc"
May 26 16:02:01.373150 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:02:01.373114    2572 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 16:02:01.373262 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:02:01.373175    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 16:02:01.373262 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:02:01.373189    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e434be1-fcca-488a-aeff-38268e150853-cert podName:5e434be1-fcca-488a-aeff-38268e150853 nodeName:}" failed. No retries permitted until 2026-05-26 16:02:05.37316756 +0000 UTC m=+41.848894894 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5e434be1-fcca-488a-aeff-38268e150853-cert") pod "ingress-canary-hqnkg" (UID: "5e434be1-fcca-488a-aeff-38268e150853") : secret "canary-serving-cert" not found
May 26 16:02:01.373262 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:02:01.373228    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b87e383-816e-4130-80ba-843dd20a187d-metrics-tls podName:7b87e383-816e-4130-80ba-843dd20a187d nodeName:}" failed. No retries permitted until 2026-05-26 16:02:05.37321149 +0000 UTC m=+41.848938839 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7b87e383-816e-4130-80ba-843dd20a187d-metrics-tls") pod "dns-default-fvwnc" (UID: "7b87e383-816e-4130-80ba-843dd20a187d") : secret "dns-default-metrics-tls" not found
May 26 16:02:05.308218 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:02:05.308176    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/be441f80-19f8-48c2-b1b2-624b3bddcb27-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-bj22q\" (UID: \"be441f80-19f8-48c2-b1b2-624b3bddcb27\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-bj22q"
May 26 16:02:05.308821 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:02:05.308301    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6610092d-3e19-47ae-8270-ba37dd62232c-registry-tls\") pod \"image-registry-546498794-d7sw5\" (UID: \"6610092d-3e19-47ae-8270-ba37dd62232c\") " pod="openshift-image-registry/image-registry-546498794-d7sw5"
May 26 16:02:05.308821 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:02:05.308339    2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 26 16:02:05.308821 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:02:05.308404    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be441f80-19f8-48c2-b1b2-624b3bddcb27-networking-console-plugin-cert podName:be441f80-19f8-48c2-b1b2-624b3bddcb27 nodeName:}" failed. No retries permitted until 2026-05-26 16:02:13.308387807 +0000 UTC m=+49.784115138 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/be441f80-19f8-48c2-b1b2-624b3bddcb27-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-bj22q" (UID: "be441f80-19f8-48c2-b1b2-624b3bddcb27") : secret "networking-console-plugin-cert" not found
May 26 16:02:05.308821 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:02:05.308407    2572 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 26 16:02:05.308821 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:02:05.308419    2572 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-546498794-d7sw5: secret "image-registry-tls" not found
May 26 16:02:05.308821 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:02:05.308465    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6610092d-3e19-47ae-8270-ba37dd62232c-registry-tls podName:6610092d-3e19-47ae-8270-ba37dd62232c nodeName:}" failed. No retries permitted until 2026-05-26 16:02:13.308451568 +0000 UTC m=+49.784178895 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6610092d-3e19-47ae-8270-ba37dd62232c-registry-tls") pod "image-registry-546498794-d7sw5" (UID: "6610092d-3e19-47ae-8270-ba37dd62232c") : secret "image-registry-tls" not found
May 26 16:02:05.409373 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:02:05.409336    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5e434be1-fcca-488a-aeff-38268e150853-cert\") pod \"ingress-canary-hqnkg\" (UID: \"5e434be1-fcca-488a-aeff-38268e150853\") " pod="openshift-ingress-canary/ingress-canary-hqnkg"
May 26 16:02:05.409562 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:02:05.409434    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b87e383-816e-4130-80ba-843dd20a187d-metrics-tls\") pod \"dns-default-fvwnc\" (UID: \"7b87e383-816e-4130-80ba-843dd20a187d\") " pod="openshift-dns/dns-default-fvwnc"
May 26 16:02:05.409562 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:02:05.409493    2572 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 16:02:05.409562 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:02:05.409542    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 16:02:05.409562 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:02:05.409561    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e434be1-fcca-488a-aeff-38268e150853-cert podName:5e434be1-fcca-488a-aeff-38268e150853 nodeName:}" failed. No retries permitted until 2026-05-26 16:02:13.409545597 +0000 UTC m=+49.885272929 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5e434be1-fcca-488a-aeff-38268e150853-cert") pod "ingress-canary-hqnkg" (UID: "5e434be1-fcca-488a-aeff-38268e150853") : secret "canary-serving-cert" not found
May 26 16:02:05.409820 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:02:05.409595    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b87e383-816e-4130-80ba-843dd20a187d-metrics-tls podName:7b87e383-816e-4130-80ba-843dd20a187d nodeName:}" failed. No retries permitted until 2026-05-26 16:02:13.409579791 +0000 UTC m=+49.885307119 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7b87e383-816e-4130-80ba-843dd20a187d-metrics-tls") pod "dns-default-fvwnc" (UID: "7b87e383-816e-4130-80ba-843dd20a187d") : secret "dns-default-metrics-tls" not found
May 26 16:02:07.334724 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:02:07.334671    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdfbb578c-xqj5q" event={"ID":"cadbc4ee-438a-4e73-b55a-95d246057641","Type":"ContainerStarted","Data":"7d6b32b28afa75173cc39d03a9ce28e42b194d4280fe04d8bce38e7d1178a3d7"}
May 26 16:02:07.335830 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:02:07.335804    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cf978fc94-qtgk9" event={"ID":"3a61b0a0-aa24-4a24-8e10-8a23103bf3bf","Type":"ContainerStarted","Data":"d9133d60490dc7fcb8a83e7d2bee0ff74a1dcbcd3ba4a6b24684a1b521d0ac8e"}
May 26 16:02:07.337035 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:02:07.337009    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c66b498d4-wltk2" event={"ID":"d6f02a68-6e6d-4a7e-850e-3172a9f2bd55","Type":"ContainerStarted","Data":"98abf90a386fbf27a0b6d391aa91f151a44358aff5343b04f6eb20bfbc89a270"}
May 26 16:02:07.337219 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:02:07.337203    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c66b498d4-wltk2"
May 26 16:02:07.338997 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:02:07.338979    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c66b498d4-wltk2"
May 26 16:02:07.340066 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:02:07.340045    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mjznd" event={"ID":"23cb5db1-ddc3-46a0-bc9d-3133631ffc58","Type":"ContainerStarted","Data":"80e540d947d43382f1c6b79429e23d585bef5e66fcfb910fd417e8cfdc7511cf"}
May 26 16:02:07.350528 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:02:07.350490    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cf978fc94-qtgk9" podStartSLOduration=3.586142363 podStartE2EDuration="11.350478806s" podCreationTimestamp="2026-05-26 16:01:56 +0000 UTC" firstStartedPulling="2026-05-26 16:01:59.089256278 +0000 UTC m=+35.564983608" lastFinishedPulling="2026-05-26 16:02:06.853592707 +0000 UTC m=+43.329320051" observedRunningTime="2026-05-26 16:02:07.350169269 +0000 UTC m=+43.825896620" watchObservedRunningTime="2026-05-26 16:02:07.350478806 +0000 UTC m=+43.826206156"
May 26 16:02:07.365434 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:02:07.365387    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c66b498d4-wltk2" podStartSLOduration=3.584009698 podStartE2EDuration="11.365374498s" podCreationTimestamp="2026-05-26 16:01:56 +0000 UTC" firstStartedPulling="2026-05-26 16:01:59.089143081 +0000 UTC m=+35.564870417" lastFinishedPulling="2026-05-26 16:02:06.870507876 +0000 UTC m=+43.346235217" observedRunningTime="2026-05-26 16:02:07.364984825 +0000 UTC m=+43.840712188" watchObservedRunningTime="2026-05-26 16:02:07.365374498 +0000 UTC m=+43.841101847"
May 26 16:02:07.386071 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:02:07.386025    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-mjznd" podStartSLOduration=10.778149108000001 podStartE2EDuration="43.386014193s" podCreationTimestamp="2026-05-26 16:01:24 +0000 UTC" firstStartedPulling="2026-05-26 16:01:26.505455532 +0000 UTC m=+2.981182861" lastFinishedPulling="2026-05-26 16:01:59.113320615 +0000 UTC m=+35.589047946" observedRunningTime="2026-05-26 16:02:07.384536613 +0000 UTC m=+43.860263966" watchObservedRunningTime="2026-05-26 16:02:07.386014193 +0000 UTC m=+43.861741543"
May 26 16:02:13.379359 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:02:13.379313    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/be441f80-19f8-48c2-b1b2-624b3bddcb27-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-bj22q\" (UID: \"be441f80-19f8-48c2-b1b2-624b3bddcb27\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-bj22q"
May 26 16:02:13.379833 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:02:13.379426    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6610092d-3e19-47ae-8270-ba37dd62232c-registry-tls\") pod \"image-registry-546498794-d7sw5\" (UID: \"6610092d-3e19-47ae-8270-ba37dd62232c\") " pod="openshift-image-registry/image-registry-546498794-d7sw5"
May 26 16:02:13.379833 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:02:13.379456    2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 26 16:02:13.379833 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:02:13.379528    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be441f80-19f8-48c2-b1b2-624b3bddcb27-networking-console-plugin-cert podName:be441f80-19f8-48c2-b1b2-624b3bddcb27 nodeName:}" failed. No retries permitted until 2026-05-26 16:02:29.379511542 +0000 UTC m=+65.855238870 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/be441f80-19f8-48c2-b1b2-624b3bddcb27-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-bj22q" (UID: "be441f80-19f8-48c2-b1b2-624b3bddcb27") : secret "networking-console-plugin-cert" not found
May 26 16:02:13.379833 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:02:13.379571    2572 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 26 16:02:13.379833 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:02:13.379585    2572 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-546498794-d7sw5: secret "image-registry-tls" not found
May 26 16:02:13.379833 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:02:13.379669    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6610092d-3e19-47ae-8270-ba37dd62232c-registry-tls podName:6610092d-3e19-47ae-8270-ba37dd62232c nodeName:}" failed. No retries permitted until 2026-05-26 16:02:29.379651377 +0000 UTC m=+65.855378722 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6610092d-3e19-47ae-8270-ba37dd62232c-registry-tls") pod "image-registry-546498794-d7sw5" (UID: "6610092d-3e19-47ae-8270-ba37dd62232c") : secret "image-registry-tls" not found
May 26 16:02:13.480623 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:02:13.480579    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b87e383-816e-4130-80ba-843dd20a187d-metrics-tls\") pod \"dns-default-fvwnc\" (UID: \"7b87e383-816e-4130-80ba-843dd20a187d\") " pod="openshift-dns/dns-default-fvwnc"
May 26 16:02:13.480781 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:02:13.480734    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 16:02:13.480828 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:02:13.480805    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b87e383-816e-4130-80ba-843dd20a187d-metrics-tls podName:7b87e383-816e-4130-80ba-843dd20a187d nodeName:}" failed. No retries permitted until 2026-05-26 16:02:29.480788782 +0000 UTC m=+65.956516132 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7b87e383-816e-4130-80ba-843dd20a187d-metrics-tls") pod "dns-default-fvwnc" (UID: "7b87e383-816e-4130-80ba-843dd20a187d") : secret "dns-default-metrics-tls" not found
May 26 16:02:13.480828 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:02:13.480824    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5e434be1-fcca-488a-aeff-38268e150853-cert\") pod \"ingress-canary-hqnkg\" (UID: \"5e434be1-fcca-488a-aeff-38268e150853\") " pod="openshift-ingress-canary/ingress-canary-hqnkg"
May 26 16:02:13.480946 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:02:13.480933    2572 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 16:02:13.480982 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:02:13.480972    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e434be1-fcca-488a-aeff-38268e150853-cert podName:5e434be1-fcca-488a-aeff-38268e150853 nodeName:}" failed. No retries permitted until 2026-05-26 16:02:29.480960417 +0000 UTC m=+65.956687752 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5e434be1-fcca-488a-aeff-38268e150853-cert") pod "ingress-canary-hqnkg" (UID: "5e434be1-fcca-488a-aeff-38268e150853") : secret "canary-serving-cert" not found
May 26 16:02:16.359633 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:02:16.359519    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdfbb578c-xqj5q" event={"ID":"cadbc4ee-438a-4e73-b55a-95d246057641","Type":"ContainerStarted","Data":"a0e51802b0947f0fbae1d4cf3f41ab841bc6adf1af9c60bd081be13dcbc3588c"}
May 26 16:02:16.359633 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:02:16.359563    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdfbb578c-xqj5q" event={"ID":"cadbc4ee-438a-4e73-b55a-95d246057641","Type":"ContainerStarted","Data":"06214f2d590d0db6ae67c8d9e48e280f1394fbc517c941177fc256357ea56c6c"}
May 26 16:02:16.377522 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:02:16.377477    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdfbb578c-xqj5q" podStartSLOduration=3.446438457 podStartE2EDuration="20.377465s" podCreationTimestamp="2026-05-26 16:01:56 +0000 UTC" firstStartedPulling="2026-05-26 16:01:59.088779971 +0000 UTC m=+35.564507313" lastFinishedPulling="2026-05-26 16:02:16.01980651 +0000 UTC m=+52.495533856" observedRunningTime="2026-05-26 16:02:16.37656839 +0000 UTC m=+52.852295741" watchObservedRunningTime="2026-05-26 16:02:16.377465 +0000 UTC m=+52.853192349"
May 26 16:02:24.300664 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:02:24.300633    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l2lg5"
May 26 16:02:29.413925 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:02:29.413880    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6610092d-3e19-47ae-8270-ba37dd62232c-registry-tls\") pod \"image-registry-546498794-d7sw5\" (UID: \"6610092d-3e19-47ae-8270-ba37dd62232c\") " pod="openshift-image-registry/image-registry-546498794-d7sw5"
May 26 16:02:29.414318 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:02:29.413972    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/be441f80-19f8-48c2-b1b2-624b3bddcb27-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-bj22q\" (UID: \"be441f80-19f8-48c2-b1b2-624b3bddcb27\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-bj22q"
May 26 16:02:29.414318 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:02:29.414037    2572 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 26 16:02:29.414318 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:02:29.414055    2572 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-546498794-d7sw5: secret "image-registry-tls" not found
May 26 16:02:29.414318 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:02:29.414073    2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 26 16:02:29.414318 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:02:29.414119    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be441f80-19f8-48c2-b1b2-624b3bddcb27-networking-console-plugin-cert podName:be441f80-19f8-48c2-b1b2-624b3bddcb27 nodeName:}" failed. No retries permitted until 2026-05-26 16:03:01.414101573 +0000 UTC m=+97.889828900 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/be441f80-19f8-48c2-b1b2-624b3bddcb27-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-bj22q" (UID: "be441f80-19f8-48c2-b1b2-624b3bddcb27") : secret "networking-console-plugin-cert" not found
May 26 16:02:29.414318 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:02:29.414136    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6610092d-3e19-47ae-8270-ba37dd62232c-registry-tls podName:6610092d-3e19-47ae-8270-ba37dd62232c nodeName:}" failed. No retries permitted until 2026-05-26 16:03:01.414127006 +0000 UTC m=+97.889854340 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6610092d-3e19-47ae-8270-ba37dd62232c-registry-tls") pod "image-registry-546498794-d7sw5" (UID: "6610092d-3e19-47ae-8270-ba37dd62232c") : secret "image-registry-tls" not found
May 26 16:02:29.514898 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:02:29.514858    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b87e383-816e-4130-80ba-843dd20a187d-metrics-tls\") pod \"dns-default-fvwnc\" (UID: \"7b87e383-816e-4130-80ba-843dd20a187d\") " pod="openshift-dns/dns-default-fvwnc"
May 26 16:02:29.515065 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:02:29.514997    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5e434be1-fcca-488a-aeff-38268e150853-cert\") pod \"ingress-canary-hqnkg\" (UID: \"5e434be1-fcca-488a-aeff-38268e150853\") " pod="openshift-ingress-canary/ingress-canary-hqnkg"
May 26 16:02:29.515065 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:02:29.515005    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 16:02:29.515065 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:02:29.515065    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b87e383-816e-4130-80ba-843dd20a187d-metrics-tls podName:7b87e383-816e-4130-80ba-843dd20a187d nodeName:}" failed. No retries permitted until 2026-05-26 16:03:01.515048215 +0000 UTC m=+97.990775543 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7b87e383-816e-4130-80ba-843dd20a187d-metrics-tls") pod "dns-default-fvwnc" (UID: "7b87e383-816e-4130-80ba-843dd20a187d") : secret "dns-default-metrics-tls" not found
May 26 16:02:29.515177 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:02:29.515083    2572 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 16:02:29.515177 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:02:29.515120    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e434be1-fcca-488a-aeff-38268e150853-cert podName:5e434be1-fcca-488a-aeff-38268e150853 nodeName:}" failed. No retries permitted until 2026-05-26 16:03:01.515108767 +0000 UTC m=+97.990836095 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5e434be1-fcca-488a-aeff-38268e150853-cert") pod "ingress-canary-hqnkg" (UID: "5e434be1-fcca-488a-aeff-38268e150853") : secret "canary-serving-cert" not found
May 26 16:02:29.818139 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:02:29.818046    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8828138d-e7ed-4d5c-ba76-26429679c7b1-metrics-certs\") pod \"network-metrics-daemon-qpzqg\" (UID: \"8828138d-e7ed-4d5c-ba76-26429679c7b1\") " pod="openshift-multus/network-metrics-daemon-qpzqg"
May 26 16:02:29.820384 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:02:29.820364    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 26 16:02:29.828958 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:02:29.828939    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 26 16:02:29.829037 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:02:29.829004    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8828138d-e7ed-4d5c-ba76-26429679c7b1-metrics-certs podName:8828138d-e7ed-4d5c-ba76-26429679c7b1 nodeName:}" failed. No retries permitted until 2026-05-26 16:03:33.828980231 +0000 UTC m=+130.304707562 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8828138d-e7ed-4d5c-ba76-26429679c7b1-metrics-certs") pod "network-metrics-daemon-qpzqg" (UID: "8828138d-e7ed-4d5c-ba76-26429679c7b1") : secret "metrics-daemon-secret" not found
May 26 16:02:29.918723 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:02:29.918689    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lbgsb\" (UniqueName: \"kubernetes.io/projected/8f54a9e9-f1b3-4073-bf4e-fef6f31dc622-kube-api-access-lbgsb\") pod \"network-check-target-rvlkc\" (UID: \"8f54a9e9-f1b3-4073-bf4e-fef6f31dc622\") " pod="openshift-network-diagnostics/network-check-target-rvlkc"
May 26 16:02:29.920989 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:02:29.920969    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 26 16:02:29.931614 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:02:29.931579    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 26 16:02:29.943794 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:02:29.943772    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbgsb\" (UniqueName: \"kubernetes.io/projected/8f54a9e9-f1b3-4073-bf4e-fef6f31dc622-kube-api-access-lbgsb\") pod \"network-check-target-rvlkc\" (UID: \"8f54a9e9-f1b3-4073-bf4e-fef6f31dc622\") " pod="openshift-network-diagnostics/network-check-target-rvlkc"
May 26 16:02:29.998301 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:02:29.998271    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bvvx5\""
May 26 16:02:30.006045 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:02:30.006021    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rvlkc"
May 26 16:02:30.125307 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:02:30.125279    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rvlkc"]
May 26 16:02:30.128709 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:02:30.128683    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f54a9e9_f1b3_4073_bf4e_fef6f31dc622.slice/crio-d23a071910aa3afc97aefe211a9d58cbe2668c03e69cf5577d266fbdddc6a3fb WatchSource:0}: Error finding container d23a071910aa3afc97aefe211a9d58cbe2668c03e69cf5577d266fbdddc6a3fb: Status 404 returned error can't find the container with id d23a071910aa3afc97aefe211a9d58cbe2668c03e69cf5577d266fbdddc6a3fb
May 26 16:02:30.388345 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:02:30.388313    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rvlkc" event={"ID":"8f54a9e9-f1b3-4073-bf4e-fef6f31dc622","Type":"ContainerStarted","Data":"d23a071910aa3afc97aefe211a9d58cbe2668c03e69cf5577d266fbdddc6a3fb"}
May 26 16:02:30.726007 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:02:30.725918    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81fbb108-f79a-4018-82a4-0b0e29541320-original-pull-secret\") pod \"global-pull-secret-syncer-2fdgm\" (UID: \"81fbb108-f79a-4018-82a4-0b0e29541320\") " pod="kube-system/global-pull-secret-syncer-2fdgm"
May 26 16:02:30.728315 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:02:30.728293    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 26 16:02:30.739710 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:02:30.739677    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81fbb108-f79a-4018-82a4-0b0e29541320-original-pull-secret\") pod \"global-pull-secret-syncer-2fdgm\" (UID: \"81fbb108-f79a-4018-82a4-0b0e29541320\") " pod="kube-system/global-pull-secret-syncer-2fdgm"
May 26 16:02:30.901421 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:02:30.901388    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2fdgm"
May 26 16:02:31.044230 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:02:31.044200    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2fdgm"]
May 26 16:02:31.047597 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:02:31.047564    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81fbb108_f79a_4018_82a4_0b0e29541320.slice/crio-5c89b990676fd1b014aecf49560572db604bd5ba66abac20fb77559713f04c0a WatchSource:0}: Error finding container 5c89b990676fd1b014aecf49560572db604bd5ba66abac20fb77559713f04c0a: Status 404 returned error can't find the container with id 5c89b990676fd1b014aecf49560572db604bd5ba66abac20fb77559713f04c0a
May 26 16:02:31.391133 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:02:31.391086    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2fdgm" event={"ID":"81fbb108-f79a-4018-82a4-0b0e29541320","Type":"ContainerStarted","Data":"5c89b990676fd1b014aecf49560572db604bd5ba66abac20fb77559713f04c0a"}
May 26 16:02:33.400649 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:02:33.400595    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rvlkc" event={"ID":"8f54a9e9-f1b3-4073-bf4e-fef6f31dc622","Type":"ContainerStarted","Data":"2c54ba3a6759dd029fe4bb2ae6d195eb5ec10e29110445df5ec427a6f7ae33f0"}
May 26 16:02:33.401089 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:02:33.400908    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rvlkc"
May 26 16:02:33.415695 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:02:33.415645    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-rvlkc" podStartSLOduration=66.477517011 podStartE2EDuration="1m9.415627548s" podCreationTimestamp="2026-05-26 16:01:24 +0000 UTC" firstStartedPulling="2026-05-26 16:02:30.131114841 +0000 UTC m=+66.606842168" lastFinishedPulling="2026-05-26 16:02:33.069225363 +0000 UTC m=+69.544952705" observedRunningTime="2026-05-26 16:02:33.415174773 +0000 UTC m=+69.890902135" watchObservedRunningTime="2026-05-26 16:02:33.415627548 +0000 UTC m=+69.891354896"
May 26 16:02:36.413122 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:02:36.413081    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2fdgm" event={"ID":"81fbb108-f79a-4018-82a4-0b0e29541320","Type":"ContainerStarted","Data":"4cf9514d0e0fa2a247f4168e2540b7e7b6894daf2c8e6842c9685c35049f14e0"}
May 26 16:02:36.426515 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:02:36.426376    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-2fdgm" podStartSLOduration=66.114551209 podStartE2EDuration="1m10.426361215s" podCreationTimestamp="2026-05-26 16:01:26 +0000 UTC" firstStartedPulling="2026-05-26 16:02:31.049558374 +0000 UTC m=+67.525285712" lastFinishedPulling="2026-05-26 16:02:35.361368377 +0000 UTC m=+71.837095718" observedRunningTime="2026-05-26 16:02:36.425998233 +0000 UTC m=+72.901725584" watchObservedRunningTime="2026-05-26 16:02:36.426361215 +0000 UTC m=+72.902088566"
May 26 16:03:01.463640 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:03:01.463576    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6610092d-3e19-47ae-8270-ba37dd62232c-registry-tls\") pod \"image-registry-546498794-d7sw5\" (UID: \"6610092d-3e19-47ae-8270-ba37dd62232c\") " pod="openshift-image-registry/image-registry-546498794-d7sw5"
May 26 16:03:01.464063 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:03:01.463666    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/be441f80-19f8-48c2-b1b2-624b3bddcb27-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-bj22q\" (UID: \"be441f80-19f8-48c2-b1b2-624b3bddcb27\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-bj22q"
May 26 16:03:01.464063 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:03:01.463742    2572 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 26 16:03:01.464063 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:03:01.463760    2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 26 16:03:01.464063 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:03:01.463770    2572 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-546498794-d7sw5: secret "image-registry-tls" not found
May 26 16:03:01.464063 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:03:01.463823    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be441f80-19f8-48c2-b1b2-624b3bddcb27-networking-console-plugin-cert podName:be441f80-19f8-48c2-b1b2-624b3bddcb27 nodeName:}" failed. No retries permitted until 2026-05-26 16:04:05.463808023 +0000 UTC m=+161.939535352 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/be441f80-19f8-48c2-b1b2-624b3bddcb27-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-bj22q" (UID: "be441f80-19f8-48c2-b1b2-624b3bddcb27") : secret "networking-console-plugin-cert" not found
May 26 16:03:01.464063 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:03:01.463837    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6610092d-3e19-47ae-8270-ba37dd62232c-registry-tls podName:6610092d-3e19-47ae-8270-ba37dd62232c nodeName:}" failed. No retries permitted until 2026-05-26 16:04:05.463829909 +0000 UTC m=+161.939557237 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6610092d-3e19-47ae-8270-ba37dd62232c-registry-tls") pod "image-registry-546498794-d7sw5" (UID: "6610092d-3e19-47ae-8270-ba37dd62232c") : secret "image-registry-tls" not found
May 26 16:03:01.564183 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:03:01.564145    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b87e383-816e-4130-80ba-843dd20a187d-metrics-tls\") pod \"dns-default-fvwnc\" (UID: \"7b87e383-816e-4130-80ba-843dd20a187d\") " pod="openshift-dns/dns-default-fvwnc"
May 26 16:03:01.564360 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:03:01.564230    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5e434be1-fcca-488a-aeff-38268e150853-cert\") pod \"ingress-canary-hqnkg\" (UID: \"5e434be1-fcca-488a-aeff-38268e150853\") " pod="openshift-ingress-canary/ingress-canary-hqnkg"
May 26 16:03:01.564360 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:03:01.564305    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 16:03:01.564438 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:03:01.564368    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b87e383-816e-4130-80ba-843dd20a187d-metrics-tls podName:7b87e383-816e-4130-80ba-843dd20a187d nodeName:}" failed. No retries permitted until 2026-05-26 16:04:05.564351644 +0000 UTC m=+162.040078979 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7b87e383-816e-4130-80ba-843dd20a187d-metrics-tls") pod "dns-default-fvwnc" (UID: "7b87e383-816e-4130-80ba-843dd20a187d") : secret "dns-default-metrics-tls" not found
May 26 16:03:01.564438 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:03:01.564370    2572 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 16:03:01.564438 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:03:01.564426    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e434be1-fcca-488a-aeff-38268e150853-cert podName:5e434be1-fcca-488a-aeff-38268e150853 nodeName:}" failed. No retries permitted until 2026-05-26 16:04:05.564413888 +0000 UTC m=+162.040141216 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5e434be1-fcca-488a-aeff-38268e150853-cert") pod "ingress-canary-hqnkg" (UID: "5e434be1-fcca-488a-aeff-38268e150853") : secret "canary-serving-cert" not found
May 26 16:03:04.406584 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:03:04.406554    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-rvlkc"
May 26 16:03:33.917818 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:03:33.917776    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8828138d-e7ed-4d5c-ba76-26429679c7b1-metrics-certs\") pod \"network-metrics-daemon-qpzqg\" (UID: \"8828138d-e7ed-4d5c-ba76-26429679c7b1\") " pod="openshift-multus/network-metrics-daemon-qpzqg"
May 26 16:03:33.918315 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:03:33.917931    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 26 16:03:33.918315 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:03:33.918004    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8828138d-e7ed-4d5c-ba76-26429679c7b1-metrics-certs podName:8828138d-e7ed-4d5c-ba76-26429679c7b1 nodeName:}" failed. No retries permitted until 2026-05-26 16:05:35.917988472 +0000 UTC m=+252.393715799 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8828138d-e7ed-4d5c-ba76-26429679c7b1-metrics-certs") pod "network-metrics-daemon-qpzqg" (UID: "8828138d-e7ed-4d5c-ba76-26429679c7b1") : secret "metrics-daemon-secret" not found
May 26 16:03:52.562369 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:03:52.562335    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lvfvd_02c134e5-0f87-48c2-80ed-427d242f1435/dns-node-resolver/0.log"
May 26 16:03:53.361358 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:03:53.361334    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qdk28_10ea1129-78a4-4d29-ab50-50b69b6d30c1/node-ca/0.log"
May 26 16:04:00.477877 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:04:00.477823    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-bj22q" podUID="be441f80-19f8-48c2-b1b2-624b3bddcb27"
May 26 16:04:00.499187 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:04:00.499142    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-546498794-d7sw5" podUID="6610092d-3e19-47ae-8270-ba37dd62232c"
May 26 16:04:00.582549 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:04:00.582500    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-hqnkg" podUID="5e434be1-fcca-488a-aeff-38268e150853"
May 26 16:04:00.603996 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:00.603971    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-546498794-d7sw5"
May 26 16:04:00.604144 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:00.603972    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hqnkg"
May 26 16:04:00.604144 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:00.603972    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-bj22q"
May 26 16:04:00.607557 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:04:00.607532    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-fvwnc" podUID="7b87e383-816e-4130-80ba-843dd20a187d"
May 26 16:04:01.087474 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:04:01.087437    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-qpzqg" podUID="8828138d-e7ed-4d5c-ba76-26429679c7b1"
May 26 16:04:01.606025 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:01.605996    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fvwnc"
May 26 16:04:05.468395 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:05.468335    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6610092d-3e19-47ae-8270-ba37dd62232c-registry-tls\") pod \"image-registry-546498794-d7sw5\" (UID: \"6610092d-3e19-47ae-8270-ba37dd62232c\") " pod="openshift-image-registry/image-registry-546498794-d7sw5"
May 26 16:04:05.468395 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:05.468412    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/be441f80-19f8-48c2-b1b2-624b3bddcb27-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-bj22q\" (UID: \"be441f80-19f8-48c2-b1b2-624b3bddcb27\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-bj22q"
May 26 16:04:05.470809 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:05.470782    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/be441f80-19f8-48c2-b1b2-624b3bddcb27-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-bj22q\" (UID: \"be441f80-19f8-48c2-b1b2-624b3bddcb27\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-bj22q"
May 26 16:04:05.470912 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:05.470864    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6610092d-3e19-47ae-8270-ba37dd62232c-registry-tls\") pod \"image-registry-546498794-d7sw5\" (UID: \"6610092d-3e19-47ae-8270-ba37dd62232c\") " pod="openshift-image-registry/image-registry-546498794-d7sw5"
May 26 16:04:05.569137 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:05.569080    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5e434be1-fcca-488a-aeff-38268e150853-cert\") pod \"ingress-canary-hqnkg\" (UID: \"5e434be1-fcca-488a-aeff-38268e150853\") " pod="openshift-ingress-canary/ingress-canary-hqnkg"
May 26 16:04:05.569343 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:05.569180    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b87e383-816e-4130-80ba-843dd20a187d-metrics-tls\") pod \"dns-default-fvwnc\" (UID: \"7b87e383-816e-4130-80ba-843dd20a187d\") " pod="openshift-dns/dns-default-fvwnc"
May 26 16:04:05.571489 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:05.571460    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b87e383-816e-4130-80ba-843dd20a187d-metrics-tls\") pod \"dns-default-fvwnc\" (UID: \"7b87e383-816e-4130-80ba-843dd20a187d\") " pod="openshift-dns/dns-default-fvwnc"
May 26 16:04:05.571598 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:05.571490    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5e434be1-fcca-488a-aeff-38268e150853-cert\") pod \"ingress-canary-hqnkg\" (UID: \"5e434be1-fcca-488a-aeff-38268e150853\") " pod="openshift-ingress-canary/ingress-canary-hqnkg"
May 26 16:04:05.708150 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:05.708119    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-q98vj\""
May 26 16:04:05.708150 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:05.708124    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-q4j2j\""
May 26 16:04:05.708351 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:05.708170    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vkcq2\""
May 26 16:04:05.715860 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:05.715837    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-bj22q"
May 26 16:04:05.715978 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:05.715863    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-546498794-d7sw5"
May 26 16:04:05.715978 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:05.715940    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hqnkg"
May 26 16:04:05.808642 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:05.808615    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-t9x9h\""
May 26 16:04:05.816891 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:05.816865    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fvwnc"
May 26 16:04:05.874508 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:05.874474    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-546498794-d7sw5"]
May 26 16:04:05.879619 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:05.879542    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hqnkg"]
May 26 16:04:05.884378 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:04:05.884352    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e434be1_fcca_488a_aeff_38268e150853.slice/crio-7cb983e71647ac8a1209763c624359ab5b6c8ae4b9d81ecd7c8127209bfa04a8 WatchSource:0}: Error finding container 7cb983e71647ac8a1209763c624359ab5b6c8ae4b9d81ecd7c8127209bfa04a8: Status 404 returned error can't find the container with id 7cb983e71647ac8a1209763c624359ab5b6c8ae4b9d81ecd7c8127209bfa04a8
May 26 16:04:05.898692 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:05.898663    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-d4d8f94f9-bj22q"]
May 26 16:04:05.910698 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:04:05.910482    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe441f80_19f8_48c2_b1b2_624b3bddcb27.slice/crio-441b6fc122c423cec0717092da5550775202c0cf0e6ddd7119576dde69b654b0 WatchSource:0}: Error finding container 441b6fc122c423cec0717092da5550775202c0cf0e6ddd7119576dde69b654b0: Status 404 returned error can't find the container with id 441b6fc122c423cec0717092da5550775202c0cf0e6ddd7119576dde69b654b0
May 26 16:04:05.954871 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:05.954845    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fvwnc"]
May 26 16:04:05.959870 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:04:05.959842    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b87e383_816e_4130_80ba_843dd20a187d.slice/crio-33158c1f699679172d6de7498989c4d478ff4c72bd95fed67a461d2f6eaf64d3 WatchSource:0}: Error finding container 33158c1f699679172d6de7498989c4d478ff4c72bd95fed67a461d2f6eaf64d3: Status 404 returned error can't find the container with id 33158c1f699679172d6de7498989c4d478ff4c72bd95fed67a461d2f6eaf64d3
May 26 16:04:06.619270 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:06.619226    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hqnkg" event={"ID":"5e434be1-fcca-488a-aeff-38268e150853","Type":"ContainerStarted","Data":"7cb983e71647ac8a1209763c624359ab5b6c8ae4b9d81ecd7c8127209bfa04a8"}
May 26 16:04:06.621113 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:06.621063    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-546498794-d7sw5" event={"ID":"6610092d-3e19-47ae-8270-ba37dd62232c","Type":"ContainerStarted","Data":"cf402b6595584bb51e95fc69498798b08652043436a97d9636510d8153e8c41f"}
May 26 16:04:06.621113 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:06.621103    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-546498794-d7sw5" event={"ID":"6610092d-3e19-47ae-8270-ba37dd62232c","Type":"ContainerStarted","Data":"1e24992606161ea101260bfc1672d854929fd71fa3e1f5baea4bcc7bd732aeee"}
May 26 16:04:06.621325 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:06.621184    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-546498794-d7sw5"
May 26 16:04:06.622541 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:06.622513    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fvwnc" event={"ID":"7b87e383-816e-4130-80ba-843dd20a187d","Type":"ContainerStarted","Data":"33158c1f699679172d6de7498989c4d478ff4c72bd95fed67a461d2f6eaf64d3"}
May 26 16:04:06.623620 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:06.623582    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-bj22q" event={"ID":"be441f80-19f8-48c2-b1b2-624b3bddcb27","Type":"ContainerStarted","Data":"441b6fc122c423cec0717092da5550775202c0cf0e6ddd7119576dde69b654b0"}
May 26 16:04:06.640547 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:06.640487    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-546498794-d7sw5" podStartSLOduration=162.640469898 podStartE2EDuration="2m42.640469898s" podCreationTimestamp="2026-05-26 16:01:24 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:04:06.640451436 +0000 UTC m=+163.116178786" watchObservedRunningTime="2026-05-26 16:04:06.640469898 +0000 UTC m=+163.116197247"
May 26 16:04:07.338412 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:07.338316    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c66b498d4-wltk2" podUID="d6f02a68-6e6d-4a7e-850e-3172a9f2bd55" containerName="acm-agent" probeResult="failure" output="Get \"http://10.132.0.11:8000/readyz\": dial tcp 10.132.0.11:8000: connect: connection refused"
May 26 16:04:07.627741 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:07.627700    2572 generic.go:358] "Generic (PLEG): container finished" podID="3a61b0a0-aa24-4a24-8e10-8a23103bf3bf" containerID="d9133d60490dc7fcb8a83e7d2bee0ff74a1dcbcd3ba4a6b24684a1b521d0ac8e" exitCode=255
May 26 16:04:07.628173 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:07.627782    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cf978fc94-qtgk9" event={"ID":"3a61b0a0-aa24-4a24-8e10-8a23103bf3bf","Type":"ContainerDied","Data":"d9133d60490dc7fcb8a83e7d2bee0ff74a1dcbcd3ba4a6b24684a1b521d0ac8e"}
May 26 16:04:07.628173 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:07.628152    2572 scope.go:117] "RemoveContainer" containerID="d9133d60490dc7fcb8a83e7d2bee0ff74a1dcbcd3ba4a6b24684a1b521d0ac8e"
May 26 16:04:07.629331 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:07.629304    2572 generic.go:358] "Generic (PLEG): container finished" podID="d6f02a68-6e6d-4a7e-850e-3172a9f2bd55" containerID="98abf90a386fbf27a0b6d391aa91f151a44358aff5343b04f6eb20bfbc89a270" exitCode=1
May 26 16:04:07.629430 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:07.629374    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c66b498d4-wltk2" event={"ID":"d6f02a68-6e6d-4a7e-850e-3172a9f2bd55","Type":"ContainerDied","Data":"98abf90a386fbf27a0b6d391aa91f151a44358aff5343b04f6eb20bfbc89a270"}
May 26 16:04:07.629812 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:07.629794    2572 scope.go:117] "RemoveContainer" containerID="98abf90a386fbf27a0b6d391aa91f151a44358aff5343b04f6eb20bfbc89a270"
May 26 16:04:07.836514 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:07.836479    2572 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cf978fc94-qtgk9"
May 26 16:04:07.873622 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:07.873576    2572 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c66b498d4-wltk2"
May 26 16:04:08.633914 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:08.633875    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c66b498d4-wltk2" event={"ID":"d6f02a68-6e6d-4a7e-850e-3172a9f2bd55","Type":"ContainerStarted","Data":"c943ad8fc7256df9de95586c914f47d8eacf79e52c9f25bee3e1bee5202d1a29"}
May 26 16:04:08.634367 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:08.634062    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c66b498d4-wltk2"
May 26 16:04:08.635116 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:08.635094    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c66b498d4-wltk2"
May 26 16:04:08.635282 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:08.635260    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-bj22q" event={"ID":"be441f80-19f8-48c2-b1b2-624b3bddcb27","Type":"ContainerStarted","Data":"1ba78e7f8bb3c0065025b29e0cd94d434655cde5a4bdc84402ad56d954d3dd90"}
May 26 16:04:08.636717 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:08.636691    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cf978fc94-qtgk9" event={"ID":"3a61b0a0-aa24-4a24-8e10-8a23103bf3bf","Type":"ContainerStarted","Data":"3dc3ffbdc20a1dee3c30c9ecba9a1b29e067d0cee9606ef4259f025a7a7f8a5d"}
May 26 16:04:08.637932 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:08.637915    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hqnkg" event={"ID":"5e434be1-fcca-488a-aeff-38268e150853","Type":"ContainerStarted","Data":"c3fa95ac40357b7b386760c52579ccb40aaf007b5d8b0fc441c14839defcb882"}
May 26 16:04:08.639367 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:08.639348    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fvwnc" event={"ID":"7b87e383-816e-4130-80ba-843dd20a187d","Type":"ContainerStarted","Data":"cbcf460d061bf78ad988d590076e5c50f6a39435b141ed6db02d86f6e8142e95"}
May 26 16:04:08.639367 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:08.639368    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fvwnc" event={"ID":"7b87e383-816e-4130-80ba-843dd20a187d","Type":"ContainerStarted","Data":"e5eb9e8bdf50823f95f3037cb6b9ebb120fbc3ba94f8b175218551af92548dbb"}
May 26 16:04:08.639528 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:08.639469    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-fvwnc"
May 26 16:04:08.661893 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:08.661844    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hqnkg" podStartSLOduration=129.439542339 podStartE2EDuration="2m11.661804518s" podCreationTimestamp="2026-05-26 16:01:57 +0000 UTC" firstStartedPulling="2026-05-26 16:04:05.886199008 +0000 UTC m=+162.361926336" lastFinishedPulling="2026-05-26 16:04:08.108461171 +0000 UTC m=+164.584188515" observedRunningTime="2026-05-26 16:04:08.661395817 +0000 UTC m=+165.137123190" watchObservedRunningTime="2026-05-26 16:04:08.661804518 +0000 UTC m=+165.137531868"
May 26 16:04:08.676546 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:08.676498    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-fvwnc" podStartSLOduration=129.533950144 podStartE2EDuration="2m11.67648311s" podCreationTimestamp="2026-05-26 16:01:57 +0000 UTC" firstStartedPulling="2026-05-26 16:04:05.961800978 +0000 UTC m=+162.437528307" lastFinishedPulling="2026-05-26 16:04:08.104333932 +0000 UTC m=+164.580061273" observedRunningTime="2026-05-26 16:04:08.67557702 +0000 UTC m=+165.151304371" watchObservedRunningTime="2026-05-26 16:04:08.67648311 +0000 UTC m=+165.152210459"
May 26 16:04:08.692006 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:08.691955    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-bj22q" podStartSLOduration=158.495237456 podStartE2EDuration="2m40.691941294s" podCreationTimestamp="2026-05-26 16:01:28 +0000 UTC" firstStartedPulling="2026-05-26 16:04:05.912685271 +0000 UTC m=+162.388412608" lastFinishedPulling="2026-05-26 16:04:08.109389106 +0000 UTC m=+164.585116446" observedRunningTime="2026-05-26 16:04:08.691673038 +0000 UTC m=+165.167400395" watchObservedRunningTime="2026-05-26 16:04:08.691941294 +0000 UTC m=+165.167668644"
May 26 16:04:13.075642 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:13.075575    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qpzqg"
May 26 16:04:14.050456 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:14.050420    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-xsqdx"]
May 26 16:04:14.056203 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:14.056185    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xsqdx"
May 26 16:04:14.058885 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:14.058865    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
May 26 16:04:14.059642 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:14.059625    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
May 26 16:04:14.059642 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:14.059636    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
May 26 16:04:14.059804 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:14.059629    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
May 26 16:04:14.060646 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:14.060629    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7shss\""
May 26 16:04:14.072322 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:14.072300    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xsqdx"]
May 26 16:04:14.138411 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:14.138373    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/556b4c71-e309-4091-88b9-c1073bd4ede3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xsqdx\" (UID: \"556b4c71-e309-4091-88b9-c1073bd4ede3\") " pod="openshift-insights/insights-runtime-extractor-xsqdx"
May 26 16:04:14.138411 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:14.138417    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/556b4c71-e309-4091-88b9-c1073bd4ede3-crio-socket\") pod \"insights-runtime-extractor-xsqdx\" (UID: \"556b4c71-e309-4091-88b9-c1073bd4ede3\") " pod="openshift-insights/insights-runtime-extractor-xsqdx"
May 26 16:04:14.138911 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:14.138489    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/556b4c71-e309-4091-88b9-c1073bd4ede3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xsqdx\" (UID: \"556b4c71-e309-4091-88b9-c1073bd4ede3\") " pod="openshift-insights/insights-runtime-extractor-xsqdx"
May 26 16:04:14.138911 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:14.138561    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/556b4c71-e309-4091-88b9-c1073bd4ede3-data-volume\") pod \"insights-runtime-extractor-xsqdx\" (UID: \"556b4c71-e309-4091-88b9-c1073bd4ede3\") " pod="openshift-insights/insights-runtime-extractor-xsqdx"
May 26 16:04:14.138911 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:14.138665    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npxb2\" (UniqueName: \"kubernetes.io/projected/556b4c71-e309-4091-88b9-c1073bd4ede3-kube-api-access-npxb2\") pod \"insights-runtime-extractor-xsqdx\" (UID: \"556b4c71-e309-4091-88b9-c1073bd4ede3\") " pod="openshift-insights/insights-runtime-extractor-xsqdx"
May 26 16:04:14.239290 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:14.239253    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-npxb2\" (UniqueName: \"kubernetes.io/projected/556b4c71-e309-4091-88b9-c1073bd4ede3-kube-api-access-npxb2\") pod \"insights-runtime-extractor-xsqdx\" (UID: \"556b4c71-e309-4091-88b9-c1073bd4ede3\") " pod="openshift-insights/insights-runtime-extractor-xsqdx"
May 26 16:04:14.239498 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:14.239316    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/556b4c71-e309-4091-88b9-c1073bd4ede3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xsqdx\" (UID: \"556b4c71-e309-4091-88b9-c1073bd4ede3\") " pod="openshift-insights/insights-runtime-extractor-xsqdx"
May 26 16:04:14.239498 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:14.239348    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/556b4c71-e309-4091-88b9-c1073bd4ede3-crio-socket\") pod \"insights-runtime-extractor-xsqdx\" (UID: \"556b4c71-e309-4091-88b9-c1073bd4ede3\") " pod="openshift-insights/insights-runtime-extractor-xsqdx"
May 26 16:04:14.239498 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:14.239374    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/556b4c71-e309-4091-88b9-c1073bd4ede3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xsqdx\" (UID: \"556b4c71-e309-4091-88b9-c1073bd4ede3\") " pod="openshift-insights/insights-runtime-extractor-xsqdx"
May 26 16:04:14.239498 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:14.239400    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/556b4c71-e309-4091-88b9-c1073bd4ede3-data-volume\") pod \"insights-runtime-extractor-xsqdx\" (UID: \"556b4c71-e309-4091-88b9-c1073bd4ede3\") " pod="openshift-insights/insights-runtime-extractor-xsqdx"
May 26 16:04:14.239498 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:14.239461    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/556b4c71-e309-4091-88b9-c1073bd4ede3-crio-socket\") pod \"insights-runtime-extractor-xsqdx\" (UID: \"556b4c71-e309-4091-88b9-c1073bd4ede3\") " pod="openshift-insights/insights-runtime-extractor-xsqdx"
May 26 16:04:14.239886 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:14.239865    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/556b4c71-e309-4091-88b9-c1073bd4ede3-data-volume\") pod \"insights-runtime-extractor-xsqdx\" (UID: \"556b4c71-e309-4091-88b9-c1073bd4ede3\") " pod="openshift-insights/insights-runtime-extractor-xsqdx"
May 26 16:04:14.239957 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:14.239944    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/556b4c71-e309-4091-88b9-c1073bd4ede3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xsqdx\" (UID: \"556b4c71-e309-4091-88b9-c1073bd4ede3\") " pod="openshift-insights/insights-runtime-extractor-xsqdx"
May 26 16:04:14.241711 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:14.241693    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/556b4c71-e309-4091-88b9-c1073bd4ede3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xsqdx\" (UID: \"556b4c71-e309-4091-88b9-c1073bd4ede3\") " pod="openshift-insights/insights-runtime-extractor-xsqdx"
May 26 16:04:14.247587 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:14.247563    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npxb2\" (UniqueName: \"kubernetes.io/projected/556b4c71-e309-4091-88b9-c1073bd4ede3-kube-api-access-npxb2\") pod \"insights-runtime-extractor-xsqdx\" (UID: \"556b4c71-e309-4091-88b9-c1073bd4ede3\") " pod="openshift-insights/insights-runtime-extractor-xsqdx"
May 26 16:04:14.366464 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:14.366348    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xsqdx"
May 26 16:04:14.501746 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:14.501711    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xsqdx"]
May 26 16:04:14.504890 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:04:14.504863    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod556b4c71_e309_4091_88b9_c1073bd4ede3.slice/crio-5c52e5fbc5cc174577ff7b8b200f8e53b09b8aa297a7ae2740daefa76af466a8 WatchSource:0}: Error finding container 5c52e5fbc5cc174577ff7b8b200f8e53b09b8aa297a7ae2740daefa76af466a8: Status 404 returned error can't find the container with id 5c52e5fbc5cc174577ff7b8b200f8e53b09b8aa297a7ae2740daefa76af466a8
May 26 16:04:14.656236 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:14.656200    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xsqdx" event={"ID":"556b4c71-e309-4091-88b9-c1073bd4ede3","Type":"ContainerStarted","Data":"1653dfd179dea7dd9c560a8b1ba2b0e973a6c45c6fe8e621e2797eda338c959f"}
May 26 16:04:14.656236 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:14.656236    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xsqdx" event={"ID":"556b4c71-e309-4091-88b9-c1073bd4ede3","Type":"ContainerStarted","Data":"5c52e5fbc5cc174577ff7b8b200f8e53b09b8aa297a7ae2740daefa76af466a8"}
May 26 16:04:15.660064 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:15.660033    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xsqdx" event={"ID":"556b4c71-e309-4091-88b9-c1073bd4ede3","Type":"ContainerStarted","Data":"2a039bc054ff01c3f6c59b80b031a36af7a9befe9770130373132c4e67a3325b"}
May 26 16:04:17.666158 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:17.666120    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xsqdx" event={"ID":"556b4c71-e309-4091-88b9-c1073bd4ede3","Type":"ContainerStarted","Data":"60a00fb6edd761058217143257bd62d0d96f7750dc53f9927f7676446dfb84e1"}
May 26 16:04:17.683191 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:17.683134    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-xsqdx" podStartSLOduration=1.33157429 podStartE2EDuration="3.683120863s" podCreationTimestamp="2026-05-26 16:04:14 +0000 UTC" firstStartedPulling="2026-05-26 16:04:14.557813075 +0000 UTC m=+171.033540405" lastFinishedPulling="2026-05-26 16:04:16.909359632 +0000 UTC m=+173.385086978" observedRunningTime="2026-05-26 16:04:17.682006169 +0000 UTC m=+174.157733521" watchObservedRunningTime="2026-05-26 16:04:17.683120863 +0000 UTC m=+174.158848211"
May 26 16:04:18.644986 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:18.644960    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-fvwnc"
May 26 16:04:25.720450 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:25.720407    2572 patch_prober.go:28] interesting pod/image-registry-546498794-d7sw5 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
May 26 16:04:25.720837 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:25.720461    2572 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-546498794-d7sw5" podUID="6610092d-3e19-47ae-8270-ba37dd62232c" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:04:27.634360 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:27.634328    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-546498794-d7sw5"
May 26 16:04:36.300836 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:36.300800    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-546498794-d7sw5"]
May 26 16:04:36.438950 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:36.438914    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-hjs7w"]
May 26 16:04:36.443397 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:36.443377    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hjs7w"
May 26 16:04:36.445702 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:36.445681    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
May 26 16:04:36.445847 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:36.445720    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
May 26 16:04:36.446067 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:36.446044    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
May 26 16:04:36.446236 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:36.446211    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
May 26 16:04:36.446361 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:36.446251    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
May 26 16:04:36.446361 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:36.446268    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5w4vj\""
May 26 16:04:36.446361 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:36.446349    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
May 26 16:04:36.519192 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:36.519151    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b94c5ba6-2dd8-47f7-8cba-723af642e2e3-node-exporter-wtmp\") pod \"node-exporter-hjs7w\" (UID: \"b94c5ba6-2dd8-47f7-8cba-723af642e2e3\") " pod="openshift-monitoring/node-exporter-hjs7w"
May 26 16:04:36.519370 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:36.519203    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b94c5ba6-2dd8-47f7-8cba-723af642e2e3-node-exporter-tls\") pod \"node-exporter-hjs7w\" (UID: \"b94c5ba6-2dd8-47f7-8cba-723af642e2e3\") " pod="openshift-monitoring/node-exporter-hjs7w"
May 26 16:04:36.519370 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:36.519233    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b94c5ba6-2dd8-47f7-8cba-723af642e2e3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hjs7w\" (UID: \"b94c5ba6-2dd8-47f7-8cba-723af642e2e3\") " pod="openshift-monitoring/node-exporter-hjs7w"
May 26 16:04:36.519370 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:36.519272    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b94c5ba6-2dd8-47f7-8cba-723af642e2e3-root\") pod \"node-exporter-hjs7w\" (UID: \"b94c5ba6-2dd8-47f7-8cba-723af642e2e3\") " pod="openshift-monitoring/node-exporter-hjs7w"
May 26 16:04:36.519370 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:36.519305    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b94c5ba6-2dd8-47f7-8cba-723af642e2e3-sys\") pod \"node-exporter-hjs7w\" (UID: \"b94c5ba6-2dd8-47f7-8cba-723af642e2e3\") " pod="openshift-monitoring/node-exporter-hjs7w"
May 26 16:04:36.519370 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:36.519360    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b94c5ba6-2dd8-47f7-8cba-723af642e2e3-metrics-client-ca\") pod \"node-exporter-hjs7w\" (UID: \"b94c5ba6-2dd8-47f7-8cba-723af642e2e3\") " pod="openshift-monitoring/node-exporter-hjs7w"
May 26 16:04:36.519551 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:36.519405    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b94c5ba6-2dd8-47f7-8cba-723af642e2e3-node-exporter-accelerators-collector-config\") pod \"node-exporter-hjs7w\" (UID: \"b94c5ba6-2dd8-47f7-8cba-723af642e2e3\") " pod="openshift-monitoring/node-exporter-hjs7w"
May 26 16:04:36.519551 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:36.519466    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b94c5ba6-2dd8-47f7-8cba-723af642e2e3-node-exporter-textfile\") pod \"node-exporter-hjs7w\" (UID: \"b94c5ba6-2dd8-47f7-8cba-723af642e2e3\") " pod="openshift-monitoring/node-exporter-hjs7w"
May 26 16:04:36.519551 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:36.519486    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgngx\" (UniqueName: \"kubernetes.io/projected/b94c5ba6-2dd8-47f7-8cba-723af642e2e3-kube-api-access-xgngx\") pod \"node-exporter-hjs7w\" (UID: \"b94c5ba6-2dd8-47f7-8cba-723af642e2e3\") " pod="openshift-monitoring/node-exporter-hjs7w"
May 26 16:04:36.620453 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:36.620423    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b94c5ba6-2dd8-47f7-8cba-723af642e2e3-metrics-client-ca\") pod \"node-exporter-hjs7w\" (UID: \"b94c5ba6-2dd8-47f7-8cba-723af642e2e3\") " pod="openshift-monitoring/node-exporter-hjs7w"
May 26 16:04:36.620587 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:36.620462    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b94c5ba6-2dd8-47f7-8cba-723af642e2e3-node-exporter-accelerators-collector-config\") pod \"node-exporter-hjs7w\" (UID: \"b94c5ba6-2dd8-47f7-8cba-723af642e2e3\") " pod="openshift-monitoring/node-exporter-hjs7w"
May 26 16:04:36.620587 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:36.620497    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b94c5ba6-2dd8-47f7-8cba-723af642e2e3-node-exporter-textfile\") pod \"node-exporter-hjs7w\" (UID: \"b94c5ba6-2dd8-47f7-8cba-723af642e2e3\") " pod="openshift-monitoring/node-exporter-hjs7w"
May 26 16:04:36.620587 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:36.620520    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xgngx\" (UniqueName: \"kubernetes.io/projected/b94c5ba6-2dd8-47f7-8cba-723af642e2e3-kube-api-access-xgngx\") pod \"node-exporter-hjs7w\" (UID: \"b94c5ba6-2dd8-47f7-8cba-723af642e2e3\") " pod="openshift-monitoring/node-exporter-hjs7w"
May 26 16:04:36.620587 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:36.620556    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b94c5ba6-2dd8-47f7-8cba-723af642e2e3-node-exporter-wtmp\") pod \"node-exporter-hjs7w\" (UID: \"b94c5ba6-2dd8-47f7-8cba-723af642e2e3\") " pod="openshift-monitoring/node-exporter-hjs7w"
May 26 16:04:36.620805 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:36.620586    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b94c5ba6-2dd8-47f7-8cba-723af642e2e3-node-exporter-tls\") pod \"node-exporter-hjs7w\" (UID: \"b94c5ba6-2dd8-47f7-8cba-723af642e2e3\") " pod="openshift-monitoring/node-exporter-hjs7w"
May 26 16:04:36.620805 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:04:36.620701    2572 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
May 26 16:04:36.620805 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:36.620733    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b94c5ba6-2dd8-47f7-8cba-723af642e2e3-node-exporter-wtmp\") pod \"node-exporter-hjs7w\" (UID: \"b94c5ba6-2dd8-47f7-8cba-723af642e2e3\") " pod="openshift-monitoring/node-exporter-hjs7w"
May 26 16:04:36.620805 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:04:36.620776    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b94c5ba6-2dd8-47f7-8cba-723af642e2e3-node-exporter-tls podName:b94c5ba6-2dd8-47f7-8cba-723af642e2e3 nodeName:}" failed. No retries permitted until 2026-05-26 16:04:37.120754461 +0000 UTC m=+193.596481790 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/b94c5ba6-2dd8-47f7-8cba-723af642e2e3-node-exporter-tls") pod "node-exporter-hjs7w" (UID: "b94c5ba6-2dd8-47f7-8cba-723af642e2e3") : secret "node-exporter-tls" not found
May 26 16:04:36.620805 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:36.620799    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b94c5ba6-2dd8-47f7-8cba-723af642e2e3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hjs7w\" (UID: \"b94c5ba6-2dd8-47f7-8cba-723af642e2e3\") " pod="openshift-monitoring/node-exporter-hjs7w"
May 26 16:04:36.621120 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:36.620851    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b94c5ba6-2dd8-47f7-8cba-723af642e2e3-root\") pod \"node-exporter-hjs7w\" (UID: \"b94c5ba6-2dd8-47f7-8cba-723af642e2e3\") " pod="openshift-monitoring/node-exporter-hjs7w"
May 26 16:04:36.621120 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:36.620870    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b94c5ba6-2dd8-47f7-8cba-723af642e2e3-node-exporter-textfile\") pod \"node-exporter-hjs7w\" (UID: \"b94c5ba6-2dd8-47f7-8cba-723af642e2e3\") " pod="openshift-monitoring/node-exporter-hjs7w"
May 26 16:04:36.621120 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:36.620894    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b94c5ba6-2dd8-47f7-8cba-723af642e2e3-sys\") pod \"node-exporter-hjs7w\" (UID: \"b94c5ba6-2dd8-47f7-8cba-723af642e2e3\") " pod="openshift-monitoring/node-exporter-hjs7w"
May 26 16:04:36.621120 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:36.620932    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b94c5ba6-2dd8-47f7-8cba-723af642e2e3-root\") pod \"node-exporter-hjs7w\" (UID: \"b94c5ba6-2dd8-47f7-8cba-723af642e2e3\") " pod="openshift-monitoring/node-exporter-hjs7w"
May 26 16:04:36.621120 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:36.620983    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b94c5ba6-2dd8-47f7-8cba-723af642e2e3-sys\") pod \"node-exporter-hjs7w\" (UID: \"b94c5ba6-2dd8-47f7-8cba-723af642e2e3\") " pod="openshift-monitoring/node-exporter-hjs7w"
May 26 16:04:36.621356 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:36.621152    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b94c5ba6-2dd8-47f7-8cba-723af642e2e3-metrics-client-ca\") pod \"node-exporter-hjs7w\" (UID: \"b94c5ba6-2dd8-47f7-8cba-723af642e2e3\") " pod="openshift-monitoring/node-exporter-hjs7w"
May 26 16:04:36.621356 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:36.621274    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b94c5ba6-2dd8-47f7-8cba-723af642e2e3-node-exporter-accelerators-collector-config\") pod \"node-exporter-hjs7w\" (UID: \"b94c5ba6-2dd8-47f7-8cba-723af642e2e3\") " pod="openshift-monitoring/node-exporter-hjs7w"
May 26 16:04:36.623119 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:36.623101    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b94c5ba6-2dd8-47f7-8cba-723af642e2e3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hjs7w\" (UID: \"b94c5ba6-2dd8-47f7-8cba-723af642e2e3\") " pod="openshift-monitoring/node-exporter-hjs7w"
May 26 16:04:36.628639 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:36.628598    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgngx\" (UniqueName: \"kubernetes.io/projected/b94c5ba6-2dd8-47f7-8cba-723af642e2e3-kube-api-access-xgngx\") pod \"node-exporter-hjs7w\" (UID: \"b94c5ba6-2dd8-47f7-8cba-723af642e2e3\") " pod="openshift-monitoring/node-exporter-hjs7w"
May 26 16:04:37.125348 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:37.125312    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b94c5ba6-2dd8-47f7-8cba-723af642e2e3-node-exporter-tls\") pod \"node-exporter-hjs7w\" (UID: \"b94c5ba6-2dd8-47f7-8cba-723af642e2e3\") " pod="openshift-monitoring/node-exporter-hjs7w"
May 26 16:04:37.127589 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:37.127557    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b94c5ba6-2dd8-47f7-8cba-723af642e2e3-node-exporter-tls\") pod \"node-exporter-hjs7w\" (UID: \"b94c5ba6-2dd8-47f7-8cba-723af642e2e3\") " pod="openshift-monitoring/node-exporter-hjs7w"
May 26 16:04:37.352619 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:37.352573    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hjs7w"
May 26 16:04:37.360968 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:04:37.360940    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb94c5ba6_2dd8_47f7_8cba_723af642e2e3.slice/crio-af9611601deabd0f68dee5450baa09614546e97aae8ab2b8a67f76aab81c321c WatchSource:0}: Error finding container af9611601deabd0f68dee5450baa09614546e97aae8ab2b8a67f76aab81c321c: Status 404 returned error can't find the container with id af9611601deabd0f68dee5450baa09614546e97aae8ab2b8a67f76aab81c321c
May 26 16:04:37.717587 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:37.717555    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hjs7w" event={"ID":"b94c5ba6-2dd8-47f7-8cba-723af642e2e3","Type":"ContainerStarted","Data":"af9611601deabd0f68dee5450baa09614546e97aae8ab2b8a67f76aab81c321c"}
May 26 16:04:38.723698 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:38.723663    2572 generic.go:358] "Generic (PLEG): container finished" podID="b94c5ba6-2dd8-47f7-8cba-723af642e2e3" containerID="087b1a840e06e53a1a59324dc9c9a0aa93012f6df03b7b9ea78b94db024591f2" exitCode=0
May 26 16:04:38.724108 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:38.723754    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hjs7w" event={"ID":"b94c5ba6-2dd8-47f7-8cba-723af642e2e3","Type":"ContainerDied","Data":"087b1a840e06e53a1a59324dc9c9a0aa93012f6df03b7b9ea78b94db024591f2"}
May 26 16:04:39.728415 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:39.728373    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hjs7w" event={"ID":"b94c5ba6-2dd8-47f7-8cba-723af642e2e3","Type":"ContainerStarted","Data":"4e9a431b490ed9bcf1b122e71f9ebb25131610a80231ada5641b9057f3c34d83"}
May 26 16:04:39.728415 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:39.728416    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hjs7w" event={"ID":"b94c5ba6-2dd8-47f7-8cba-723af642e2e3","Type":"ContainerStarted","Data":"3d772fbf11eb7bac66b3d96d4d8587c375c614d79e128db273cdb2ed6497c253"}
May 26 16:04:39.747458 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:39.747409    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-hjs7w" podStartSLOduration=2.796489772 podStartE2EDuration="3.747393188s" podCreationTimestamp="2026-05-26 16:04:36 +0000 UTC" firstStartedPulling="2026-05-26 16:04:37.362953755 +0000 UTC m=+193.838681086" lastFinishedPulling="2026-05-26 16:04:38.313857159 +0000 UTC m=+194.789584502" observedRunningTime="2026-05-26 16:04:39.746019683 +0000 UTC m=+196.221747032" watchObservedRunningTime="2026-05-26 16:04:39.747393188 +0000 UTC m=+196.223120538"
May 26 16:04:57.845774 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:04:57.845729    2572 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdfbb578c-xqj5q" podUID="cadbc4ee-438a-4e73-b55a-95d246057641" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
May 26 16:05:01.322727 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:05:01.322660    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-image-registry/image-registry-546498794-d7sw5" podUID="6610092d-3e19-47ae-8270-ba37dd62232c" containerName="registry" containerID="cri-o://cf402b6595584bb51e95fc69498798b08652043436a97d9636510d8153e8c41f" gracePeriod=30
May 26 16:05:01.567872 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:05:01.567848    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-546498794-d7sw5"
May 26 16:05:01.623280 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:05:01.623251    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6610092d-3e19-47ae-8270-ba37dd62232c-registry-certificates\") pod \"6610092d-3e19-47ae-8270-ba37dd62232c\" (UID: \"6610092d-3e19-47ae-8270-ba37dd62232c\") "
May 26 16:05:01.623439 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:05:01.623299    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6610092d-3e19-47ae-8270-ba37dd62232c-bound-sa-token\") pod \"6610092d-3e19-47ae-8270-ba37dd62232c\" (UID: \"6610092d-3e19-47ae-8270-ba37dd62232c\") "
May 26 16:05:01.623439 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:05:01.623342    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6610092d-3e19-47ae-8270-ba37dd62232c-image-registry-private-configuration\") pod \"6610092d-3e19-47ae-8270-ba37dd62232c\" (UID: \"6610092d-3e19-47ae-8270-ba37dd62232c\") "
May 26 16:05:01.623553 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:05:01.623448    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6610092d-3e19-47ae-8270-ba37dd62232c-installation-pull-secrets\") pod \"6610092d-3e19-47ae-8270-ba37dd62232c\" (UID: \"6610092d-3e19-47ae-8270-ba37dd62232c\") "
May 26 16:05:01.623553 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:05:01.623501    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6610092d-3e19-47ae-8270-ba37dd62232c-registry-tls\") pod \"6610092d-3e19-47ae-8270-ba37dd62232c\" (UID: \"6610092d-3e19-47ae-8270-ba37dd62232c\") "
May 26 16:05:01.623690 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:05:01.623551    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6610092d-3e19-47ae-8270-ba37dd62232c-ca-trust-extracted\") pod \"6610092d-3e19-47ae-8270-ba37dd62232c\" (UID: \"6610092d-3e19-47ae-8270-ba37dd62232c\") "
May 26 16:05:01.623690 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:05:01.623585    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fs2f9\" (UniqueName: \"kubernetes.io/projected/6610092d-3e19-47ae-8270-ba37dd62232c-kube-api-access-fs2f9\") pod \"6610092d-3e19-47ae-8270-ba37dd62232c\" (UID: \"6610092d-3e19-47ae-8270-ba37dd62232c\") "
May 26 16:05:01.623690 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:05:01.623645    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6610092d-3e19-47ae-8270-ba37dd62232c-trusted-ca\") pod \"6610092d-3e19-47ae-8270-ba37dd62232c\" (UID: \"6610092d-3e19-47ae-8270-ba37dd62232c\") "
May 26 16:05:01.623948 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:05:01.623888    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6610092d-3e19-47ae-8270-ba37dd62232c-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "6610092d-3e19-47ae-8270-ba37dd62232c" (UID: "6610092d-3e19-47ae-8270-ba37dd62232c"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:05:01.624248 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:05:01.624223    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6610092d-3e19-47ae-8270-ba37dd62232c-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "6610092d-3e19-47ae-8270-ba37dd62232c" (UID: "6610092d-3e19-47ae-8270-ba37dd62232c"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:05:01.626066 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:05:01.626034    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6610092d-3e19-47ae-8270-ba37dd62232c-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "6610092d-3e19-47ae-8270-ba37dd62232c" (UID: "6610092d-3e19-47ae-8270-ba37dd62232c"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:05:01.626204 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:05:01.626034    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6610092d-3e19-47ae-8270-ba37dd62232c-kube-api-access-fs2f9" (OuterVolumeSpecName: "kube-api-access-fs2f9") pod "6610092d-3e19-47ae-8270-ba37dd62232c" (UID: "6610092d-3e19-47ae-8270-ba37dd62232c"). InnerVolumeSpecName "kube-api-access-fs2f9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:05:01.626204 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:05:01.626183    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6610092d-3e19-47ae-8270-ba37dd62232c-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "6610092d-3e19-47ae-8270-ba37dd62232c" (UID: "6610092d-3e19-47ae-8270-ba37dd62232c"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:05:01.626280 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:05:01.626211    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6610092d-3e19-47ae-8270-ba37dd62232c-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "6610092d-3e19-47ae-8270-ba37dd62232c" (UID: "6610092d-3e19-47ae-8270-ba37dd62232c"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:05:01.626280 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:05:01.626216    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6610092d-3e19-47ae-8270-ba37dd62232c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "6610092d-3e19-47ae-8270-ba37dd62232c" (UID: "6610092d-3e19-47ae-8270-ba37dd62232c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:05:01.632897 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:05:01.632855    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6610092d-3e19-47ae-8270-ba37dd62232c-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "6610092d-3e19-47ae-8270-ba37dd62232c" (UID: "6610092d-3e19-47ae-8270-ba37dd62232c"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:05:01.724272 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:05:01.724238    2572 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6610092d-3e19-47ae-8270-ba37dd62232c-bound-sa-token\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:05:01.724272 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:05:01.724267    2572 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6610092d-3e19-47ae-8270-ba37dd62232c-image-registry-private-configuration\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:05:01.724272 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:05:01.724277    2572 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6610092d-3e19-47ae-8270-ba37dd62232c-installation-pull-secrets\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:05:01.724461 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:05:01.724288    2572 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6610092d-3e19-47ae-8270-ba37dd62232c-registry-tls\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:05:01.724461 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:05:01.724298    2572 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6610092d-3e19-47ae-8270-ba37dd62232c-ca-trust-extracted\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:05:01.724461 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:05:01.724306    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fs2f9\" (UniqueName: \"kubernetes.io/projected/6610092d-3e19-47ae-8270-ba37dd62232c-kube-api-access-fs2f9\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:05:01.724461 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:05:01.724315    2572 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6610092d-3e19-47ae-8270-ba37dd62232c-trusted-ca\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:05:01.724461 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:05:01.724323    2572 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6610092d-3e19-47ae-8270-ba37dd62232c-registry-certificates\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:05:01.791912 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:05:01.791878    2572 generic.go:358] "Generic (PLEG): container finished" podID="6610092d-3e19-47ae-8270-ba37dd62232c" containerID="cf402b6595584bb51e95fc69498798b08652043436a97d9636510d8153e8c41f" exitCode=0
May 26 16:05:01.792067 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:05:01.791938    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-546498794-d7sw5"
May 26 16:05:01.792067 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:05:01.791956    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-546498794-d7sw5" event={"ID":"6610092d-3e19-47ae-8270-ba37dd62232c","Type":"ContainerDied","Data":"cf402b6595584bb51e95fc69498798b08652043436a97d9636510d8153e8c41f"}
May 26 16:05:01.792067 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:05:01.791991    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-546498794-d7sw5" event={"ID":"6610092d-3e19-47ae-8270-ba37dd62232c","Type":"ContainerDied","Data":"1e24992606161ea101260bfc1672d854929fd71fa3e1f5baea4bcc7bd732aeee"}
May 26 16:05:01.792067 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:05:01.792015    2572 scope.go:117] "RemoveContainer" containerID="cf402b6595584bb51e95fc69498798b08652043436a97d9636510d8153e8c41f"
May 26 16:05:01.800882 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:05:01.800859    2572 scope.go:117] "RemoveContainer" containerID="cf402b6595584bb51e95fc69498798b08652043436a97d9636510d8153e8c41f"
May 26 16:05:01.801196 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:05:01.801171    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf402b6595584bb51e95fc69498798b08652043436a97d9636510d8153e8c41f\": container with ID starting with cf402b6595584bb51e95fc69498798b08652043436a97d9636510d8153e8c41f not found: ID does not exist" containerID="cf402b6595584bb51e95fc69498798b08652043436a97d9636510d8153e8c41f"
May 26 16:05:01.801259 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:05:01.801206    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf402b6595584bb51e95fc69498798b08652043436a97d9636510d8153e8c41f"} err="failed to get container status \"cf402b6595584bb51e95fc69498798b08652043436a97d9636510d8153e8c41f\": rpc error: code = NotFound desc = could not find container \"cf402b6595584bb51e95fc69498798b08652043436a97d9636510d8153e8c41f\": container with ID starting with cf402b6595584bb51e95fc69498798b08652043436a97d9636510d8153e8c41f not found: ID does not exist"
May 26 16:05:01.811427 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:05:01.811394    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-546498794-d7sw5"]
May 26 16:05:01.816277 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:05:01.816247    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-546498794-d7sw5"]
May 26 16:05:02.078857 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:05:02.078779    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6610092d-3e19-47ae-8270-ba37dd62232c" path="/var/lib/kubelet/pods/6610092d-3e19-47ae-8270-ba37dd62232c/volumes"
May 26 16:05:07.845909 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:05:07.845870    2572 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdfbb578c-xqj5q" podUID="cadbc4ee-438a-4e73-b55a-95d246057641" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
May 26 16:05:17.845582 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:05:17.845544    2572 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdfbb578c-xqj5q" podUID="cadbc4ee-438a-4e73-b55a-95d246057641" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
May 26 16:05:17.845971 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:05:17.845645    2572 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdfbb578c-xqj5q"
May 26 16:05:17.846102 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:05:17.846085    2572 kuberuntime_manager.go:1107] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"a0e51802b0947f0fbae1d4cf3f41ab841bc6adf1af9c60bd081be13dcbc3588c"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdfbb578c-xqj5q" containerMessage="Container service-proxy failed liveness probe, will be restarted"
May 26 16:05:17.846143 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:05:17.846120    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdfbb578c-xqj5q" podUID="cadbc4ee-438a-4e73-b55a-95d246057641" containerName="service-proxy" containerID="cri-o://a0e51802b0947f0fbae1d4cf3f41ab841bc6adf1af9c60bd081be13dcbc3588c" gracePeriod=30
May 26 16:05:18.842210 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:05:18.842174    2572 generic.go:358] "Generic (PLEG): container finished" podID="cadbc4ee-438a-4e73-b55a-95d246057641" containerID="a0e51802b0947f0fbae1d4cf3f41ab841bc6adf1af9c60bd081be13dcbc3588c" exitCode=2
May 26 16:05:18.842377 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:05:18.842225    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdfbb578c-xqj5q" event={"ID":"cadbc4ee-438a-4e73-b55a-95d246057641","Type":"ContainerDied","Data":"a0e51802b0947f0fbae1d4cf3f41ab841bc6adf1af9c60bd081be13dcbc3588c"}
May 26 16:05:18.842377 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:05:18.842250    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-cdfbb578c-xqj5q" event={"ID":"cadbc4ee-438a-4e73-b55a-95d246057641","Type":"ContainerStarted","Data":"c1bfced6896d53ef84af7c594d24401eac7bd2a9e08bcd751081d5ba47901958"}
May 26 16:05:35.972393 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:05:35.972334    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8828138d-e7ed-4d5c-ba76-26429679c7b1-metrics-certs\") pod \"network-metrics-daemon-qpzqg\" (UID: \"8828138d-e7ed-4d5c-ba76-26429679c7b1\") " pod="openshift-multus/network-metrics-daemon-qpzqg"
May 26 16:05:35.974719 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:05:35.974698    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8828138d-e7ed-4d5c-ba76-26429679c7b1-metrics-certs\") pod \"network-metrics-daemon-qpzqg\" (UID: \"8828138d-e7ed-4d5c-ba76-26429679c7b1\") " pod="openshift-multus/network-metrics-daemon-qpzqg"
May 26 16:05:36.178733 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:05:36.178699    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fjs6m\""
May 26 16:05:36.186857 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:05:36.186834    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qpzqg"
May 26 16:05:36.310139 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:05:36.310107    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qpzqg"]
May 26 16:05:36.313089 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:05:36.313061    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8828138d_e7ed_4d5c_ba76_26429679c7b1.slice/crio-4322df4dbe838e50721b6041c48ee5065b89c39ee54dd7e7500e95798198d244 WatchSource:0}: Error finding container 4322df4dbe838e50721b6041c48ee5065b89c39ee54dd7e7500e95798198d244: Status 404 returned error can't find the container with id 4322df4dbe838e50721b6041c48ee5065b89c39ee54dd7e7500e95798198d244
May 26 16:05:36.891169 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:05:36.891132    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qpzqg" event={"ID":"8828138d-e7ed-4d5c-ba76-26429679c7b1","Type":"ContainerStarted","Data":"4322df4dbe838e50721b6041c48ee5065b89c39ee54dd7e7500e95798198d244"}
May 26 16:05:37.894996 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:05:37.894959    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qpzqg" event={"ID":"8828138d-e7ed-4d5c-ba76-26429679c7b1","Type":"ContainerStarted","Data":"86a7c035979fec5ac36d229d240cbfa43ce51c89e30ba43c66dbb3c9c2ff0765"}
May 26 16:05:37.894996 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:05:37.894996    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qpzqg" event={"ID":"8828138d-e7ed-4d5c-ba76-26429679c7b1","Type":"ContainerStarted","Data":"711bd611dbacab08819df09afc3139359f85f294ff2b44e565f466cc8821f850"}
May 26 16:05:37.910546 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:05:37.910492    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qpzqg" podStartSLOduration=253.022196502 podStartE2EDuration="4m13.910477191s" podCreationTimestamp="2026-05-26 16:01:24 +0000 UTC" firstStartedPulling="2026-05-26 16:05:36.314955043 +0000 UTC m=+252.790682375" lastFinishedPulling="2026-05-26 16:05:37.203235721 +0000 UTC m=+253.678963064" observedRunningTime="2026-05-26 16:05:37.909488155 +0000 UTC m=+254.385215506" watchObservedRunningTime="2026-05-26 16:05:37.910477191 +0000 UTC m=+254.386204541"
May 26 16:06:23.987528 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:06:23.987496    2572 kubelet.go:1628] "Image garbage collection succeeded"
May 26 16:09:33.131752 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:09:33.131714    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-6f7545b5b9-dr7pm"]
May 26 16:09:33.132209 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:09:33.131973    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6610092d-3e19-47ae-8270-ba37dd62232c" containerName="registry"
May 26 16:09:33.132209 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:09:33.131985    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="6610092d-3e19-47ae-8270-ba37dd62232c" containerName="registry"
May 26 16:09:33.132209 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:09:33.132024    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="6610092d-3e19-47ae-8270-ba37dd62232c" containerName="registry"
May 26 16:09:33.134815 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:09:33.134798    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6f7545b5b9-dr7pm"
May 26 16:09:33.137198 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:09:33.137176    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
May 26 16:09:33.138203 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:09:33.138182    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
May 26 16:09:33.138341 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:09:33.138230    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
May 26 16:09:33.138341 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:09:33.138276    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-7plsf\""
May 26 16:09:33.145348 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:09:33.145323    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6f7545b5b9-dr7pm"]
May 26 16:09:33.209797 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:09:33.209750    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cg52\" (UniqueName: \"kubernetes.io/projected/dde8c566-c09e-4913-b32a-d96ba2c908be-kube-api-access-6cg52\") pod \"kserve-controller-manager-6f7545b5b9-dr7pm\" (UID: \"dde8c566-c09e-4913-b32a-d96ba2c908be\") " pod="kserve/kserve-controller-manager-6f7545b5b9-dr7pm"
May 26 16:09:33.209967 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:09:33.209826    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dde8c566-c09e-4913-b32a-d96ba2c908be-cert\") pod \"kserve-controller-manager-6f7545b5b9-dr7pm\" (UID: \"dde8c566-c09e-4913-b32a-d96ba2c908be\") " pod="kserve/kserve-controller-manager-6f7545b5b9-dr7pm"
May 26 16:09:33.310181 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:09:33.310143    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6cg52\" (UniqueName: \"kubernetes.io/projected/dde8c566-c09e-4913-b32a-d96ba2c908be-kube-api-access-6cg52\") pod \"kserve-controller-manager-6f7545b5b9-dr7pm\" (UID: \"dde8c566-c09e-4913-b32a-d96ba2c908be\") " pod="kserve/kserve-controller-manager-6f7545b5b9-dr7pm"
May 26 16:09:33.310348 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:09:33.310202    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dde8c566-c09e-4913-b32a-d96ba2c908be-cert\") pod \"kserve-controller-manager-6f7545b5b9-dr7pm\" (UID: \"dde8c566-c09e-4913-b32a-d96ba2c908be\") " pod="kserve/kserve-controller-manager-6f7545b5b9-dr7pm"
May 26 16:09:33.312668 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:09:33.312646    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dde8c566-c09e-4913-b32a-d96ba2c908be-cert\") pod \"kserve-controller-manager-6f7545b5b9-dr7pm\" (UID: \"dde8c566-c09e-4913-b32a-d96ba2c908be\") " pod="kserve/kserve-controller-manager-6f7545b5b9-dr7pm"
May 26 16:09:33.318057 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:09:33.318029    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cg52\" (UniqueName: \"kubernetes.io/projected/dde8c566-c09e-4913-b32a-d96ba2c908be-kube-api-access-6cg52\") pod \"kserve-controller-manager-6f7545b5b9-dr7pm\" (UID: \"dde8c566-c09e-4913-b32a-d96ba2c908be\") " pod="kserve/kserve-controller-manager-6f7545b5b9-dr7pm"
May 26 16:09:33.445488 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:09:33.445394    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6f7545b5b9-dr7pm"
May 26 16:09:33.565405 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:09:33.565289    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6f7545b5b9-dr7pm"]
May 26 16:09:33.567999 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:09:33.567972    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddde8c566_c09e_4913_b32a_d96ba2c908be.slice/crio-a5208c49db51d0d864a02afbdf24dbf53503043c12be0c17a2a27748af3f8507 WatchSource:0}: Error finding container a5208c49db51d0d864a02afbdf24dbf53503043c12be0c17a2a27748af3f8507: Status 404 returned error can't find the container with id a5208c49db51d0d864a02afbdf24dbf53503043c12be0c17a2a27748af3f8507
May 26 16:09:33.569221 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:09:33.569206    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 16:09:34.478002 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:09:34.477946    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6f7545b5b9-dr7pm" event={"ID":"dde8c566-c09e-4913-b32a-d96ba2c908be","Type":"ContainerStarted","Data":"a5208c49db51d0d864a02afbdf24dbf53503043c12be0c17a2a27748af3f8507"}
May 26 16:09:36.484955 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:09:36.484914    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6f7545b5b9-dr7pm" event={"ID":"dde8c566-c09e-4913-b32a-d96ba2c908be","Type":"ContainerStarted","Data":"605aad5c340081c13aea3497ccc9fc7a57edb2d6e452d5f973430fbcd112c2e4"}
May 26 16:09:36.485363 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:09:36.485141    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-6f7545b5b9-dr7pm"
May 26 16:09:36.499320 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:09:36.499276    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/kserve-controller-manager-6f7545b5b9-dr7pm" podStartSLOduration=0.797021536 podStartE2EDuration="3.499263031s" podCreationTimestamp="2026-05-26 16:09:33 +0000 UTC" firstStartedPulling="2026-05-26 16:09:33.569325739 +0000 UTC m=+490.045053067" lastFinishedPulling="2026-05-26 16:09:36.271567235 +0000 UTC m=+492.747294562" observedRunningTime="2026-05-26 16:09:36.498143771 +0000 UTC m=+492.973871122" watchObservedRunningTime="2026-05-26 16:09:36.499263031 +0000 UTC m=+492.974990381"
May 26 16:10:07.493557 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:07.493528    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-6f7545b5b9-dr7pm"
May 26 16:10:10.336406 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:10.336371    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-6f7545b5b9-dr7pm"]
May 26 16:10:10.336941 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:10.336590    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve/kserve-controller-manager-6f7545b5b9-dr7pm" podUID="dde8c566-c09e-4913-b32a-d96ba2c908be" containerName="manager" containerID="cri-o://605aad5c340081c13aea3497ccc9fc7a57edb2d6e452d5f973430fbcd112c2e4" gracePeriod=10
May 26 16:10:10.360418 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:10.360389    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-6f7545b5b9-98fbz"]
May 26 16:10:10.363370 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:10.363352    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6f7545b5b9-98fbz"
May 26 16:10:10.371542 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:10.371519    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6f7545b5b9-98fbz"]
May 26 16:10:10.466283 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:10.466255    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d17d333-ec9a-4a4c-8fc0-08eddf7ab0f5-cert\") pod \"kserve-controller-manager-6f7545b5b9-98fbz\" (UID: \"3d17d333-ec9a-4a4c-8fc0-08eddf7ab0f5\") " pod="kserve/kserve-controller-manager-6f7545b5b9-98fbz"
May 26 16:10:10.466431 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:10.466317    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l622\" (UniqueName: \"kubernetes.io/projected/3d17d333-ec9a-4a4c-8fc0-08eddf7ab0f5-kube-api-access-9l622\") pod \"kserve-controller-manager-6f7545b5b9-98fbz\" (UID: \"3d17d333-ec9a-4a4c-8fc0-08eddf7ab0f5\") " pod="kserve/kserve-controller-manager-6f7545b5b9-98fbz"
May 26 16:10:10.566935 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:10.566909    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9l622\" (UniqueName: \"kubernetes.io/projected/3d17d333-ec9a-4a4c-8fc0-08eddf7ab0f5-kube-api-access-9l622\") pod \"kserve-controller-manager-6f7545b5b9-98fbz\" (UID: \"3d17d333-ec9a-4a4c-8fc0-08eddf7ab0f5\") " pod="kserve/kserve-controller-manager-6f7545b5b9-98fbz"
May 26 16:10:10.567045 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:10.566954    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d17d333-ec9a-4a4c-8fc0-08eddf7ab0f5-cert\") pod \"kserve-controller-manager-6f7545b5b9-98fbz\" (UID: \"3d17d333-ec9a-4a4c-8fc0-08eddf7ab0f5\") " pod="kserve/kserve-controller-manager-6f7545b5b9-98fbz"
May 26 16:10:10.569442 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:10.569414    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d17d333-ec9a-4a4c-8fc0-08eddf7ab0f5-cert\") pod \"kserve-controller-manager-6f7545b5b9-98fbz\" (UID: \"3d17d333-ec9a-4a4c-8fc0-08eddf7ab0f5\") " pod="kserve/kserve-controller-manager-6f7545b5b9-98fbz"
May 26 16:10:10.569676 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:10.569660    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6f7545b5b9-dr7pm"
May 26 16:10:10.575029 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:10.574990    2572 generic.go:358] "Generic (PLEG): container finished" podID="dde8c566-c09e-4913-b32a-d96ba2c908be" containerID="605aad5c340081c13aea3497ccc9fc7a57edb2d6e452d5f973430fbcd112c2e4" exitCode=0
May 26 16:10:10.575153 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:10.575051    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6f7545b5b9-dr7pm"
May 26 16:10:10.575153 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:10.575057    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6f7545b5b9-dr7pm" event={"ID":"dde8c566-c09e-4913-b32a-d96ba2c908be","Type":"ContainerDied","Data":"605aad5c340081c13aea3497ccc9fc7a57edb2d6e452d5f973430fbcd112c2e4"}
May 26 16:10:10.575153 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:10.575091    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6f7545b5b9-dr7pm" event={"ID":"dde8c566-c09e-4913-b32a-d96ba2c908be","Type":"ContainerDied","Data":"a5208c49db51d0d864a02afbdf24dbf53503043c12be0c17a2a27748af3f8507"}
May 26 16:10:10.575153 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:10.575112    2572 scope.go:117] "RemoveContainer" containerID="605aad5c340081c13aea3497ccc9fc7a57edb2d6e452d5f973430fbcd112c2e4"
May 26 16:10:10.576152 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:10.576135    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l622\" (UniqueName: \"kubernetes.io/projected/3d17d333-ec9a-4a4c-8fc0-08eddf7ab0f5-kube-api-access-9l622\") pod \"kserve-controller-manager-6f7545b5b9-98fbz\" (UID: \"3d17d333-ec9a-4a4c-8fc0-08eddf7ab0f5\") " pod="kserve/kserve-controller-manager-6f7545b5b9-98fbz"
May 26 16:10:10.584194 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:10.584170    2572 scope.go:117] "RemoveContainer" containerID="605aad5c340081c13aea3497ccc9fc7a57edb2d6e452d5f973430fbcd112c2e4"
May 26 16:10:10.584489 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:10:10.584440    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"605aad5c340081c13aea3497ccc9fc7a57edb2d6e452d5f973430fbcd112c2e4\": container with ID starting with 605aad5c340081c13aea3497ccc9fc7a57edb2d6e452d5f973430fbcd112c2e4 not found: ID does not exist" containerID="605aad5c340081c13aea3497ccc9fc7a57edb2d6e452d5f973430fbcd112c2e4"
May 26 16:10:10.584582 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:10.584500    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"605aad5c340081c13aea3497ccc9fc7a57edb2d6e452d5f973430fbcd112c2e4"} err="failed to get container status \"605aad5c340081c13aea3497ccc9fc7a57edb2d6e452d5f973430fbcd112c2e4\": rpc error: code = NotFound desc = could not find container \"605aad5c340081c13aea3497ccc9fc7a57edb2d6e452d5f973430fbcd112c2e4\": container with ID starting with 605aad5c340081c13aea3497ccc9fc7a57edb2d6e452d5f973430fbcd112c2e4 not found: ID does not exist"
May 26 16:10:10.667686 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:10.667649    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6cg52\" (UniqueName: \"kubernetes.io/projected/dde8c566-c09e-4913-b32a-d96ba2c908be-kube-api-access-6cg52\") pod \"dde8c566-c09e-4913-b32a-d96ba2c908be\" (UID: \"dde8c566-c09e-4913-b32a-d96ba2c908be\") "
May 26 16:10:10.667686 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:10.667699    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dde8c566-c09e-4913-b32a-d96ba2c908be-cert\") pod \"dde8c566-c09e-4913-b32a-d96ba2c908be\" (UID: \"dde8c566-c09e-4913-b32a-d96ba2c908be\") "
May 26 16:10:10.669985 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:10.669959    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dde8c566-c09e-4913-b32a-d96ba2c908be-cert" (OuterVolumeSpecName: "cert") pod "dde8c566-c09e-4913-b32a-d96ba2c908be" (UID: "dde8c566-c09e-4913-b32a-d96ba2c908be"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:10:10.670065 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:10.669978    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dde8c566-c09e-4913-b32a-d96ba2c908be-kube-api-access-6cg52" (OuterVolumeSpecName: "kube-api-access-6cg52") pod "dde8c566-c09e-4913-b32a-d96ba2c908be" (UID: "dde8c566-c09e-4913-b32a-d96ba2c908be"). InnerVolumeSpecName "kube-api-access-6cg52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:10:10.709176 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:10.709130    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6f7545b5b9-98fbz"
May 26 16:10:10.768425 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:10.768393    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cg52\" (UniqueName: \"kubernetes.io/projected/dde8c566-c09e-4913-b32a-d96ba2c908be-kube-api-access-6cg52\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:10:10.768425 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:10.768422    2572 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dde8c566-c09e-4913-b32a-d96ba2c908be-cert\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:10:10.834441 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:10.834409    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6f7545b5b9-98fbz"]
May 26 16:10:10.836900 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:10:10.836871    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d17d333_ec9a_4a4c_8fc0_08eddf7ab0f5.slice/crio-01d5c97965e6eaf6488d71aec91105b3c7d95438e3ca4df49bce2aabb5a295fc WatchSource:0}: Error finding container 01d5c97965e6eaf6488d71aec91105b3c7d95438e3ca4df49bce2aabb5a295fc: Status 404 returned error can't find the container with id 01d5c97965e6eaf6488d71aec91105b3c7d95438e3ca4df49bce2aabb5a295fc
May 26 16:10:10.895341 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:10.895312    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-6f7545b5b9-dr7pm"]
May 26 16:10:10.898847 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:10.898823    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-6f7545b5b9-dr7pm"]
May 26 16:10:11.579853 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:11.579755    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6f7545b5b9-98fbz" event={"ID":"3d17d333-ec9a-4a4c-8fc0-08eddf7ab0f5","Type":"ContainerStarted","Data":"a3b6e0ef975325b8c42d192378f65db7dba947fccf9cef2b1db8e8be20d35ad6"}
May 26 16:10:11.579853 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:11.579801    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6f7545b5b9-98fbz" event={"ID":"3d17d333-ec9a-4a4c-8fc0-08eddf7ab0f5","Type":"ContainerStarted","Data":"01d5c97965e6eaf6488d71aec91105b3c7d95438e3ca4df49bce2aabb5a295fc"}
May 26 16:10:11.579853 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:11.579830    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-6f7545b5b9-98fbz"
May 26 16:10:11.595036 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:11.594982    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/kserve-controller-manager-6f7545b5b9-98fbz" podStartSLOduration=1.251089645 podStartE2EDuration="1.594967992s" podCreationTimestamp="2026-05-26 16:10:10 +0000 UTC" firstStartedPulling="2026-05-26 16:10:10.838158686 +0000 UTC m=+527.313886014" lastFinishedPulling="2026-05-26 16:10:11.182037033 +0000 UTC m=+527.657764361" observedRunningTime="2026-05-26 16:10:11.59436205 +0000 UTC m=+528.070089391" watchObservedRunningTime="2026-05-26 16:10:11.594967992 +0000 UTC m=+528.070695342"
May 26 16:10:12.080518 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:12.080479    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dde8c566-c09e-4913-b32a-d96ba2c908be" path="/var/lib/kubelet/pods/dde8c566-c09e-4913-b32a-d96ba2c908be/volumes"
May 26 16:10:42.587774 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:42.587743    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-6f7545b5b9-98fbz"
May 26 16:10:43.524173 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:43.524144    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-696fc77849-b7h68"]
May 26 16:10:43.524407 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:43.524395    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dde8c566-c09e-4913-b32a-d96ba2c908be" containerName="manager"
May 26 16:10:43.524450 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:43.524408    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="dde8c566-c09e-4913-b32a-d96ba2c908be" containerName="manager"
May 26 16:10:43.524483 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:43.524459    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="dde8c566-c09e-4913-b32a-d96ba2c908be" containerName="manager"
May 26 16:10:43.528350 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:43.528332    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-696fc77849-b7h68"
May 26 16:10:43.530596 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:43.530574    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-zxrk4\""
May 26 16:10:43.530738 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:43.530713    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
May 26 16:10:43.534755 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:43.534730    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-696fc77849-b7h68"]
May 26 16:10:43.598318 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:43.598290    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkqqv\" (UniqueName: \"kubernetes.io/projected/b08cd028-080d-4719-842c-93bc6dfa6cce-kube-api-access-qkqqv\") pod \"odh-model-controller-696fc77849-b7h68\" (UID: \"b08cd028-080d-4719-842c-93bc6dfa6cce\") " pod="kserve/odh-model-controller-696fc77849-b7h68"
May 26 16:10:43.598318 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:43.598323    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b08cd028-080d-4719-842c-93bc6dfa6cce-cert\") pod \"odh-model-controller-696fc77849-b7h68\" (UID: \"b08cd028-080d-4719-842c-93bc6dfa6cce\") " pod="kserve/odh-model-controller-696fc77849-b7h68"
May 26 16:10:43.699503 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:43.699466    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qkqqv\" (UniqueName: \"kubernetes.io/projected/b08cd028-080d-4719-842c-93bc6dfa6cce-kube-api-access-qkqqv\") pod \"odh-model-controller-696fc77849-b7h68\" (UID: \"b08cd028-080d-4719-842c-93bc6dfa6cce\") " pod="kserve/odh-model-controller-696fc77849-b7h68"
May 26 16:10:43.699503 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:43.699503    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b08cd028-080d-4719-842c-93bc6dfa6cce-cert\") pod \"odh-model-controller-696fc77849-b7h68\" (UID: \"b08cd028-080d-4719-842c-93bc6dfa6cce\") " pod="kserve/odh-model-controller-696fc77849-b7h68"
May 26 16:10:43.701986 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:43.701954    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b08cd028-080d-4719-842c-93bc6dfa6cce-cert\") pod \"odh-model-controller-696fc77849-b7h68\" (UID: \"b08cd028-080d-4719-842c-93bc6dfa6cce\") " pod="kserve/odh-model-controller-696fc77849-b7h68"
May 26 16:10:43.709391 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:43.709363    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkqqv\" (UniqueName: \"kubernetes.io/projected/b08cd028-080d-4719-842c-93bc6dfa6cce-kube-api-access-qkqqv\") pod \"odh-model-controller-696fc77849-b7h68\" (UID: \"b08cd028-080d-4719-842c-93bc6dfa6cce\") " pod="kserve/odh-model-controller-696fc77849-b7h68"
May 26 16:10:43.839482 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:43.839388    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-696fc77849-b7h68"
May 26 16:10:43.963543 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:43.963511    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-696fc77849-b7h68"]
May 26 16:10:43.966316 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:10:43.966288    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb08cd028_080d_4719_842c_93bc6dfa6cce.slice/crio-45052c947bea98ecbd4cf33c1196c780237110de34e49931840a9307fb3a2748 WatchSource:0}: Error finding container 45052c947bea98ecbd4cf33c1196c780237110de34e49931840a9307fb3a2748: Status 404 returned error can't find the container with id 45052c947bea98ecbd4cf33c1196c780237110de34e49931840a9307fb3a2748
May 26 16:10:44.665828 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:44.665792    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-696fc77849-b7h68" event={"ID":"b08cd028-080d-4719-842c-93bc6dfa6cce","Type":"ContainerStarted","Data":"45052c947bea98ecbd4cf33c1196c780237110de34e49931840a9307fb3a2748"}
May 26 16:10:47.674982 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:47.674945    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-696fc77849-b7h68" event={"ID":"b08cd028-080d-4719-842c-93bc6dfa6cce","Type":"ContainerStarted","Data":"c52a75c37ec12e9e3c01e1f7898c944ae8bd8aa4a21e5c5a896cd139523b6804"}
May 26 16:10:47.675372 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:47.675062    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-696fc77849-b7h68"
May 26 16:10:47.690882 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:47.690839    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/odh-model-controller-696fc77849-b7h68" podStartSLOduration=1.486380844 podStartE2EDuration="4.690827796s" podCreationTimestamp="2026-05-26 16:10:43 +0000 UTC" firstStartedPulling="2026-05-26 16:10:43.967528049 +0000 UTC m=+560.443255378" lastFinishedPulling="2026-05-26 16:10:47.171974988 +0000 UTC m=+563.647702330" observedRunningTime="2026-05-26 16:10:47.690281006 +0000 UTC m=+564.166008369" watchObservedRunningTime="2026-05-26 16:10:47.690827796 +0000 UTC m=+564.166555146"
May 26 16:10:58.681082 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:58.681053    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-696fc77849-b7h68"
May 26 16:10:59.470534 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:59.470498    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/s3-init-lqlhh"]
May 26 16:10:59.473916 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:59.473891    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-lqlhh"
May 26 16:10:59.476016 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:59.475974    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
May 26 16:10:59.476136 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:59.476123    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-gzm8h\""
May 26 16:10:59.480295 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:59.480270    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-lqlhh"]
May 26 16:10:59.619374 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:59.619341    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tspx\" (UniqueName: \"kubernetes.io/projected/a40c3e2d-ef26-45c2-b3e8-d6bd68cc3813-kube-api-access-5tspx\") pod \"s3-init-lqlhh\" (UID: \"a40c3e2d-ef26-45c2-b3e8-d6bd68cc3813\") " pod="kserve/s3-init-lqlhh"
May 26 16:10:59.720023 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:59.719986    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5tspx\" (UniqueName: \"kubernetes.io/projected/a40c3e2d-ef26-45c2-b3e8-d6bd68cc3813-kube-api-access-5tspx\") pod \"s3-init-lqlhh\" (UID: \"a40c3e2d-ef26-45c2-b3e8-d6bd68cc3813\") " pod="kserve/s3-init-lqlhh"
May 26 16:10:59.730331 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:59.730261    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tspx\" (UniqueName: \"kubernetes.io/projected/a40c3e2d-ef26-45c2-b3e8-d6bd68cc3813-kube-api-access-5tspx\") pod \"s3-init-lqlhh\" (UID: \"a40c3e2d-ef26-45c2-b3e8-d6bd68cc3813\") " pod="kserve/s3-init-lqlhh"
May 26 16:10:59.783755 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:59.783716    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-lqlhh"
May 26 16:10:59.911616 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:10:59.911580    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-lqlhh"]
May 26 16:10:59.914144 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:10:59.914116    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda40c3e2d_ef26_45c2_b3e8_d6bd68cc3813.slice/crio-07100ca4d2971e86202807cc54cc36c1f1902ad2508796dfdf726ac46ae1c4e7 WatchSource:0}: Error finding container 07100ca4d2971e86202807cc54cc36c1f1902ad2508796dfdf726ac46ae1c4e7: Status 404 returned error can't find the container with id 07100ca4d2971e86202807cc54cc36c1f1902ad2508796dfdf726ac46ae1c4e7
May 26 16:11:00.715409 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:00.715366    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-lqlhh" event={"ID":"a40c3e2d-ef26-45c2-b3e8-d6bd68cc3813","Type":"ContainerStarted","Data":"07100ca4d2971e86202807cc54cc36c1f1902ad2508796dfdf726ac46ae1c4e7"}
May 26 16:11:04.729729 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:04.729691    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-lqlhh" event={"ID":"a40c3e2d-ef26-45c2-b3e8-d6bd68cc3813","Type":"ContainerStarted","Data":"9c1238f7be65c2f6d789a697f5537fa5c45216c47a7bc32fb2174e25aa10517c"}
May 26 16:11:04.746411 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:04.746314    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/s3-init-lqlhh" podStartSLOduration=1.168944267 podStartE2EDuration="5.746298715s" podCreationTimestamp="2026-05-26 16:10:59 +0000 UTC" firstStartedPulling="2026-05-26 16:10:59.915775079 +0000 UTC m=+576.391502407" lastFinishedPulling="2026-05-26 16:11:04.493129527 +0000 UTC m=+580.968856855" observedRunningTime="2026-05-26 16:11:04.744864112 +0000 UTC m=+581.220591456" watchObservedRunningTime="2026-05-26 16:11:04.746298715 +0000 UTC m=+581.222026064"
May 26 16:11:07.739001 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:07.738969    2572 generic.go:358] "Generic (PLEG): container finished" podID="a40c3e2d-ef26-45c2-b3e8-d6bd68cc3813" containerID="9c1238f7be65c2f6d789a697f5537fa5c45216c47a7bc32fb2174e25aa10517c" exitCode=0
May 26 16:11:07.739364 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:07.739025    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-lqlhh" event={"ID":"a40c3e2d-ef26-45c2-b3e8-d6bd68cc3813","Type":"ContainerDied","Data":"9c1238f7be65c2f6d789a697f5537fa5c45216c47a7bc32fb2174e25aa10517c"}
May 26 16:11:08.863356 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:08.863334    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-lqlhh"
May 26 16:11:08.994630 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:08.994529    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5tspx\" (UniqueName: \"kubernetes.io/projected/a40c3e2d-ef26-45c2-b3e8-d6bd68cc3813-kube-api-access-5tspx\") pod \"a40c3e2d-ef26-45c2-b3e8-d6bd68cc3813\" (UID: \"a40c3e2d-ef26-45c2-b3e8-d6bd68cc3813\") "
May 26 16:11:08.996551 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:08.996519    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a40c3e2d-ef26-45c2-b3e8-d6bd68cc3813-kube-api-access-5tspx" (OuterVolumeSpecName: "kube-api-access-5tspx") pod "a40c3e2d-ef26-45c2-b3e8-d6bd68cc3813" (UID: "a40c3e2d-ef26-45c2-b3e8-d6bd68cc3813"). InnerVolumeSpecName "kube-api-access-5tspx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:11:09.095247 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:09.095210    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tspx\" (UniqueName: \"kubernetes.io/projected/a40c3e2d-ef26-45c2-b3e8-d6bd68cc3813-kube-api-access-5tspx\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:11:09.746048 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:09.746016    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-lqlhh"
May 26 16:11:09.746216 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:09.746022    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-lqlhh" event={"ID":"a40c3e2d-ef26-45c2-b3e8-d6bd68cc3813","Type":"ContainerDied","Data":"07100ca4d2971e86202807cc54cc36c1f1902ad2508796dfdf726ac46ae1c4e7"}
May 26 16:11:09.746216 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:09.746123    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07100ca4d2971e86202807cc54cc36c1f1902ad2508796dfdf726ac46ae1c4e7"
May 26 16:11:10.462149 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:10.462113    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-ddd4dbfd-kvdvc"]
May 26 16:11:10.462540 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:10.462362    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a40c3e2d-ef26-45c2-b3e8-d6bd68cc3813" containerName="s3-init"
May 26 16:11:10.462540 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:10.462373    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a40c3e2d-ef26-45c2-b3e8-d6bd68cc3813" containerName="s3-init"
May 26 16:11:10.462540 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:10.462413    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="a40c3e2d-ef26-45c2-b3e8-d6bd68cc3813" containerName="s3-init"
May 26 16:11:10.464597 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:10.464582    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-ddd4dbfd-kvdvc"
May 26 16:11:10.467116 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:10.467088    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-gzm8h\""
May 26 16:11:10.467225 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:10.467094    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
May 26 16:11:10.472408 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:10.472384    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-ddd4dbfd-kvdvc"]
May 26 16:11:10.607056 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:10.607016    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e1bce4d4-6cc9-451b-8888-2f7e69a0b888-data\") pod \"seaweedfs-tls-custom-ddd4dbfd-kvdvc\" (UID: \"e1bce4d4-6cc9-451b-8888-2f7e69a0b888\") " pod="kserve/seaweedfs-tls-custom-ddd4dbfd-kvdvc"
May 26 16:11:10.607228 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:10.607081    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5np7b\" (UniqueName: \"kubernetes.io/projected/e1bce4d4-6cc9-451b-8888-2f7e69a0b888-kube-api-access-5np7b\") pod \"seaweedfs-tls-custom-ddd4dbfd-kvdvc\" (UID: \"e1bce4d4-6cc9-451b-8888-2f7e69a0b888\") " pod="kserve/seaweedfs-tls-custom-ddd4dbfd-kvdvc"
May 26 16:11:10.708016 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:10.707984    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e1bce4d4-6cc9-451b-8888-2f7e69a0b888-data\") pod \"seaweedfs-tls-custom-ddd4dbfd-kvdvc\" (UID: \"e1bce4d4-6cc9-451b-8888-2f7e69a0b888\") " pod="kserve/seaweedfs-tls-custom-ddd4dbfd-kvdvc"
May 26 16:11:10.708187 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:10.708038    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5np7b\" (UniqueName: \"kubernetes.io/projected/e1bce4d4-6cc9-451b-8888-2f7e69a0b888-kube-api-access-5np7b\") pod \"seaweedfs-tls-custom-ddd4dbfd-kvdvc\" (UID: \"e1bce4d4-6cc9-451b-8888-2f7e69a0b888\") " pod="kserve/seaweedfs-tls-custom-ddd4dbfd-kvdvc"
May 26 16:11:10.708353 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:10.708333    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e1bce4d4-6cc9-451b-8888-2f7e69a0b888-data\") pod \"seaweedfs-tls-custom-ddd4dbfd-kvdvc\" (UID: \"e1bce4d4-6cc9-451b-8888-2f7e69a0b888\") " pod="kserve/seaweedfs-tls-custom-ddd4dbfd-kvdvc"
May 26 16:11:10.716882 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:10.716830    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5np7b\" (UniqueName: \"kubernetes.io/projected/e1bce4d4-6cc9-451b-8888-2f7e69a0b888-kube-api-access-5np7b\") pod \"seaweedfs-tls-custom-ddd4dbfd-kvdvc\" (UID: \"e1bce4d4-6cc9-451b-8888-2f7e69a0b888\") " pod="kserve/seaweedfs-tls-custom-ddd4dbfd-kvdvc"
May 26 16:11:10.774290 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:10.774255    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-ddd4dbfd-kvdvc"
May 26 16:11:10.896345 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:10.896313    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-ddd4dbfd-kvdvc"]
May 26 16:11:10.899109 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:11:10.899082    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1bce4d4_6cc9_451b_8888_2f7e69a0b888.slice/crio-7c8cc6dd969b15e9751ae1734b2ec3732f464206bb71fec2e9419a0e96fdaaa7 WatchSource:0}: Error finding container 7c8cc6dd969b15e9751ae1734b2ec3732f464206bb71fec2e9419a0e96fdaaa7: Status 404 returned error can't find the container with id 7c8cc6dd969b15e9751ae1734b2ec3732f464206bb71fec2e9419a0e96fdaaa7
May 26 16:11:11.753055 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:11.753017    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-ddd4dbfd-kvdvc" event={"ID":"e1bce4d4-6cc9-451b-8888-2f7e69a0b888","Type":"ContainerStarted","Data":"7c8cc6dd969b15e9751ae1734b2ec3732f464206bb71fec2e9419a0e96fdaaa7"}
May 26 16:11:13.759532 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:13.759497    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-ddd4dbfd-kvdvc" event={"ID":"e1bce4d4-6cc9-451b-8888-2f7e69a0b888","Type":"ContainerStarted","Data":"fcc9163753260b8e899ed7f74d23030c98bc1d12676708704b204ab4292aa5d8"}
May 26 16:11:13.775743 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:13.775689    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-ddd4dbfd-kvdvc" podStartSLOduration=1.1279162010000001 podStartE2EDuration="3.775673864s" podCreationTimestamp="2026-05-26 16:11:10 +0000 UTC" firstStartedPulling="2026-05-26 16:11:10.900262001 +0000 UTC m=+587.375989328" lastFinishedPulling="2026-05-26 16:11:13.548019651 +0000 UTC m=+590.023746991" observedRunningTime="2026-05-26 16:11:13.774173801 +0000 UTC m=+590.249901152" watchObservedRunningTime="2026-05-26 16:11:13.775673864 +0000 UTC m=+590.251401213"
May 26 16:11:14.687129 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:14.687093    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-ddd4dbfd-kvdvc"]
May 26 16:11:15.764654 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:15.764588    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve/seaweedfs-tls-custom-ddd4dbfd-kvdvc" podUID="e1bce4d4-6cc9-451b-8888-2f7e69a0b888" containerName="seaweedfs-tls-custom" containerID="cri-o://fcc9163753260b8e899ed7f74d23030c98bc1d12676708704b204ab4292aa5d8" gracePeriod=30
May 26 16:11:44.701899 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:44.701875    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-ddd4dbfd-kvdvc"
May 26 16:11:44.738687 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:44.738656    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e1bce4d4-6cc9-451b-8888-2f7e69a0b888-data\") pod \"e1bce4d4-6cc9-451b-8888-2f7e69a0b888\" (UID: \"e1bce4d4-6cc9-451b-8888-2f7e69a0b888\") "
May 26 16:11:44.738845 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:44.738705    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5np7b\" (UniqueName: \"kubernetes.io/projected/e1bce4d4-6cc9-451b-8888-2f7e69a0b888-kube-api-access-5np7b\") pod \"e1bce4d4-6cc9-451b-8888-2f7e69a0b888\" (UID: \"e1bce4d4-6cc9-451b-8888-2f7e69a0b888\") "
May 26 16:11:44.739846 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:44.739818    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1bce4d4-6cc9-451b-8888-2f7e69a0b888-data" (OuterVolumeSpecName: "data") pod "e1bce4d4-6cc9-451b-8888-2f7e69a0b888" (UID: "e1bce4d4-6cc9-451b-8888-2f7e69a0b888"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:11:44.740932 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:44.740907    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1bce4d4-6cc9-451b-8888-2f7e69a0b888-kube-api-access-5np7b" (OuterVolumeSpecName: "kube-api-access-5np7b") pod "e1bce4d4-6cc9-451b-8888-2f7e69a0b888" (UID: "e1bce4d4-6cc9-451b-8888-2f7e69a0b888"). InnerVolumeSpecName "kube-api-access-5np7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:11:44.839167 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:44.839076    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5np7b\" (UniqueName: \"kubernetes.io/projected/e1bce4d4-6cc9-451b-8888-2f7e69a0b888-kube-api-access-5np7b\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:11:44.839167 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:44.839101    2572 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e1bce4d4-6cc9-451b-8888-2f7e69a0b888-data\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:11:44.841346 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:44.841315    2572 generic.go:358] "Generic (PLEG): container finished" podID="e1bce4d4-6cc9-451b-8888-2f7e69a0b888" containerID="fcc9163753260b8e899ed7f74d23030c98bc1d12676708704b204ab4292aa5d8" exitCode=0
May 26 16:11:44.841485 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:44.841364    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-ddd4dbfd-kvdvc" event={"ID":"e1bce4d4-6cc9-451b-8888-2f7e69a0b888","Type":"ContainerDied","Data":"fcc9163753260b8e899ed7f74d23030c98bc1d12676708704b204ab4292aa5d8"}
May 26 16:11:44.841485 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:44.841371    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-ddd4dbfd-kvdvc"
May 26 16:11:44.841485 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:44.841390    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-ddd4dbfd-kvdvc" event={"ID":"e1bce4d4-6cc9-451b-8888-2f7e69a0b888","Type":"ContainerDied","Data":"7c8cc6dd969b15e9751ae1734b2ec3732f464206bb71fec2e9419a0e96fdaaa7"}
May 26 16:11:44.841485 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:44.841405    2572 scope.go:117] "RemoveContainer" containerID="fcc9163753260b8e899ed7f74d23030c98bc1d12676708704b204ab4292aa5d8"
May 26 16:11:44.850372 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:44.850352    2572 scope.go:117] "RemoveContainer" containerID="fcc9163753260b8e899ed7f74d23030c98bc1d12676708704b204ab4292aa5d8"
May 26 16:11:44.850627 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:11:44.850590    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcc9163753260b8e899ed7f74d23030c98bc1d12676708704b204ab4292aa5d8\": container with ID starting with fcc9163753260b8e899ed7f74d23030c98bc1d12676708704b204ab4292aa5d8 not found: ID does not exist" containerID="fcc9163753260b8e899ed7f74d23030c98bc1d12676708704b204ab4292aa5d8"
May 26 16:11:44.850719 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:44.850635    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcc9163753260b8e899ed7f74d23030c98bc1d12676708704b204ab4292aa5d8"} err="failed to get container status \"fcc9163753260b8e899ed7f74d23030c98bc1d12676708704b204ab4292aa5d8\": rpc error: code = NotFound desc = could not find container \"fcc9163753260b8e899ed7f74d23030c98bc1d12676708704b204ab4292aa5d8\": container with ID starting with fcc9163753260b8e899ed7f74d23030c98bc1d12676708704b204ab4292aa5d8 not found: ID does not exist"
May 26 16:11:44.860116 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:44.860091    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-ddd4dbfd-kvdvc"]
May 26 16:11:44.864011 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:44.863975    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve/seaweedfs-tls-custom-ddd4dbfd-kvdvc"]
May 26 16:11:44.893871 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:44.893839    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-5c88b85bb7-sgf6n"]
May 26 16:11:44.894080 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:44.894068    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1bce4d4-6cc9-451b-8888-2f7e69a0b888" containerName="seaweedfs-tls-custom"
May 26 16:11:44.894131 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:44.894081    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1bce4d4-6cc9-451b-8888-2f7e69a0b888" containerName="seaweedfs-tls-custom"
May 26 16:11:44.894131 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:44.894127    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="e1bce4d4-6cc9-451b-8888-2f7e69a0b888" containerName="seaweedfs-tls-custom"
May 26 16:11:44.897259 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:44.897235    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-5c88b85bb7-sgf6n"
May 26 16:11:44.899365 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:44.899344    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
May 26 16:11:44.899539 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:44.899518    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-gzm8h\""
May 26 16:11:44.899627 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:44.899583    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom\""
May 26 16:11:44.904312 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:44.904284    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-5c88b85bb7-sgf6n"]
May 26 16:11:44.939998 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:44.939967    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/56b94dfa-4f52-4112-bd8c-ddda1a1916fb-data\") pod \"seaweedfs-tls-custom-5c88b85bb7-sgf6n\" (UID: \"56b94dfa-4f52-4112-bd8c-ddda1a1916fb\") " pod="kserve/seaweedfs-tls-custom-5c88b85bb7-sgf6n"
May 26 16:11:44.940156 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:44.940018    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/56b94dfa-4f52-4112-bd8c-ddda1a1916fb-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-5c88b85bb7-sgf6n\" (UID: \"56b94dfa-4f52-4112-bd8c-ddda1a1916fb\") " pod="kserve/seaweedfs-tls-custom-5c88b85bb7-sgf6n"
May 26 16:11:44.940156 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:44.940040    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-694v6\" (UniqueName: \"kubernetes.io/projected/56b94dfa-4f52-4112-bd8c-ddda1a1916fb-kube-api-access-694v6\") pod \"seaweedfs-tls-custom-5c88b85bb7-sgf6n\" (UID: \"56b94dfa-4f52-4112-bd8c-ddda1a1916fb\") " pod="kserve/seaweedfs-tls-custom-5c88b85bb7-sgf6n"
May 26 16:11:45.040651 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:45.040596    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/56b94dfa-4f52-4112-bd8c-ddda1a1916fb-data\") pod \"seaweedfs-tls-custom-5c88b85bb7-sgf6n\" (UID: \"56b94dfa-4f52-4112-bd8c-ddda1a1916fb\") " pod="kserve/seaweedfs-tls-custom-5c88b85bb7-sgf6n"
May 26 16:11:45.040800 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:45.040678    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/56b94dfa-4f52-4112-bd8c-ddda1a1916fb-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-5c88b85bb7-sgf6n\" (UID: \"56b94dfa-4f52-4112-bd8c-ddda1a1916fb\") " pod="kserve/seaweedfs-tls-custom-5c88b85bb7-sgf6n"
May 26 16:11:45.040800 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:45.040698    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-694v6\" (UniqueName: \"kubernetes.io/projected/56b94dfa-4f52-4112-bd8c-ddda1a1916fb-kube-api-access-694v6\") pod \"seaweedfs-tls-custom-5c88b85bb7-sgf6n\" (UID: \"56b94dfa-4f52-4112-bd8c-ddda1a1916fb\") " pod="kserve/seaweedfs-tls-custom-5c88b85bb7-sgf6n"
May 26 16:11:45.040996 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:45.040972    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/56b94dfa-4f52-4112-bd8c-ddda1a1916fb-data\") pod \"seaweedfs-tls-custom-5c88b85bb7-sgf6n\" (UID: \"56b94dfa-4f52-4112-bd8c-ddda1a1916fb\") " pod="kserve/seaweedfs-tls-custom-5c88b85bb7-sgf6n"
May 26 16:11:45.043173 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:45.043147    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/56b94dfa-4f52-4112-bd8c-ddda1a1916fb-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-5c88b85bb7-sgf6n\" (UID: \"56b94dfa-4f52-4112-bd8c-ddda1a1916fb\") " pod="kserve/seaweedfs-tls-custom-5c88b85bb7-sgf6n"
May 26 16:11:45.050131 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:45.050110    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-694v6\" (UniqueName: \"kubernetes.io/projected/56b94dfa-4f52-4112-bd8c-ddda1a1916fb-kube-api-access-694v6\") pod \"seaweedfs-tls-custom-5c88b85bb7-sgf6n\" (UID: \"56b94dfa-4f52-4112-bd8c-ddda1a1916fb\") " pod="kserve/seaweedfs-tls-custom-5c88b85bb7-sgf6n"
May 26 16:11:45.207087 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:45.207055    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-5c88b85bb7-sgf6n"
May 26 16:11:45.332309 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:45.332280    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-5c88b85bb7-sgf6n"]
May 26 16:11:45.335136 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:11:45.335107    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56b94dfa_4f52_4112_bd8c_ddda1a1916fb.slice/crio-3721d4edd224b71f7eb717538550f68a07e1af1b5c8873823a4bd0c8f49a5527 WatchSource:0}: Error finding container 3721d4edd224b71f7eb717538550f68a07e1af1b5c8873823a4bd0c8f49a5527: Status 404 returned error can't find the container with id 3721d4edd224b71f7eb717538550f68a07e1af1b5c8873823a4bd0c8f49a5527
May 26 16:11:45.845896 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:45.845861    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-5c88b85bb7-sgf6n" event={"ID":"56b94dfa-4f52-4112-bd8c-ddda1a1916fb","Type":"ContainerStarted","Data":"dfee8477f2a4c74f154062430af7e77ca896ea1cbad5481f929d6e62dbd34429"}
May 26 16:11:45.845896 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:45.845899    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-5c88b85bb7-sgf6n" event={"ID":"56b94dfa-4f52-4112-bd8c-ddda1a1916fb","Type":"ContainerStarted","Data":"3721d4edd224b71f7eb717538550f68a07e1af1b5c8873823a4bd0c8f49a5527"}
May 26 16:11:45.861875 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:45.861756    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-5c88b85bb7-sgf6n" podStartSLOduration=1.5955249820000001 podStartE2EDuration="1.861740867s" podCreationTimestamp="2026-05-26 16:11:44 +0000 UTC" firstStartedPulling="2026-05-26 16:11:45.336370381 +0000 UTC m=+621.812097710" lastFinishedPulling="2026-05-26 16:11:45.602586267 +0000 UTC m=+622.078313595" observedRunningTime="2026-05-26 16:11:45.860152936 +0000 UTC m=+622.335880286" watchObservedRunningTime="2026-05-26 16:11:45.861740867 +0000 UTC m=+622.337468218"
May 26 16:11:46.078686 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:46.078650    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1bce4d4-6cc9-451b-8888-2f7e69a0b888" path="/var/lib/kubelet/pods/e1bce4d4-6cc9-451b-8888-2f7e69a0b888/volumes"
May 26 16:11:59.246653 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:59.246612    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-serving-7fd5766db9-8zs7n"]
May 26 16:11:59.248727 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:59.248712    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-7fd5766db9-8zs7n"
May 26 16:11:59.250878 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:59.250850    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving\""
May 26 16:11:59.250967 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:59.250901    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
May 26 16:11:59.258395 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:59.258364    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-7fd5766db9-8zs7n"]
May 26 16:11:59.332414 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:59.332369    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2kkp\" (UniqueName: \"kubernetes.io/projected/75f178e5-c6fa-4767-b2d4-35d1f685ab34-kube-api-access-k2kkp\") pod \"seaweedfs-tls-serving-7fd5766db9-8zs7n\" (UID: \"75f178e5-c6fa-4767-b2d4-35d1f685ab34\") " pod="kserve/seaweedfs-tls-serving-7fd5766db9-8zs7n"
May 26 16:11:59.332594 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:59.332426    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/75f178e5-c6fa-4767-b2d4-35d1f685ab34-data\") pod \"seaweedfs-tls-serving-7fd5766db9-8zs7n\" (UID: \"75f178e5-c6fa-4767-b2d4-35d1f685ab34\") " pod="kserve/seaweedfs-tls-serving-7fd5766db9-8zs7n"
May 26 16:11:59.332594 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:59.332478    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/75f178e5-c6fa-4767-b2d4-35d1f685ab34-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-7fd5766db9-8zs7n\" (UID: \"75f178e5-c6fa-4767-b2d4-35d1f685ab34\") " pod="kserve/seaweedfs-tls-serving-7fd5766db9-8zs7n"
May 26 16:11:59.433695 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:59.433660    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/75f178e5-c6fa-4767-b2d4-35d1f685ab34-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-7fd5766db9-8zs7n\" (UID: \"75f178e5-c6fa-4767-b2d4-35d1f685ab34\") " pod="kserve/seaweedfs-tls-serving-7fd5766db9-8zs7n"
May 26 16:11:59.433902 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:59.433705    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-k2kkp\" (UniqueName: \"kubernetes.io/projected/75f178e5-c6fa-4767-b2d4-35d1f685ab34-kube-api-access-k2kkp\") pod \"seaweedfs-tls-serving-7fd5766db9-8zs7n\" (UID: \"75f178e5-c6fa-4767-b2d4-35d1f685ab34\") " pod="kserve/seaweedfs-tls-serving-7fd5766db9-8zs7n"
May 26 16:11:59.433902 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:59.433731    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/75f178e5-c6fa-4767-b2d4-35d1f685ab34-data\") pod \"seaweedfs-tls-serving-7fd5766db9-8zs7n\" (UID: \"75f178e5-c6fa-4767-b2d4-35d1f685ab34\") " pod="kserve/seaweedfs-tls-serving-7fd5766db9-8zs7n"
May 26 16:11:59.433902 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:11:59.433836    2572 projected.go:264] Couldn't get secret kserve/seaweedfs-tls-serving: secret "seaweedfs-tls-serving" not found
May 26 16:11:59.433902 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:11:59.433859    2572 projected.go:194] Error preparing data for projected volume seaweedfs-tls-serving for pod kserve/seaweedfs-tls-serving-7fd5766db9-8zs7n: secret "seaweedfs-tls-serving" not found
May 26 16:11:59.434113 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:11:59.433933    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/75f178e5-c6fa-4767-b2d4-35d1f685ab34-seaweedfs-tls-serving podName:75f178e5-c6fa-4767-b2d4-35d1f685ab34 nodeName:}" failed. No retries permitted until 2026-05-26 16:11:59.933913983 +0000 UTC m=+636.409641318 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "seaweedfs-tls-serving" (UniqueName: "kubernetes.io/projected/75f178e5-c6fa-4767-b2d4-35d1f685ab34-seaweedfs-tls-serving") pod "seaweedfs-tls-serving-7fd5766db9-8zs7n" (UID: "75f178e5-c6fa-4767-b2d4-35d1f685ab34") : secret "seaweedfs-tls-serving" not found
May 26 16:11:59.434113 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:59.434043    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/75f178e5-c6fa-4767-b2d4-35d1f685ab34-data\") pod \"seaweedfs-tls-serving-7fd5766db9-8zs7n\" (UID: \"75f178e5-c6fa-4767-b2d4-35d1f685ab34\") " pod="kserve/seaweedfs-tls-serving-7fd5766db9-8zs7n"
May 26 16:11:59.442945 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:59.442916    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2kkp\" (UniqueName: \"kubernetes.io/projected/75f178e5-c6fa-4767-b2d4-35d1f685ab34-kube-api-access-k2kkp\") pod \"seaweedfs-tls-serving-7fd5766db9-8zs7n\" (UID: \"75f178e5-c6fa-4767-b2d4-35d1f685ab34\") " pod="kserve/seaweedfs-tls-serving-7fd5766db9-8zs7n"
May 26 16:11:59.937219 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:59.937181    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/75f178e5-c6fa-4767-b2d4-35d1f685ab34-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-7fd5766db9-8zs7n\" (UID: \"75f178e5-c6fa-4767-b2d4-35d1f685ab34\") " pod="kserve/seaweedfs-tls-serving-7fd5766db9-8zs7n"
May 26 16:11:59.939532 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:11:59.939513    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/75f178e5-c6fa-4767-b2d4-35d1f685ab34-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-7fd5766db9-8zs7n\" (UID: \"75f178e5-c6fa-4767-b2d4-35d1f685ab34\") " pod="kserve/seaweedfs-tls-serving-7fd5766db9-8zs7n"
May 26 16:12:00.158159 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:12:00.158121    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-7fd5766db9-8zs7n"
May 26 16:12:00.279218 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:12:00.279185    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-7fd5766db9-8zs7n"]
May 26 16:12:00.282254 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:12:00.282222    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75f178e5_c6fa_4767_b2d4_35d1f685ab34.slice/crio-fce0980bda6f8bb126ac0856a4f030c9585d80f8aa895ad7cbd1b62ccbe5ffc4 WatchSource:0}: Error finding container fce0980bda6f8bb126ac0856a4f030c9585d80f8aa895ad7cbd1b62ccbe5ffc4: Status 404 returned error can't find the container with id fce0980bda6f8bb126ac0856a4f030c9585d80f8aa895ad7cbd1b62ccbe5ffc4
May 26 16:12:00.885354 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:12:00.885318    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-7fd5766db9-8zs7n" event={"ID":"75f178e5-c6fa-4767-b2d4-35d1f685ab34","Type":"ContainerStarted","Data":"1665d378f8c6ca1ff6d654ce6f106afbcac1b712be843025ae912c5cab337960"}
May 26 16:12:00.885354 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:12:00.885358    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-7fd5766db9-8zs7n" event={"ID":"75f178e5-c6fa-4767-b2d4-35d1f685ab34","Type":"ContainerStarted","Data":"fce0980bda6f8bb126ac0856a4f030c9585d80f8aa895ad7cbd1b62ccbe5ffc4"}
May 26 16:12:00.900161 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:12:00.900107    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/seaweedfs-tls-serving-7fd5766db9-8zs7n" podStartSLOduration=1.626394779 podStartE2EDuration="1.900093046s" podCreationTimestamp="2026-05-26 16:11:59 +0000 UTC" firstStartedPulling="2026-05-26 16:12:00.283583568 +0000 UTC m=+636.759310897" lastFinishedPulling="2026-05-26 16:12:00.557281836 +0000 UTC m=+637.033009164" observedRunningTime="2026-05-26 16:12:00.899848207 +0000 UTC m=+637.375575676" watchObservedRunningTime="2026-05-26 16:12:00.900093046 +0000 UTC m=+637.375820396"
May 26 16:12:17.768615 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:12:17.768576    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m"]
May 26 16:12:17.774925 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:12:17.773548    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m"
May 26 16:12:17.776048 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:12:17.776022    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
May 26 16:12:17.776175 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:12:17.776031    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-c9tjh\""
May 26 16:12:17.776353 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:12:17.776334    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 26 16:12:17.776447 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:12:17.776379    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
May 26 16:12:17.776752 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:12:17.776735    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 26 16:12:17.782095 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:12:17.782073    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m"]
May 26 16:12:17.871399 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:12:17.871365    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb62w\" (UniqueName: \"kubernetes.io/projected/6c95e294-0011-45ea-b102-11aa9234b4f8-kube-api-access-cb62w\") pod \"isvc-sklearn-batcher-predictor-54b87fc785-r767m\" (UID: \"6c95e294-0011-45ea-b102-11aa9234b4f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m"
May 26 16:12:17.871566 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:12:17.871425    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c95e294-0011-45ea-b102-11aa9234b4f8-isvc-sklearn-batcher-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-54b87fc785-r767m\" (UID: \"6c95e294-0011-45ea-b102-11aa9234b4f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m"
May 26 16:12:17.871566 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:12:17.871502    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c95e294-0011-45ea-b102-11aa9234b4f8-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-54b87fc785-r767m\" (UID: \"6c95e294-0011-45ea-b102-11aa9234b4f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m"
May 26 16:12:17.871566 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:12:17.871546    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c95e294-0011-45ea-b102-11aa9234b4f8-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-54b87fc785-r767m\" (UID: \"6c95e294-0011-45ea-b102-11aa9234b4f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m"
May 26 16:12:17.972860 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:12:17.972824    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c95e294-0011-45ea-b102-11aa9234b4f8-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-54b87fc785-r767m\" (UID: \"6c95e294-0011-45ea-b102-11aa9234b4f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m"
May 26 16:12:17.972860 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:12:17.972869    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cb62w\" (UniqueName: \"kubernetes.io/projected/6c95e294-0011-45ea-b102-11aa9234b4f8-kube-api-access-cb62w\") pod \"isvc-sklearn-batcher-predictor-54b87fc785-r767m\" (UID: \"6c95e294-0011-45ea-b102-11aa9234b4f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m"
May 26 16:12:17.973127 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:12:17.972930    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c95e294-0011-45ea-b102-11aa9234b4f8-isvc-sklearn-batcher-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-54b87fc785-r767m\" (UID: \"6c95e294-0011-45ea-b102-11aa9234b4f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m"
May 26 16:12:17.973127 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:12:17.972970    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c95e294-0011-45ea-b102-11aa9234b4f8-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-54b87fc785-r767m\" (UID: \"6c95e294-0011-45ea-b102-11aa9234b4f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m"
May 26 16:12:17.973127 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:12:17.973077    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-serving-cert: secret "isvc-sklearn-batcher-predictor-serving-cert" not found
May 26 16:12:17.973280 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:12:17.973159    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c95e294-0011-45ea-b102-11aa9234b4f8-proxy-tls podName:6c95e294-0011-45ea-b102-11aa9234b4f8 nodeName:}" failed. No retries permitted until 2026-05-26 16:12:18.473135944 +0000 UTC m=+654.948863274 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6c95e294-0011-45ea-b102-11aa9234b4f8-proxy-tls") pod "isvc-sklearn-batcher-predictor-54b87fc785-r767m" (UID: "6c95e294-0011-45ea-b102-11aa9234b4f8") : secret "isvc-sklearn-batcher-predictor-serving-cert" not found
May 26 16:12:17.973351 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:12:17.973296    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c95e294-0011-45ea-b102-11aa9234b4f8-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-54b87fc785-r767m\" (UID: \"6c95e294-0011-45ea-b102-11aa9234b4f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m"
May 26 16:12:17.973729 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:12:17.973707    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c95e294-0011-45ea-b102-11aa9234b4f8-isvc-sklearn-batcher-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-54b87fc785-r767m\" (UID: \"6c95e294-0011-45ea-b102-11aa9234b4f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m"
May 26 16:12:17.983165 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:12:17.983139    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb62w\" (UniqueName: \"kubernetes.io/projected/6c95e294-0011-45ea-b102-11aa9234b4f8-kube-api-access-cb62w\") pod \"isvc-sklearn-batcher-predictor-54b87fc785-r767m\" (UID: \"6c95e294-0011-45ea-b102-11aa9234b4f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m"
May 26 16:12:18.476899 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:12:18.476856    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c95e294-0011-45ea-b102-11aa9234b4f8-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-54b87fc785-r767m\" (UID: \"6c95e294-0011-45ea-b102-11aa9234b4f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m"
May 26 16:12:18.479390 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:12:18.479364    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c95e294-0011-45ea-b102-11aa9234b4f8-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-54b87fc785-r767m\" (UID: \"6c95e294-0011-45ea-b102-11aa9234b4f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m"
May 26 16:12:18.685786 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:12:18.685752    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m"
May 26 16:12:18.813856 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:12:18.813816    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m"]
May 26 16:12:18.818563 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:12:18.818531    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c95e294_0011_45ea_b102_11aa9234b4f8.slice/crio-03af5e1f211ab3ec2ca55549ea56dc05b902d10db22e0d2544bbe49cb229a08c WatchSource:0}: Error finding container 03af5e1f211ab3ec2ca55549ea56dc05b902d10db22e0d2544bbe49cb229a08c: Status 404 returned error can't find the container with id 03af5e1f211ab3ec2ca55549ea56dc05b902d10db22e0d2544bbe49cb229a08c
May 26 16:12:18.934513 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:12:18.934477    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m" event={"ID":"6c95e294-0011-45ea-b102-11aa9234b4f8","Type":"ContainerStarted","Data":"03af5e1f211ab3ec2ca55549ea56dc05b902d10db22e0d2544bbe49cb229a08c"}
May 26 16:12:22.948260 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:12:22.948217    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m" event={"ID":"6c95e294-0011-45ea-b102-11aa9234b4f8","Type":"ContainerStarted","Data":"f3708034b68945ca5d019cc9fb51343195c3701c112e0ac661578efe51d005d2"}
May 26 16:12:26.960344 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:12:26.960309    2572 generic.go:358] "Generic (PLEG): container finished" podID="6c95e294-0011-45ea-b102-11aa9234b4f8" containerID="f3708034b68945ca5d019cc9fb51343195c3701c112e0ac661578efe51d005d2" exitCode=0
May 26 16:12:26.960722 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:12:26.960365    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m" event={"ID":"6c95e294-0011-45ea-b102-11aa9234b4f8","Type":"ContainerDied","Data":"f3708034b68945ca5d019cc9fb51343195c3701c112e0ac661578efe51d005d2"}
May 26 16:12:41.005165 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:12:41.005127    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m" event={"ID":"6c95e294-0011-45ea-b102-11aa9234b4f8","Type":"ContainerStarted","Data":"ff598fbaf8950d4b66c98d1f583a07b473a3597e898cc03a56a318b3cf1e5c6e"}
May 26 16:12:43.015693 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:12:43.015661    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m" event={"ID":"6c95e294-0011-45ea-b102-11aa9234b4f8","Type":"ContainerStarted","Data":"d8f37bd8c0ea50ef507515f6f33fc6360409b9cdddf286d16c21cf4853fcface"}
May 26 16:12:45.024026 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:12:45.023986    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m" event={"ID":"6c95e294-0011-45ea-b102-11aa9234b4f8","Type":"ContainerStarted","Data":"716c1093b05bcb12c103b5132f7cd0f9069cc8258eebe6476e644397ab81c6e2"}
May 26 16:12:45.024473 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:12:45.024242    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m"
May 26 16:12:45.024473 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:12:45.024277    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m"
May 26 16:12:45.025430 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:12:45.025399    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m" podUID="6c95e294-0011-45ea-b102-11aa9234b4f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
May 26 16:12:45.043317 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:12:45.043247    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m" podStartSLOduration=2.058469384 podStartE2EDuration="28.043226882s" podCreationTimestamp="2026-05-26 16:12:17 +0000 UTC" firstStartedPulling="2026-05-26 16:12:18.820831336 +0000 UTC m=+655.296558664" lastFinishedPulling="2026-05-26 16:12:44.805588833 +0000 UTC m=+681.281316162" observedRunningTime="2026-05-26 16:12:45.040961221 +0000 UTC m=+681.516688571" watchObservedRunningTime="2026-05-26 16:12:45.043226882 +0000 UTC m=+681.518954236"
May 26 16:12:46.026618 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:12:46.026559    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m"
May 26 16:12:46.027095 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:12:46.026747    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m" podUID="6c95e294-0011-45ea-b102-11aa9234b4f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
May 26 16:12:46.027685 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:12:46.027661    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m" podUID="6c95e294-0011-45ea-b102-11aa9234b4f8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:12:47.028978 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:12:47.028939    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m" podUID="6c95e294-0011-45ea-b102-11aa9234b4f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
May 26 16:12:47.029471 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:12:47.029444    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m" podUID="6c95e294-0011-45ea-b102-11aa9234b4f8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:12:47.032412 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:12:47.032393    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m"
May 26 16:12:48.031882 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:12:48.031842    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m" podUID="6c95e294-0011-45ea-b102-11aa9234b4f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
May 26 16:12:48.032310 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:12:48.032182    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m" podUID="6c95e294-0011-45ea-b102-11aa9234b4f8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:12:58.032771 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:12:58.032721    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m" podUID="6c95e294-0011-45ea-b102-11aa9234b4f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
May 26 16:12:58.033278 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:12:58.033179    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m" podUID="6c95e294-0011-45ea-b102-11aa9234b4f8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:13:08.032804 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:13:08.032748    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m" podUID="6c95e294-0011-45ea-b102-11aa9234b4f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
May 26 16:13:08.033278 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:13:08.033253    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m" podUID="6c95e294-0011-45ea-b102-11aa9234b4f8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:13:18.032637 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:13:18.032514    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m" podUID="6c95e294-0011-45ea-b102-11aa9234b4f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
May 26 16:13:18.033183 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:13:18.032999    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m" podUID="6c95e294-0011-45ea-b102-11aa9234b4f8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:13:28.032368 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:13:28.032319    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m" podUID="6c95e294-0011-45ea-b102-11aa9234b4f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
May 26 16:13:28.032801 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:13:28.032743    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m" podUID="6c95e294-0011-45ea-b102-11aa9234b4f8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:13:38.032440 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:13:38.032387    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m" podUID="6c95e294-0011-45ea-b102-11aa9234b4f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
May 26 16:13:38.032916 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:13:38.032868    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m" podUID="6c95e294-0011-45ea-b102-11aa9234b4f8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:13:48.032850 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:13:48.032812    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m"
May 26 16:13:48.033240 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:13:48.033059    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m"
May 26 16:14:02.779890 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:02.779853    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m"]
May 26 16:14:02.780351 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:02.780237    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m" podUID="6c95e294-0011-45ea-b102-11aa9234b4f8" containerName="kserve-container" containerID="cri-o://ff598fbaf8950d4b66c98d1f583a07b473a3597e898cc03a56a318b3cf1e5c6e" gracePeriod=30
May 26 16:14:02.780351 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:02.780274    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m" podUID="6c95e294-0011-45ea-b102-11aa9234b4f8" containerName="kube-rbac-proxy" containerID="cri-o://d8f37bd8c0ea50ef507515f6f33fc6360409b9cdddf286d16c21cf4853fcface" gracePeriod=30
May 26 16:14:02.780452 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:02.780268    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m" podUID="6c95e294-0011-45ea-b102-11aa9234b4f8" containerName="agent" containerID="cri-o://716c1093b05bcb12c103b5132f7cd0f9069cc8258eebe6476e644397ab81c6e2" gracePeriod=30
May 26 16:14:02.891412 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:02.891377    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk"]
May 26 16:14:02.899690 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:02.899649    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk"
May 26 16:14:02.901761 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:02.901731    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
May 26 16:14:02.901761 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:02.901735    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
May 26 16:14:02.906761 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:02.906731    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk"]
May 26 16:14:03.007589 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:03.007554    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f70d2026-ed8e-48b4-a081-a0bdffa145d3-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk\" (UID: \"f70d2026-ed8e-48b4-a081-a0bdffa145d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk"
May 26 16:14:03.007770 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:03.007622    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f70d2026-ed8e-48b4-a081-a0bdffa145d3-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk\" (UID: \"f70d2026-ed8e-48b4-a081-a0bdffa145d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk"
May 26 16:14:03.007770 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:03.007645    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f70d2026-ed8e-48b4-a081-a0bdffa145d3-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk\" (UID: \"f70d2026-ed8e-48b4-a081-a0bdffa145d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk"
May 26 16:14:03.007770 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:03.007666    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk5ls\" (UniqueName: \"kubernetes.io/projected/f70d2026-ed8e-48b4-a081-a0bdffa145d3-kube-api-access-vk5ls\") pod \"isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk\" (UID: \"f70d2026-ed8e-48b4-a081-a0bdffa145d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk"
May 26 16:14:03.109055 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:03.108973    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f70d2026-ed8e-48b4-a081-a0bdffa145d3-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk\" (UID: \"f70d2026-ed8e-48b4-a081-a0bdffa145d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk"
May 26 16:14:03.109055 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:03.109022    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f70d2026-ed8e-48b4-a081-a0bdffa145d3-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk\" (UID: \"f70d2026-ed8e-48b4-a081-a0bdffa145d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk"
May 26 16:14:03.109055 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:03.109046    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f70d2026-ed8e-48b4-a081-a0bdffa145d3-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk\" (UID: \"f70d2026-ed8e-48b4-a081-a0bdffa145d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk"
May 26 16:14:03.109279 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:03.109067    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vk5ls\" (UniqueName: \"kubernetes.io/projected/f70d2026-ed8e-48b4-a081-a0bdffa145d3-kube-api-access-vk5ls\") pod \"isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk\" (UID: \"f70d2026-ed8e-48b4-a081-a0bdffa145d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk"
May 26 16:14:03.109279 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:14:03.109207    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-serving-cert: secret "isvc-sklearn-batcher-custom-predictor-serving-cert" not found
May 26 16:14:03.109279 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:14:03.109270    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f70d2026-ed8e-48b4-a081-a0bdffa145d3-proxy-tls podName:f70d2026-ed8e-48b4-a081-a0bdffa145d3 nodeName:}" failed. No retries permitted until 2026-05-26 16:14:03.609249706 +0000 UTC m=+760.084977041 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f70d2026-ed8e-48b4-a081-a0bdffa145d3-proxy-tls") pod "isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk" (UID: "f70d2026-ed8e-48b4-a081-a0bdffa145d3") : secret "isvc-sklearn-batcher-custom-predictor-serving-cert" not found
May 26 16:14:03.109401 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:03.109388    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f70d2026-ed8e-48b4-a081-a0bdffa145d3-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk\" (UID: \"f70d2026-ed8e-48b4-a081-a0bdffa145d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk"
May 26 16:14:03.109709 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:03.109682    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f70d2026-ed8e-48b4-a081-a0bdffa145d3-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk\" (UID: \"f70d2026-ed8e-48b4-a081-a0bdffa145d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk"
May 26 16:14:03.117789 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:03.117766    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk5ls\" (UniqueName: \"kubernetes.io/projected/f70d2026-ed8e-48b4-a081-a0bdffa145d3-kube-api-access-vk5ls\") pod \"isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk\" (UID: \"f70d2026-ed8e-48b4-a081-a0bdffa145d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk"
May 26 16:14:03.235430 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:03.235397    2572 generic.go:358] "Generic (PLEG): container finished" podID="6c95e294-0011-45ea-b102-11aa9234b4f8" containerID="d8f37bd8c0ea50ef507515f6f33fc6360409b9cdddf286d16c21cf4853fcface" exitCode=2
May 26 16:14:03.235598 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:03.235436    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m" event={"ID":"6c95e294-0011-45ea-b102-11aa9234b4f8","Type":"ContainerDied","Data":"d8f37bd8c0ea50ef507515f6f33fc6360409b9cdddf286d16c21cf4853fcface"}
May 26 16:14:03.613312 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:03.613275    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f70d2026-ed8e-48b4-a081-a0bdffa145d3-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk\" (UID: \"f70d2026-ed8e-48b4-a081-a0bdffa145d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk"
May 26 16:14:03.615664 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:03.615644    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f70d2026-ed8e-48b4-a081-a0bdffa145d3-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk\" (UID: \"f70d2026-ed8e-48b4-a081-a0bdffa145d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk"
May 26 16:14:03.812300 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:03.812261    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk"
May 26 16:14:03.939191 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:03.939152    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk"]
May 26 16:14:03.942392 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:14:03.942352    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf70d2026_ed8e_48b4_a081_a0bdffa145d3.slice/crio-d28b01e59563c7d24f93e807c98c7d3e00e7dea8d580ac0e18a45b908c88ccc9 WatchSource:0}: Error finding container d28b01e59563c7d24f93e807c98c7d3e00e7dea8d580ac0e18a45b908c88ccc9: Status 404 returned error can't find the container with id d28b01e59563c7d24f93e807c98c7d3e00e7dea8d580ac0e18a45b908c88ccc9
May 26 16:14:04.239923 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:04.239843    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk" event={"ID":"f70d2026-ed8e-48b4-a081-a0bdffa145d3","Type":"ContainerStarted","Data":"6b5633cd7390dde935bd1bb8cc74f40836ed44a9e0b7f041da19eb2372db884d"}
May 26 16:14:04.239923 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:04.239881    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk" event={"ID":"f70d2026-ed8e-48b4-a081-a0bdffa145d3","Type":"ContainerStarted","Data":"d28b01e59563c7d24f93e807c98c7d3e00e7dea8d580ac0e18a45b908c88ccc9"}
May 26 16:14:07.029827 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:07.029780    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m" podUID="6c95e294-0011-45ea-b102-11aa9234b4f8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.21:8643/healthz\": dial tcp 10.132.0.21:8643: connect: connection refused"
May 26 16:14:07.250587 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:07.250551    2572 generic.go:358] "Generic (PLEG): container finished" podID="6c95e294-0011-45ea-b102-11aa9234b4f8" containerID="ff598fbaf8950d4b66c98d1f583a07b473a3597e898cc03a56a318b3cf1e5c6e" exitCode=0
May 26 16:14:07.250760 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:07.250622    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m" event={"ID":"6c95e294-0011-45ea-b102-11aa9234b4f8","Type":"ContainerDied","Data":"ff598fbaf8950d4b66c98d1f583a07b473a3597e898cc03a56a318b3cf1e5c6e"}
May 26 16:14:08.031866 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:08.031819    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m" podUID="6c95e294-0011-45ea-b102-11aa9234b4f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
May 26 16:14:08.032298 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:08.032117    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m" podUID="6c95e294-0011-45ea-b102-11aa9234b4f8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:14:08.254435 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:08.254400    2572 generic.go:358] "Generic (PLEG): container finished" podID="f70d2026-ed8e-48b4-a081-a0bdffa145d3" containerID="6b5633cd7390dde935bd1bb8cc74f40836ed44a9e0b7f041da19eb2372db884d" exitCode=0
May 26 16:14:08.254613 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:08.254470    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk" event={"ID":"f70d2026-ed8e-48b4-a081-a0bdffa145d3","Type":"ContainerDied","Data":"6b5633cd7390dde935bd1bb8cc74f40836ed44a9e0b7f041da19eb2372db884d"}
May 26 16:14:09.259924 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:09.259891    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk" event={"ID":"f70d2026-ed8e-48b4-a081-a0bdffa145d3","Type":"ContainerStarted","Data":"1039cbe48396eb20f4efc882d095ec7a87e8a98978d43f4964ff5af584539891"}
May 26 16:14:09.259924 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:09.259928    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk" event={"ID":"f70d2026-ed8e-48b4-a081-a0bdffa145d3","Type":"ContainerStarted","Data":"74a3be64103435df4f037b00bf18d9531723f2c9dc01a97ce8c225800ce3af68"}
May 26 16:14:09.260440 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:09.259940    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk" event={"ID":"f70d2026-ed8e-48b4-a081-a0bdffa145d3","Type":"ContainerStarted","Data":"19916178039f04b09b71add91571a8d62db99c6952186c23b4bab32488aec0b3"}
May 26 16:14:09.260440 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:09.260298    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk"
May 26 16:14:09.260440 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:09.260324    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk"
May 26 16:14:09.261631 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:09.261584    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk" podUID="f70d2026-ed8e-48b4-a081-a0bdffa145d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:5000: connect: connection refused"
May 26 16:14:09.278528 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:09.278461    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk" podStartSLOduration=7.278444152 podStartE2EDuration="7.278444152s" podCreationTimestamp="2026-05-26 16:14:02 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:14:09.277439348 +0000 UTC m=+765.753166699" watchObservedRunningTime="2026-05-26 16:14:09.278444152 +0000 UTC m=+765.754171502"
May 26 16:14:10.263751 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:10.263719    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk"
May 26 16:14:10.264146 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:10.263752    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk" podUID="f70d2026-ed8e-48b4-a081-a0bdffa145d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:5000: connect: connection refused"
May 26 16:14:10.264745 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:10.264720    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk" podUID="f70d2026-ed8e-48b4-a081-a0bdffa145d3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:14:11.267000 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:11.266967    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk" podUID="f70d2026-ed8e-48b4-a081-a0bdffa145d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:5000: connect: connection refused"
May 26 16:14:11.267410 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:11.267313    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk" podUID="f70d2026-ed8e-48b4-a081-a0bdffa145d3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:14:12.030103 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:12.030064    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m" podUID="6c95e294-0011-45ea-b102-11aa9234b4f8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.21:8643/healthz\": dial tcp 10.132.0.21:8643: connect: connection refused"
May 26 16:14:16.271285 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:16.271255    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk"
May 26 16:14:16.271860 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:16.271831    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk" podUID="f70d2026-ed8e-48b4-a081-a0bdffa145d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:5000: connect: connection refused"
May 26 16:14:16.272087 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:16.272055    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk" podUID="f70d2026-ed8e-48b4-a081-a0bdffa145d3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:14:17.029871 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:17.029818    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m" podUID="6c95e294-0011-45ea-b102-11aa9234b4f8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.21:8643/healthz\": dial tcp 10.132.0.21:8643: connect: connection refused"
May 26 16:14:17.030063 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:17.029951    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m"
May 26 16:14:18.032191 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:18.032147    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m" podUID="6c95e294-0011-45ea-b102-11aa9234b4f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
May 26 16:14:18.032678 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:18.032516    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m" podUID="6c95e294-0011-45ea-b102-11aa9234b4f8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:14:22.029428 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:22.029386    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m" podUID="6c95e294-0011-45ea-b102-11aa9234b4f8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.21:8643/healthz\": dial tcp 10.132.0.21:8643: connect: connection refused"
May 26 16:14:26.272311 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:26.272262    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk" podUID="f70d2026-ed8e-48b4-a081-a0bdffa145d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:5000: connect: connection refused"
May 26 16:14:26.272816 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:26.272768    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk" podUID="f70d2026-ed8e-48b4-a081-a0bdffa145d3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:14:27.029301 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:27.029259    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m" podUID="6c95e294-0011-45ea-b102-11aa9234b4f8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.21:8643/healthz\": dial tcp 10.132.0.21:8643: connect: connection refused"
May 26 16:14:28.032334 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:28.032289    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m" podUID="6c95e294-0011-45ea-b102-11aa9234b4f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
May 26 16:14:28.032778 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:28.032443    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m"
May 26 16:14:28.032778 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:28.032648    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m" podUID="6c95e294-0011-45ea-b102-11aa9234b4f8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:14:28.032778 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:28.032748    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m"
May 26 16:14:32.029741 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:32.029693    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m" podUID="6c95e294-0011-45ea-b102-11aa9234b4f8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.21:8643/healthz\": dial tcp 10.132.0.21:8643: connect: connection refused"
May 26 16:14:32.930057 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:32.930031    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m"
May 26 16:14:33.019322 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:33.019285    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c95e294-0011-45ea-b102-11aa9234b4f8-proxy-tls\") pod \"6c95e294-0011-45ea-b102-11aa9234b4f8\" (UID: \"6c95e294-0011-45ea-b102-11aa9234b4f8\") "
May 26 16:14:33.019512 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:33.019346    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c95e294-0011-45ea-b102-11aa9234b4f8-kserve-provision-location\") pod \"6c95e294-0011-45ea-b102-11aa9234b4f8\" (UID: \"6c95e294-0011-45ea-b102-11aa9234b4f8\") "
May 26 16:14:33.019512 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:33.019383    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cb62w\" (UniqueName: \"kubernetes.io/projected/6c95e294-0011-45ea-b102-11aa9234b4f8-kube-api-access-cb62w\") pod \"6c95e294-0011-45ea-b102-11aa9234b4f8\" (UID: \"6c95e294-0011-45ea-b102-11aa9234b4f8\") "
May 26 16:14:33.019512 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:33.019428    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c95e294-0011-45ea-b102-11aa9234b4f8-isvc-sklearn-batcher-kube-rbac-proxy-sar-config\") pod \"6c95e294-0011-45ea-b102-11aa9234b4f8\" (UID: \"6c95e294-0011-45ea-b102-11aa9234b4f8\") "
May 26 16:14:33.019751 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:33.019724    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c95e294-0011-45ea-b102-11aa9234b4f8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6c95e294-0011-45ea-b102-11aa9234b4f8" (UID: "6c95e294-0011-45ea-b102-11aa9234b4f8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:14:33.019873 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:33.019848    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c95e294-0011-45ea-b102-11aa9234b4f8-isvc-sklearn-batcher-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-batcher-kube-rbac-proxy-sar-config") pod "6c95e294-0011-45ea-b102-11aa9234b4f8" (UID: "6c95e294-0011-45ea-b102-11aa9234b4f8"). InnerVolumeSpecName "isvc-sklearn-batcher-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:14:33.021563 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:33.021538    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c95e294-0011-45ea-b102-11aa9234b4f8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6c95e294-0011-45ea-b102-11aa9234b4f8" (UID: "6c95e294-0011-45ea-b102-11aa9234b4f8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:14:33.021691 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:33.021670    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c95e294-0011-45ea-b102-11aa9234b4f8-kube-api-access-cb62w" (OuterVolumeSpecName: "kube-api-access-cb62w") pod "6c95e294-0011-45ea-b102-11aa9234b4f8" (UID: "6c95e294-0011-45ea-b102-11aa9234b4f8"). InnerVolumeSpecName "kube-api-access-cb62w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:14:33.120273 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:33.120238    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c95e294-0011-45ea-b102-11aa9234b4f8-proxy-tls\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:14:33.120649 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:33.120277    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c95e294-0011-45ea-b102-11aa9234b4f8-kserve-provision-location\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:14:33.120649 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:33.120291    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb62w\" (UniqueName: \"kubernetes.io/projected/6c95e294-0011-45ea-b102-11aa9234b4f8-kube-api-access-cb62w\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:14:33.120649 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:33.120306    2572 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c95e294-0011-45ea-b102-11aa9234b4f8-isvc-sklearn-batcher-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:14:33.333469 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:33.333430    2572 generic.go:358] "Generic (PLEG): container finished" podID="6c95e294-0011-45ea-b102-11aa9234b4f8" containerID="716c1093b05bcb12c103b5132f7cd0f9069cc8258eebe6476e644397ab81c6e2" exitCode=0
May 26 16:14:33.333469 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:33.333474    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m" event={"ID":"6c95e294-0011-45ea-b102-11aa9234b4f8","Type":"ContainerDied","Data":"716c1093b05bcb12c103b5132f7cd0f9069cc8258eebe6476e644397ab81c6e2"}
May 26 16:14:33.333738 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:33.333497    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m" event={"ID":"6c95e294-0011-45ea-b102-11aa9234b4f8","Type":"ContainerDied","Data":"03af5e1f211ab3ec2ca55549ea56dc05b902d10db22e0d2544bbe49cb229a08c"}
May 26 16:14:33.333738 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:33.333512    2572 scope.go:117] "RemoveContainer" containerID="716c1093b05bcb12c103b5132f7cd0f9069cc8258eebe6476e644397ab81c6e2"
May 26 16:14:33.333738 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:33.333525    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m"
May 26 16:14:33.344885 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:33.344864    2572 scope.go:117] "RemoveContainer" containerID="d8f37bd8c0ea50ef507515f6f33fc6360409b9cdddf286d16c21cf4853fcface"
May 26 16:14:33.352184 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:33.352165    2572 scope.go:117] "RemoveContainer" containerID="ff598fbaf8950d4b66c98d1f583a07b473a3597e898cc03a56a318b3cf1e5c6e"
May 26 16:14:33.355894 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:33.355868    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m"]
May 26 16:14:33.359465 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:33.359440    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-54b87fc785-r767m"]
May 26 16:14:33.360857 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:33.360836    2572 scope.go:117] "RemoveContainer" containerID="f3708034b68945ca5d019cc9fb51343195c3701c112e0ac661578efe51d005d2"
May 26 16:14:33.367737 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:33.367717    2572 scope.go:117] "RemoveContainer" containerID="716c1093b05bcb12c103b5132f7cd0f9069cc8258eebe6476e644397ab81c6e2"
May 26 16:14:33.367955 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:14:33.367936    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"716c1093b05bcb12c103b5132f7cd0f9069cc8258eebe6476e644397ab81c6e2\": container with ID starting with 716c1093b05bcb12c103b5132f7cd0f9069cc8258eebe6476e644397ab81c6e2 not found: ID does not exist" containerID="716c1093b05bcb12c103b5132f7cd0f9069cc8258eebe6476e644397ab81c6e2"
May 26 16:14:33.368010 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:33.367965    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"716c1093b05bcb12c103b5132f7cd0f9069cc8258eebe6476e644397ab81c6e2"} err="failed to get container status \"716c1093b05bcb12c103b5132f7cd0f9069cc8258eebe6476e644397ab81c6e2\": rpc error: code = NotFound desc = could not find container \"716c1093b05bcb12c103b5132f7cd0f9069cc8258eebe6476e644397ab81c6e2\": container with ID starting with 716c1093b05bcb12c103b5132f7cd0f9069cc8258eebe6476e644397ab81c6e2 not found: ID does not exist"
May 26 16:14:33.368010 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:33.367986    2572 scope.go:117] "RemoveContainer" containerID="d8f37bd8c0ea50ef507515f6f33fc6360409b9cdddf286d16c21cf4853fcface"
May 26 16:14:33.368190 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:14:33.368176    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8f37bd8c0ea50ef507515f6f33fc6360409b9cdddf286d16c21cf4853fcface\": container with ID starting with d8f37bd8c0ea50ef507515f6f33fc6360409b9cdddf286d16c21cf4853fcface not found: ID does not exist" containerID="d8f37bd8c0ea50ef507515f6f33fc6360409b9cdddf286d16c21cf4853fcface"
May 26 16:14:33.368242 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:33.368193    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8f37bd8c0ea50ef507515f6f33fc6360409b9cdddf286d16c21cf4853fcface"} err="failed to get container status \"d8f37bd8c0ea50ef507515f6f33fc6360409b9cdddf286d16c21cf4853fcface\": rpc error: code = NotFound desc = could not find container \"d8f37bd8c0ea50ef507515f6f33fc6360409b9cdddf286d16c21cf4853fcface\": container with ID starting with d8f37bd8c0ea50ef507515f6f33fc6360409b9cdddf286d16c21cf4853fcface not found: ID does not exist"
May 26 16:14:33.368242 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:33.368206    2572 scope.go:117] "RemoveContainer" containerID="ff598fbaf8950d4b66c98d1f583a07b473a3597e898cc03a56a318b3cf1e5c6e"
May 26 16:14:33.368434 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:14:33.368422    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff598fbaf8950d4b66c98d1f583a07b473a3597e898cc03a56a318b3cf1e5c6e\": container with ID starting with ff598fbaf8950d4b66c98d1f583a07b473a3597e898cc03a56a318b3cf1e5c6e not found: ID does not exist" containerID="ff598fbaf8950d4b66c98d1f583a07b473a3597e898cc03a56a318b3cf1e5c6e"
May 26 16:14:33.368478 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:33.368436    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff598fbaf8950d4b66c98d1f583a07b473a3597e898cc03a56a318b3cf1e5c6e"} err="failed to get container status \"ff598fbaf8950d4b66c98d1f583a07b473a3597e898cc03a56a318b3cf1e5c6e\": rpc error: code = NotFound desc = could not find container \"ff598fbaf8950d4b66c98d1f583a07b473a3597e898cc03a56a318b3cf1e5c6e\": container with ID starting with ff598fbaf8950d4b66c98d1f583a07b473a3597e898cc03a56a318b3cf1e5c6e not found: ID does not exist"
May 26 16:14:33.368478 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:33.368447    2572 scope.go:117] "RemoveContainer" containerID="f3708034b68945ca5d019cc9fb51343195c3701c112e0ac661578efe51d005d2"
May 26 16:14:33.368626 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:14:33.368595    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3708034b68945ca5d019cc9fb51343195c3701c112e0ac661578efe51d005d2\": container with ID starting with f3708034b68945ca5d019cc9fb51343195c3701c112e0ac661578efe51d005d2 not found: ID does not exist" containerID="f3708034b68945ca5d019cc9fb51343195c3701c112e0ac661578efe51d005d2"
May 26 16:14:33.368669 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:33.368630    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3708034b68945ca5d019cc9fb51343195c3701c112e0ac661578efe51d005d2"} err="failed to get container status \"f3708034b68945ca5d019cc9fb51343195c3701c112e0ac661578efe51d005d2\": rpc error: code = NotFound desc = could not find container \"f3708034b68945ca5d019cc9fb51343195c3701c112e0ac661578efe51d005d2\": container with ID starting with f3708034b68945ca5d019cc9fb51343195c3701c112e0ac661578efe51d005d2 not found: ID does not exist"
May 26 16:14:34.079053 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:34.079023    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c95e294-0011-45ea-b102-11aa9234b4f8" path="/var/lib/kubelet/pods/6c95e294-0011-45ea-b102-11aa9234b4f8/volumes"
May 26 16:14:36.271840 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:36.271802    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk" podUID="f70d2026-ed8e-48b4-a081-a0bdffa145d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:5000: connect: connection refused"
May 26 16:14:36.272278 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:36.272253    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk" podUID="f70d2026-ed8e-48b4-a081-a0bdffa145d3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:14:46.272288 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:46.272193    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk" podUID="f70d2026-ed8e-48b4-a081-a0bdffa145d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:5000: connect: connection refused"
May 26 16:14:46.272683 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:46.272620    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk" podUID="f70d2026-ed8e-48b4-a081-a0bdffa145d3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:14:56.272405 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:56.272357    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk" podUID="f70d2026-ed8e-48b4-a081-a0bdffa145d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:5000: connect: connection refused"
May 26 16:14:56.272842 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:14:56.272799    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk" podUID="f70d2026-ed8e-48b4-a081-a0bdffa145d3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:15:06.271799 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:06.271754    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk" podUID="f70d2026-ed8e-48b4-a081-a0bdffa145d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:5000: connect: connection refused"
May 26 16:15:06.272306 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:06.272279    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk" podUID="f70d2026-ed8e-48b4-a081-a0bdffa145d3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:15:16.272749 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:16.272719    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk"
May 26 16:15:16.273203 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:16.272777    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk"
May 26 16:15:27.947004 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:27.946956    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk"]
May 26 16:15:27.947504 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:27.947432    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk" podUID="f70d2026-ed8e-48b4-a081-a0bdffa145d3" containerName="kserve-container" containerID="cri-o://19916178039f04b09b71add91571a8d62db99c6952186c23b4bab32488aec0b3" gracePeriod=30
May 26 16:15:27.947579 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:27.947475    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk" podUID="f70d2026-ed8e-48b4-a081-a0bdffa145d3" containerName="agent" containerID="cri-o://1039cbe48396eb20f4efc882d095ec7a87e8a98978d43f4964ff5af584539891" gracePeriod=30
May 26 16:15:27.947579 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:27.947525    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk" podUID="f70d2026-ed8e-48b4-a081-a0bdffa145d3" containerName="kube-rbac-proxy" containerID="cri-o://74a3be64103435df4f037b00bf18d9531723f2c9dc01a97ce8c225800ce3af68" gracePeriod=30
May 26 16:15:28.014521 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:28.014466    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-7bgsr"]
May 26 16:15:28.014762 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:28.014749    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c95e294-0011-45ea-b102-11aa9234b4f8" containerName="kube-rbac-proxy"
May 26 16:15:28.014808 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:28.014764    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c95e294-0011-45ea-b102-11aa9234b4f8" containerName="kube-rbac-proxy"
May 26 16:15:28.014808 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:28.014779    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c95e294-0011-45ea-b102-11aa9234b4f8" containerName="storage-initializer"
May 26 16:15:28.014808 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:28.014784    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c95e294-0011-45ea-b102-11aa9234b4f8" containerName="storage-initializer"
May 26 16:15:28.014808 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:28.014798    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c95e294-0011-45ea-b102-11aa9234b4f8" containerName="kserve-container"
May 26 16:15:28.014808 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:28.014804    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c95e294-0011-45ea-b102-11aa9234b4f8" containerName="kserve-container"
May 26 16:15:28.014808 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:28.014809    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c95e294-0011-45ea-b102-11aa9234b4f8" containerName="agent"
May 26 16:15:28.015005 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:28.014814    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c95e294-0011-45ea-b102-11aa9234b4f8" containerName="agent"
May 26 16:15:28.015005 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:28.014859    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="6c95e294-0011-45ea-b102-11aa9234b4f8" containerName="kube-rbac-proxy"
May 26 16:15:28.015005 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:28.014869    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="6c95e294-0011-45ea-b102-11aa9234b4f8" containerName="agent"
May 26 16:15:28.015005 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:28.014877    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="6c95e294-0011-45ea-b102-11aa9234b4f8" containerName="kserve-container"
May 26 16:15:28.017223 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:28.017201    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-7bgsr"
May 26 16:15:28.019325 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:28.019302    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
May 26 16:15:28.019419 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:28.019347    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
May 26 16:15:28.028844 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:28.028817    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-7bgsr"]
May 26 16:15:28.143019 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:28.142983    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"message-dumper-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f46a336-6162-4e1a-9ee8-86c83137f04b-message-dumper-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-c7d86bcbd-7bgsr\" (UID: \"8f46a336-6162-4e1a-9ee8-86c83137f04b\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-7bgsr"
May 26 16:15:28.143019 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:28.143025    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7fv8\" (UniqueName: \"kubernetes.io/projected/8f46a336-6162-4e1a-9ee8-86c83137f04b-kube-api-access-l7fv8\") pod \"message-dumper-predictor-c7d86bcbd-7bgsr\" (UID: \"8f46a336-6162-4e1a-9ee8-86c83137f04b\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-7bgsr"
May 26 16:15:28.143208 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:28.143056    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f46a336-6162-4e1a-9ee8-86c83137f04b-proxy-tls\") pod \"message-dumper-predictor-c7d86bcbd-7bgsr\" (UID: \"8f46a336-6162-4e1a-9ee8-86c83137f04b\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-7bgsr"
May 26 16:15:28.243869 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:28.243777    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"message-dumper-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f46a336-6162-4e1a-9ee8-86c83137f04b-message-dumper-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-c7d86bcbd-7bgsr\" (UID: \"8f46a336-6162-4e1a-9ee8-86c83137f04b\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-7bgsr"
May 26 16:15:28.243869 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:28.243816    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-l7fv8\" (UniqueName: \"kubernetes.io/projected/8f46a336-6162-4e1a-9ee8-86c83137f04b-kube-api-access-l7fv8\") pod \"message-dumper-predictor-c7d86bcbd-7bgsr\" (UID: \"8f46a336-6162-4e1a-9ee8-86c83137f04b\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-7bgsr"
May 26 16:15:28.244100 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:28.243871    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f46a336-6162-4e1a-9ee8-86c83137f04b-proxy-tls\") pod \"message-dumper-predictor-c7d86bcbd-7bgsr\" (UID: \"8f46a336-6162-4e1a-9ee8-86c83137f04b\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-7bgsr"
May 26 16:15:28.244100 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:15:28.243988    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-predictor-serving-cert: secret "message-dumper-predictor-serving-cert" not found
May 26 16:15:28.244100 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:15:28.244060    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f46a336-6162-4e1a-9ee8-86c83137f04b-proxy-tls podName:8f46a336-6162-4e1a-9ee8-86c83137f04b nodeName:}" failed. No retries permitted until 2026-05-26 16:15:28.744039705 +0000 UTC m=+845.219767046 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8f46a336-6162-4e1a-9ee8-86c83137f04b-proxy-tls") pod "message-dumper-predictor-c7d86bcbd-7bgsr" (UID: "8f46a336-6162-4e1a-9ee8-86c83137f04b") : secret "message-dumper-predictor-serving-cert" not found
May 26 16:15:28.244547 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:28.244527    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"message-dumper-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f46a336-6162-4e1a-9ee8-86c83137f04b-message-dumper-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-c7d86bcbd-7bgsr\" (UID: \"8f46a336-6162-4e1a-9ee8-86c83137f04b\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-7bgsr"
May 26 16:15:28.251911 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:28.251885    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7fv8\" (UniqueName: \"kubernetes.io/projected/8f46a336-6162-4e1a-9ee8-86c83137f04b-kube-api-access-l7fv8\") pod \"message-dumper-predictor-c7d86bcbd-7bgsr\" (UID: \"8f46a336-6162-4e1a-9ee8-86c83137f04b\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-7bgsr"
May 26 16:15:28.487200 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:28.487167    2572 generic.go:358] "Generic (PLEG): container finished" podID="f70d2026-ed8e-48b4-a081-a0bdffa145d3" containerID="74a3be64103435df4f037b00bf18d9531723f2c9dc01a97ce8c225800ce3af68" exitCode=2
May 26 16:15:28.487200 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:28.487204    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk" event={"ID":"f70d2026-ed8e-48b4-a081-a0bdffa145d3","Type":"ContainerDied","Data":"74a3be64103435df4f037b00bf18d9531723f2c9dc01a97ce8c225800ce3af68"}
May 26 16:15:28.748523 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:28.748489    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f46a336-6162-4e1a-9ee8-86c83137f04b-proxy-tls\") pod \"message-dumper-predictor-c7d86bcbd-7bgsr\" (UID: \"8f46a336-6162-4e1a-9ee8-86c83137f04b\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-7bgsr"
May 26 16:15:28.751023 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:28.750996    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f46a336-6162-4e1a-9ee8-86c83137f04b-proxy-tls\") pod \"message-dumper-predictor-c7d86bcbd-7bgsr\" (UID: \"8f46a336-6162-4e1a-9ee8-86c83137f04b\") " pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-7bgsr"
May 26 16:15:28.926730 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:28.926690    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-7bgsr"
May 26 16:15:29.050330 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:29.050248    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-7bgsr"]
May 26 16:15:29.052429 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:15:29.052395    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f46a336_6162_4e1a_9ee8_86c83137f04b.slice/crio-a30a4fd7d77d45d8a7ac7f17b89ec44cfbf63e4adfb6982b55b71eebe7545365 WatchSource:0}: Error finding container a30a4fd7d77d45d8a7ac7f17b89ec44cfbf63e4adfb6982b55b71eebe7545365: Status 404 returned error can't find the container with id a30a4fd7d77d45d8a7ac7f17b89ec44cfbf63e4adfb6982b55b71eebe7545365
May 26 16:15:29.054440 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:29.054425    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 16:15:29.491425 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:29.491392    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-7bgsr" event={"ID":"8f46a336-6162-4e1a-9ee8-86c83137f04b","Type":"ContainerStarted","Data":"a30a4fd7d77d45d8a7ac7f17b89ec44cfbf63e4adfb6982b55b71eebe7545365"}
May 26 16:15:30.496164 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:30.496125    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-7bgsr" event={"ID":"8f46a336-6162-4e1a-9ee8-86c83137f04b","Type":"ContainerStarted","Data":"ae43ba558c83cc4dbc9239834aa30cb5aa06e4e0abe4fb5f987d465ee1ba2b36"}
May 26 16:15:30.496164 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:30.496162    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-7bgsr" event={"ID":"8f46a336-6162-4e1a-9ee8-86c83137f04b","Type":"ContainerStarted","Data":"7bec13a917e2ccef6b7d8f37c76c8b1f7602373c6340470bcdc42d84323fa659"}
May 26 16:15:30.496580 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:30.496235    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-7bgsr"
May 26 16:15:30.512029 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:30.511974    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-7bgsr" podStartSLOduration=2.629228468 podStartE2EDuration="3.511956497s" podCreationTimestamp="2026-05-26 16:15:27 +0000 UTC" firstStartedPulling="2026-05-26 16:15:29.054555921 +0000 UTC m=+845.530283249" lastFinishedPulling="2026-05-26 16:15:29.937283935 +0000 UTC m=+846.413011278" observedRunningTime="2026-05-26 16:15:30.510696418 +0000 UTC m=+846.986423770" watchObservedRunningTime="2026-05-26 16:15:30.511956497 +0000 UTC m=+846.987683846"
May 26 16:15:31.267507 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:31.267464    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk" podUID="f70d2026-ed8e-48b4-a081-a0bdffa145d3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
May 26 16:15:31.499181 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:31.499150    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-7bgsr"
May 26 16:15:31.500890 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:31.500867    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-7bgsr"
May 26 16:15:32.503392 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:32.503302    2572 generic.go:358] "Generic (PLEG): container finished" podID="f70d2026-ed8e-48b4-a081-a0bdffa145d3" containerID="19916178039f04b09b71add91571a8d62db99c6952186c23b4bab32488aec0b3" exitCode=0
May 26 16:15:32.503392 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:32.503375    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk" event={"ID":"f70d2026-ed8e-48b4-a081-a0bdffa145d3","Type":"ContainerDied","Data":"19916178039f04b09b71add91571a8d62db99c6952186c23b4bab32488aec0b3"}
May 26 16:15:36.267199 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:36.267157    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk" podUID="f70d2026-ed8e-48b4-a081-a0bdffa145d3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
May 26 16:15:36.272550 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:36.272516    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk" podUID="f70d2026-ed8e-48b4-a081-a0bdffa145d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:5000: connect: connection refused"
May 26 16:15:36.272870 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:36.272848    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk" podUID="f70d2026-ed8e-48b4-a081-a0bdffa145d3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:15:38.511098 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:38.511070    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-7bgsr"
May 26 16:15:41.267340 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:41.267288    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk" podUID="f70d2026-ed8e-48b4-a081-a0bdffa145d3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
May 26 16:15:41.267796 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:41.267455    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk"
May 26 16:15:46.267584 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:46.267533    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk" podUID="f70d2026-ed8e-48b4-a081-a0bdffa145d3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
May 26 16:15:46.271853 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:46.271818    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk" podUID="f70d2026-ed8e-48b4-a081-a0bdffa145d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:5000: connect: connection refused"
May 26 16:15:46.272154 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:46.272124    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk" podUID="f70d2026-ed8e-48b4-a081-a0bdffa145d3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:15:48.069941 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:48.069905    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv"]
May 26 16:15:48.073212 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:48.073187    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv"
May 26 16:15:48.075158 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:48.075130    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
May 26 16:15:48.075360 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:48.075336    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
May 26 16:15:48.083267 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:48.083242    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv"]
May 26 16:15:48.099338 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:48.099309    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnzps\" (UniqueName: \"kubernetes.io/projected/0463102e-e1e1-4cc6-9ff6-fd058cf08130-kube-api-access-hnzps\") pod \"isvc-logger-predictor-6d94fc7bdd-jxjrv\" (UID: \"0463102e-e1e1-4cc6-9ff6-fd058cf08130\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv"
May 26 16:15:48.099477 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:48.099346    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-logger-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0463102e-e1e1-4cc6-9ff6-fd058cf08130-isvc-logger-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-6d94fc7bdd-jxjrv\" (UID: \"0463102e-e1e1-4cc6-9ff6-fd058cf08130\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv"
May 26 16:15:48.099477 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:48.099367    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0463102e-e1e1-4cc6-9ff6-fd058cf08130-proxy-tls\") pod \"isvc-logger-predictor-6d94fc7bdd-jxjrv\" (UID: \"0463102e-e1e1-4cc6-9ff6-fd058cf08130\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv"
May 26 16:15:48.099477 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:48.099409    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0463102e-e1e1-4cc6-9ff6-fd058cf08130-kserve-provision-location\") pod \"isvc-logger-predictor-6d94fc7bdd-jxjrv\" (UID: \"0463102e-e1e1-4cc6-9ff6-fd058cf08130\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv"
May 26 16:15:48.200308 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:48.200277    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0463102e-e1e1-4cc6-9ff6-fd058cf08130-kserve-provision-location\") pod \"isvc-logger-predictor-6d94fc7bdd-jxjrv\" (UID: \"0463102e-e1e1-4cc6-9ff6-fd058cf08130\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv"
May 26 16:15:48.200480 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:48.200339    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hnzps\" (UniqueName: \"kubernetes.io/projected/0463102e-e1e1-4cc6-9ff6-fd058cf08130-kube-api-access-hnzps\") pod \"isvc-logger-predictor-6d94fc7bdd-jxjrv\" (UID: \"0463102e-e1e1-4cc6-9ff6-fd058cf08130\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv"
May 26 16:15:48.200480 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:48.200362    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-logger-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0463102e-e1e1-4cc6-9ff6-fd058cf08130-isvc-logger-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-6d94fc7bdd-jxjrv\" (UID: \"0463102e-e1e1-4cc6-9ff6-fd058cf08130\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv"
May 26 16:15:48.200480 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:48.200384    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0463102e-e1e1-4cc6-9ff6-fd058cf08130-proxy-tls\") pod \"isvc-logger-predictor-6d94fc7bdd-jxjrv\" (UID: \"0463102e-e1e1-4cc6-9ff6-fd058cf08130\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv"
May 26 16:15:48.200674 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:15:48.200484    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-predictor-serving-cert: secret "isvc-logger-predictor-serving-cert" not found
May 26 16:15:48.200674 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:15:48.200546    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0463102e-e1e1-4cc6-9ff6-fd058cf08130-proxy-tls podName:0463102e-e1e1-4cc6-9ff6-fd058cf08130 nodeName:}" failed. No retries permitted until 2026-05-26 16:15:48.700528252 +0000 UTC m=+865.176255596 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0463102e-e1e1-4cc6-9ff6-fd058cf08130-proxy-tls") pod "isvc-logger-predictor-6d94fc7bdd-jxjrv" (UID: "0463102e-e1e1-4cc6-9ff6-fd058cf08130") : secret "isvc-logger-predictor-serving-cert" not found
May 26 16:15:48.200754 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:48.200730    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0463102e-e1e1-4cc6-9ff6-fd058cf08130-kserve-provision-location\") pod \"isvc-logger-predictor-6d94fc7bdd-jxjrv\" (UID: \"0463102e-e1e1-4cc6-9ff6-fd058cf08130\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv"
May 26 16:15:48.201086 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:48.201068    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-logger-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0463102e-e1e1-4cc6-9ff6-fd058cf08130-isvc-logger-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-6d94fc7bdd-jxjrv\" (UID: \"0463102e-e1e1-4cc6-9ff6-fd058cf08130\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv"
May 26 16:15:48.208509 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:48.208481    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnzps\" (UniqueName: \"kubernetes.io/projected/0463102e-e1e1-4cc6-9ff6-fd058cf08130-kube-api-access-hnzps\") pod \"isvc-logger-predictor-6d94fc7bdd-jxjrv\" (UID: \"0463102e-e1e1-4cc6-9ff6-fd058cf08130\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv"
May 26 16:15:48.704569 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:48.704529    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0463102e-e1e1-4cc6-9ff6-fd058cf08130-proxy-tls\") pod \"isvc-logger-predictor-6d94fc7bdd-jxjrv\" (UID: \"0463102e-e1e1-4cc6-9ff6-fd058cf08130\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv"
May 26 16:15:48.706938 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:48.706915    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0463102e-e1e1-4cc6-9ff6-fd058cf08130-proxy-tls\") pod \"isvc-logger-predictor-6d94fc7bdd-jxjrv\" (UID: \"0463102e-e1e1-4cc6-9ff6-fd058cf08130\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv"
May 26 16:15:48.984939 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:48.984848    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv"
May 26 16:15:49.113919 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:49.113870    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv"]
May 26 16:15:49.116525 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:15:49.116489    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0463102e_e1e1_4cc6_9ff6_fd058cf08130.slice/crio-5b50d1a44a16359a62d8b4d1832e5d99cd9dc0fc9e8300f0c7d823c71abb0c6d WatchSource:0}: Error finding container 5b50d1a44a16359a62d8b4d1832e5d99cd9dc0fc9e8300f0c7d823c71abb0c6d: Status 404 returned error can't find the container with id 5b50d1a44a16359a62d8b4d1832e5d99cd9dc0fc9e8300f0c7d823c71abb0c6d
May 26 16:15:49.563043 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:49.563008    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv" event={"ID":"0463102e-e1e1-4cc6-9ff6-fd058cf08130","Type":"ContainerStarted","Data":"29b3c777e67a39b3983b8cc39c026ec22d922dd50f949f4725b889eb42fee277"}
May 26 16:15:49.563207 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:49.563054    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv" event={"ID":"0463102e-e1e1-4cc6-9ff6-fd058cf08130","Type":"ContainerStarted","Data":"5b50d1a44a16359a62d8b4d1832e5d99cd9dc0fc9e8300f0c7d823c71abb0c6d"}
May 26 16:15:51.268053 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:51.268014    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk" podUID="f70d2026-ed8e-48b4-a081-a0bdffa145d3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
May 26 16:15:53.577448 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:53.577412    2572 generic.go:358] "Generic (PLEG): container finished" podID="0463102e-e1e1-4cc6-9ff6-fd058cf08130" containerID="29b3c777e67a39b3983b8cc39c026ec22d922dd50f949f4725b889eb42fee277" exitCode=0
May 26 16:15:53.577844 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:53.577463    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv" event={"ID":"0463102e-e1e1-4cc6-9ff6-fd058cf08130","Type":"ContainerDied","Data":"29b3c777e67a39b3983b8cc39c026ec22d922dd50f949f4725b889eb42fee277"}
May 26 16:15:54.582475 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:54.582439    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv" event={"ID":"0463102e-e1e1-4cc6-9ff6-fd058cf08130","Type":"ContainerStarted","Data":"43e430ae16d1e0b6a3ae08019d0101914b55184ecc86921939f254ba47874769"}
May 26 16:15:54.582475 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:54.582476    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv" event={"ID":"0463102e-e1e1-4cc6-9ff6-fd058cf08130","Type":"ContainerStarted","Data":"cb0b36f1f228f4386a75d0bb980fef5ed9c93b6b0b3919a35b0838fca700ce5b"}
May 26 16:15:54.582982 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:54.582486    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv" event={"ID":"0463102e-e1e1-4cc6-9ff6-fd058cf08130","Type":"ContainerStarted","Data":"539ed5b2609581212f3a707481623949956534a06bd2e9f866aadfaade195127"}
May 26 16:15:54.582982 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:54.582879    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv"
May 26 16:15:54.582982 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:54.582905    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv"
May 26 16:15:54.582982 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:54.582918    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv"
May 26 16:15:54.584312 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:54.584270    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv" podUID="0463102e-e1e1-4cc6-9ff6-fd058cf08130" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
May 26 16:15:54.584879 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:54.584855    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv" podUID="0463102e-e1e1-4cc6-9ff6-fd058cf08130" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:15:54.601650 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:54.601588    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv" podStartSLOduration=6.6015762989999995 podStartE2EDuration="6.601576299s" podCreationTimestamp="2026-05-26 16:15:48 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:15:54.600139408 +0000 UTC m=+871.075866760" watchObservedRunningTime="2026-05-26 16:15:54.601576299 +0000 UTC m=+871.077303648"
May 26 16:15:55.585258 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:55.585211    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv" podUID="0463102e-e1e1-4cc6-9ff6-fd058cf08130" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
May 26 16:15:55.585764 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:55.585740    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv" podUID="0463102e-e1e1-4cc6-9ff6-fd058cf08130" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:15:56.267256 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:56.267213    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk" podUID="f70d2026-ed8e-48b4-a081-a0bdffa145d3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
May 26 16:15:56.271727 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:56.271700    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk" podUID="f70d2026-ed8e-48b4-a081-a0bdffa145d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:5000: connect: connection refused"
May 26 16:15:56.271830 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:56.271816    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk"
May 26 16:15:56.272009 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:56.271990    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk" podUID="f70d2026-ed8e-48b4-a081-a0bdffa145d3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:15:56.272077 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:56.272067    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk"
May 26 16:15:58.095890 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:58.095860    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk"
May 26 16:15:58.177306 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:58.177271    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f70d2026-ed8e-48b4-a081-a0bdffa145d3-kserve-provision-location\") pod \"f70d2026-ed8e-48b4-a081-a0bdffa145d3\" (UID: \"f70d2026-ed8e-48b4-a081-a0bdffa145d3\") "
May 26 16:15:58.177306 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:58.177313    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f70d2026-ed8e-48b4-a081-a0bdffa145d3-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\") pod \"f70d2026-ed8e-48b4-a081-a0bdffa145d3\" (UID: \"f70d2026-ed8e-48b4-a081-a0bdffa145d3\") "
May 26 16:15:58.177528 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:58.177336    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vk5ls\" (UniqueName: \"kubernetes.io/projected/f70d2026-ed8e-48b4-a081-a0bdffa145d3-kube-api-access-vk5ls\") pod \"f70d2026-ed8e-48b4-a081-a0bdffa145d3\" (UID: \"f70d2026-ed8e-48b4-a081-a0bdffa145d3\") "
May 26 16:15:58.177528 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:58.177395    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f70d2026-ed8e-48b4-a081-a0bdffa145d3-proxy-tls\") pod \"f70d2026-ed8e-48b4-a081-a0bdffa145d3\" (UID: \"f70d2026-ed8e-48b4-a081-a0bdffa145d3\") "
May 26 16:15:58.177717 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:58.177689    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f70d2026-ed8e-48b4-a081-a0bdffa145d3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f70d2026-ed8e-48b4-a081-a0bdffa145d3" (UID: "f70d2026-ed8e-48b4-a081-a0bdffa145d3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:15:58.177767 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:58.177710    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f70d2026-ed8e-48b4-a081-a0bdffa145d3-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config") pod "f70d2026-ed8e-48b4-a081-a0bdffa145d3" (UID: "f70d2026-ed8e-48b4-a081-a0bdffa145d3"). InnerVolumeSpecName "isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:15:58.179483 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:58.179457    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f70d2026-ed8e-48b4-a081-a0bdffa145d3-kube-api-access-vk5ls" (OuterVolumeSpecName: "kube-api-access-vk5ls") pod "f70d2026-ed8e-48b4-a081-a0bdffa145d3" (UID: "f70d2026-ed8e-48b4-a081-a0bdffa145d3"). InnerVolumeSpecName "kube-api-access-vk5ls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:15:58.179550 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:58.179470    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f70d2026-ed8e-48b4-a081-a0bdffa145d3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f70d2026-ed8e-48b4-a081-a0bdffa145d3" (UID: "f70d2026-ed8e-48b4-a081-a0bdffa145d3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:15:58.278825 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:58.278725    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f70d2026-ed8e-48b4-a081-a0bdffa145d3-proxy-tls\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:15:58.278825 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:58.278769    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f70d2026-ed8e-48b4-a081-a0bdffa145d3-kserve-provision-location\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:15:58.278825 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:58.278780    2572 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f70d2026-ed8e-48b4-a081-a0bdffa145d3-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:15:58.278825 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:58.278791    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk5ls\" (UniqueName: \"kubernetes.io/projected/f70d2026-ed8e-48b4-a081-a0bdffa145d3-kube-api-access-vk5ls\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:15:58.595981 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:58.595892    2572 generic.go:358] "Generic (PLEG): container finished" podID="f70d2026-ed8e-48b4-a081-a0bdffa145d3" containerID="1039cbe48396eb20f4efc882d095ec7a87e8a98978d43f4964ff5af584539891" exitCode=0
May 26 16:15:58.596167 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:58.595980    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk"
May 26 16:15:58.596167 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:58.595979    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk" event={"ID":"f70d2026-ed8e-48b4-a081-a0bdffa145d3","Type":"ContainerDied","Data":"1039cbe48396eb20f4efc882d095ec7a87e8a98978d43f4964ff5af584539891"}
May 26 16:15:58.596167 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:58.596020    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk" event={"ID":"f70d2026-ed8e-48b4-a081-a0bdffa145d3","Type":"ContainerDied","Data":"d28b01e59563c7d24f93e807c98c7d3e00e7dea8d580ac0e18a45b908c88ccc9"}
May 26 16:15:58.596167 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:58.596035    2572 scope.go:117] "RemoveContainer" containerID="1039cbe48396eb20f4efc882d095ec7a87e8a98978d43f4964ff5af584539891"
May 26 16:15:58.604501 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:58.604484    2572 scope.go:117] "RemoveContainer" containerID="74a3be64103435df4f037b00bf18d9531723f2c9dc01a97ce8c225800ce3af68"
May 26 16:15:58.611502 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:58.611485    2572 scope.go:117] "RemoveContainer" containerID="19916178039f04b09b71add91571a8d62db99c6952186c23b4bab32488aec0b3"
May 26 16:15:58.615984 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:58.615962    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk"]
May 26 16:15:58.618809 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:58.618787    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-755c4698cb-wh8gk"]
May 26 16:15:58.619802 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:58.619787    2572 scope.go:117] "RemoveContainer" containerID="6b5633cd7390dde935bd1bb8cc74f40836ed44a9e0b7f041da19eb2372db884d"
May 26 16:15:58.627181 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:58.627163    2572 scope.go:117] "RemoveContainer" containerID="1039cbe48396eb20f4efc882d095ec7a87e8a98978d43f4964ff5af584539891"
May 26 16:15:58.627431 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:15:58.627411    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1039cbe48396eb20f4efc882d095ec7a87e8a98978d43f4964ff5af584539891\": container with ID starting with 1039cbe48396eb20f4efc882d095ec7a87e8a98978d43f4964ff5af584539891 not found: ID does not exist" containerID="1039cbe48396eb20f4efc882d095ec7a87e8a98978d43f4964ff5af584539891"
May 26 16:15:58.627477 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:58.627441    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1039cbe48396eb20f4efc882d095ec7a87e8a98978d43f4964ff5af584539891"} err="failed to get container status \"1039cbe48396eb20f4efc882d095ec7a87e8a98978d43f4964ff5af584539891\": rpc error: code = NotFound desc = could not find container \"1039cbe48396eb20f4efc882d095ec7a87e8a98978d43f4964ff5af584539891\": container with ID starting with 1039cbe48396eb20f4efc882d095ec7a87e8a98978d43f4964ff5af584539891 not found: ID does not exist"
May 26 16:15:58.627477 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:58.627458    2572 scope.go:117] "RemoveContainer" containerID="74a3be64103435df4f037b00bf18d9531723f2c9dc01a97ce8c225800ce3af68"
May 26 16:15:58.627749 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:15:58.627731    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74a3be64103435df4f037b00bf18d9531723f2c9dc01a97ce8c225800ce3af68\": container with ID starting with 74a3be64103435df4f037b00bf18d9531723f2c9dc01a97ce8c225800ce3af68 not found: ID does not exist" containerID="74a3be64103435df4f037b00bf18d9531723f2c9dc01a97ce8c225800ce3af68"
May 26 16:15:58.627798 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:58.627756    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74a3be64103435df4f037b00bf18d9531723f2c9dc01a97ce8c225800ce3af68"} err="failed to get container status \"74a3be64103435df4f037b00bf18d9531723f2c9dc01a97ce8c225800ce3af68\": rpc error: code = NotFound desc = could not find container \"74a3be64103435df4f037b00bf18d9531723f2c9dc01a97ce8c225800ce3af68\": container with ID starting with 74a3be64103435df4f037b00bf18d9531723f2c9dc01a97ce8c225800ce3af68 not found: ID does not exist"
May 26 16:15:58.627798 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:58.627772    2572 scope.go:117] "RemoveContainer" containerID="19916178039f04b09b71add91571a8d62db99c6952186c23b4bab32488aec0b3"
May 26 16:15:58.627994 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:15:58.627975    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19916178039f04b09b71add91571a8d62db99c6952186c23b4bab32488aec0b3\": container with ID starting with 19916178039f04b09b71add91571a8d62db99c6952186c23b4bab32488aec0b3 not found: ID does not exist" containerID="19916178039f04b09b71add91571a8d62db99c6952186c23b4bab32488aec0b3"
May 26 16:15:58.628034 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:58.627999    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19916178039f04b09b71add91571a8d62db99c6952186c23b4bab32488aec0b3"} err="failed to get container status \"19916178039f04b09b71add91571a8d62db99c6952186c23b4bab32488aec0b3\": rpc error: code = NotFound desc = could not find container \"19916178039f04b09b71add91571a8d62db99c6952186c23b4bab32488aec0b3\": container with ID starting with 19916178039f04b09b71add91571a8d62db99c6952186c23b4bab32488aec0b3 not found: ID does not exist"
May 26 16:15:58.628034 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:58.628016    2572 scope.go:117] "RemoveContainer" containerID="6b5633cd7390dde935bd1bb8cc74f40836ed44a9e0b7f041da19eb2372db884d"
May 26 16:15:58.628247 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:15:58.628231    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b5633cd7390dde935bd1bb8cc74f40836ed44a9e0b7f041da19eb2372db884d\": container with ID starting with 6b5633cd7390dde935bd1bb8cc74f40836ed44a9e0b7f041da19eb2372db884d not found: ID does not exist" containerID="6b5633cd7390dde935bd1bb8cc74f40836ed44a9e0b7f041da19eb2372db884d"
May 26 16:15:58.628300 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:15:58.628249    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b5633cd7390dde935bd1bb8cc74f40836ed44a9e0b7f041da19eb2372db884d"} err="failed to get container status \"6b5633cd7390dde935bd1bb8cc74f40836ed44a9e0b7f041da19eb2372db884d\": rpc error: code = NotFound desc = could not find container \"6b5633cd7390dde935bd1bb8cc74f40836ed44a9e0b7f041da19eb2372db884d\": container with ID starting with 6b5633cd7390dde935bd1bb8cc74f40836ed44a9e0b7f041da19eb2372db884d not found: ID does not exist"
May 26 16:16:00.080822 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:16:00.080793    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f70d2026-ed8e-48b4-a081-a0bdffa145d3" path="/var/lib/kubelet/pods/f70d2026-ed8e-48b4-a081-a0bdffa145d3/volumes"
May 26 16:16:00.589436 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:16:00.589408    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv"
May 26 16:16:00.590014 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:16:00.589980    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv" podUID="0463102e-e1e1-4cc6-9ff6-fd058cf08130" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
May 26 16:16:00.590374 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:16:00.590353    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv" podUID="0463102e-e1e1-4cc6-9ff6-fd058cf08130" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:16:10.590477 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:16:10.590427    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv" podUID="0463102e-e1e1-4cc6-9ff6-fd058cf08130" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
May 26 16:16:10.591054 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:16:10.590839    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv" podUID="0463102e-e1e1-4cc6-9ff6-fd058cf08130" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:16:20.590079 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:16:20.590000    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv" podUID="0463102e-e1e1-4cc6-9ff6-fd058cf08130" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
May 26 16:16:20.590533 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:16:20.590503    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv" podUID="0463102e-e1e1-4cc6-9ff6-fd058cf08130" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:16:30.590098 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:16:30.590048    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv" podUID="0463102e-e1e1-4cc6-9ff6-fd058cf08130" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
May 26 16:16:30.590516 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:16:30.590339    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv" podUID="0463102e-e1e1-4cc6-9ff6-fd058cf08130" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:16:40.590271 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:16:40.590228    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv" podUID="0463102e-e1e1-4cc6-9ff6-fd058cf08130" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
May 26 16:16:40.590799 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:16:40.590773    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv" podUID="0463102e-e1e1-4cc6-9ff6-fd058cf08130" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:16:50.589908 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:16:50.589853    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv" podUID="0463102e-e1e1-4cc6-9ff6-fd058cf08130" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
May 26 16:16:50.590447 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:16:50.590315    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv" podUID="0463102e-e1e1-4cc6-9ff6-fd058cf08130" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:17:00.591298 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:00.591269    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv"
May 26 16:17:00.591779 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:00.591635    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv"
May 26 16:17:13.093304 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.093268    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-c7d86bcbd-7bgsr_8f46a336-6162-4e1a-9ee8-86c83137f04b/kserve-container/0.log"
May 26 16:17:13.264372 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.264337    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv"]
May 26 16:17:13.265010 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.264820    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv" podUID="0463102e-e1e1-4cc6-9ff6-fd058cf08130" containerName="kserve-container" containerID="cri-o://539ed5b2609581212f3a707481623949956534a06bd2e9f866aadfaade195127" gracePeriod=30
May 26 16:17:13.265010 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.264853    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv" podUID="0463102e-e1e1-4cc6-9ff6-fd058cf08130" containerName="kube-rbac-proxy" containerID="cri-o://cb0b36f1f228f4386a75d0bb980fef5ed9c93b6b0b3919a35b0838fca700ce5b" gracePeriod=30
May 26 16:17:13.265010 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.264851    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv" podUID="0463102e-e1e1-4cc6-9ff6-fd058cf08130" containerName="agent" containerID="cri-o://43e430ae16d1e0b6a3ae08019d0101914b55184ecc86921939f254ba47874769" gracePeriod=30
May 26 16:17:13.308467 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.308432    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-r64vj"]
May 26 16:17:13.308752 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.308739    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f70d2026-ed8e-48b4-a081-a0bdffa145d3" containerName="kserve-container"
May 26 16:17:13.308800 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.308754    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f70d2026-ed8e-48b4-a081-a0bdffa145d3" containerName="kserve-container"
May 26 16:17:13.308800 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.308768    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f70d2026-ed8e-48b4-a081-a0bdffa145d3" containerName="agent"
May 26 16:17:13.308800 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.308774    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f70d2026-ed8e-48b4-a081-a0bdffa145d3" containerName="agent"
May 26 16:17:13.308800 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.308780    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f70d2026-ed8e-48b4-a081-a0bdffa145d3" containerName="storage-initializer"
May 26 16:17:13.308800 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.308785    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f70d2026-ed8e-48b4-a081-a0bdffa145d3" containerName="storage-initializer"
May 26 16:17:13.308800 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.308799    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f70d2026-ed8e-48b4-a081-a0bdffa145d3" containerName="kube-rbac-proxy"
May 26 16:17:13.308978 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.308804    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f70d2026-ed8e-48b4-a081-a0bdffa145d3" containerName="kube-rbac-proxy"
May 26 16:17:13.308978 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.308851    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="f70d2026-ed8e-48b4-a081-a0bdffa145d3" containerName="kserve-container"
May 26 16:17:13.308978 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.308860    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="f70d2026-ed8e-48b4-a081-a0bdffa145d3" containerName="kube-rbac-proxy"
May 26 16:17:13.308978 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.308867    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="f70d2026-ed8e-48b4-a081-a0bdffa145d3" containerName="agent"
May 26 16:17:13.311853 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.311832    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-r64vj"
May 26 16:17:13.314032 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.314012    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-predictor-serving-cert\""
May 26 16:17:13.314273 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.314261    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
May 26 16:17:13.323241 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.323191    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-r64vj"]
May 26 16:17:13.352973 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.352879    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxgbj\" (UniqueName: \"kubernetes.io/projected/4976badd-ed26-4af2-9205-21260aa36ef3-kube-api-access-zxgbj\") pod \"isvc-lightgbm-predictor-bdf964bd-r64vj\" (UID: \"4976badd-ed26-4af2-9205-21260aa36ef3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-r64vj"
May 26 16:17:13.352973 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.352924    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4976badd-ed26-4af2-9205-21260aa36ef3-proxy-tls\") pod \"isvc-lightgbm-predictor-bdf964bd-r64vj\" (UID: \"4976badd-ed26-4af2-9205-21260aa36ef3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-r64vj"
May 26 16:17:13.352973 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.352954    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4976badd-ed26-4af2-9205-21260aa36ef3-isvc-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-bdf964bd-r64vj\" (UID: \"4976badd-ed26-4af2-9205-21260aa36ef3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-r64vj"
May 26 16:17:13.353212 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.353106    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4976badd-ed26-4af2-9205-21260aa36ef3-kserve-provision-location\") pod \"isvc-lightgbm-predictor-bdf964bd-r64vj\" (UID: \"4976badd-ed26-4af2-9205-21260aa36ef3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-r64vj"
May 26 16:17:13.376882 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.376853    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-7bgsr"]
May 26 16:17:13.377162 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.377133    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-7bgsr" podUID="8f46a336-6162-4e1a-9ee8-86c83137f04b" containerName="kserve-container" containerID="cri-o://7bec13a917e2ccef6b7d8f37c76c8b1f7602373c6340470bcdc42d84323fa659" gracePeriod=30
May 26 16:17:13.377303 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.377158    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-7bgsr" podUID="8f46a336-6162-4e1a-9ee8-86c83137f04b" containerName="kube-rbac-proxy" containerID="cri-o://ae43ba558c83cc4dbc9239834aa30cb5aa06e4e0abe4fb5f987d465ee1ba2b36" gracePeriod=30
May 26 16:17:13.454186 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.454150    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4976badd-ed26-4af2-9205-21260aa36ef3-kserve-provision-location\") pod \"isvc-lightgbm-predictor-bdf964bd-r64vj\" (UID: \"4976badd-ed26-4af2-9205-21260aa36ef3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-r64vj"
May 26 16:17:13.454364 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.454196    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zxgbj\" (UniqueName: \"kubernetes.io/projected/4976badd-ed26-4af2-9205-21260aa36ef3-kube-api-access-zxgbj\") pod \"isvc-lightgbm-predictor-bdf964bd-r64vj\" (UID: \"4976badd-ed26-4af2-9205-21260aa36ef3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-r64vj"
May 26 16:17:13.454364 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.454216    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4976badd-ed26-4af2-9205-21260aa36ef3-proxy-tls\") pod \"isvc-lightgbm-predictor-bdf964bd-r64vj\" (UID: \"4976badd-ed26-4af2-9205-21260aa36ef3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-r64vj"
May 26 16:17:13.454364 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.454235    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4976badd-ed26-4af2-9205-21260aa36ef3-isvc-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-bdf964bd-r64vj\" (UID: \"4976badd-ed26-4af2-9205-21260aa36ef3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-r64vj"
May 26 16:17:13.454486 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:17:13.454367    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-predictor-serving-cert: secret "isvc-lightgbm-predictor-serving-cert" not found
May 26 16:17:13.454486 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:17:13.454433    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4976badd-ed26-4af2-9205-21260aa36ef3-proxy-tls podName:4976badd-ed26-4af2-9205-21260aa36ef3 nodeName:}" failed. No retries permitted until 2026-05-26 16:17:13.954411632 +0000 UTC m=+950.430138971 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4976badd-ed26-4af2-9205-21260aa36ef3-proxy-tls") pod "isvc-lightgbm-predictor-bdf964bd-r64vj" (UID: "4976badd-ed26-4af2-9205-21260aa36ef3") : secret "isvc-lightgbm-predictor-serving-cert" not found
May 26 16:17:13.454566 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.454542    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4976badd-ed26-4af2-9205-21260aa36ef3-kserve-provision-location\") pod \"isvc-lightgbm-predictor-bdf964bd-r64vj\" (UID: \"4976badd-ed26-4af2-9205-21260aa36ef3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-r64vj"
May 26 16:17:13.454874 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.454857    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4976badd-ed26-4af2-9205-21260aa36ef3-isvc-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-bdf964bd-r64vj\" (UID: \"4976badd-ed26-4af2-9205-21260aa36ef3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-r64vj"
May 26 16:17:13.462389 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.462366    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxgbj\" (UniqueName: \"kubernetes.io/projected/4976badd-ed26-4af2-9205-21260aa36ef3-kube-api-access-zxgbj\") pod \"isvc-lightgbm-predictor-bdf964bd-r64vj\" (UID: \"4976badd-ed26-4af2-9205-21260aa36ef3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-r64vj"
May 26 16:17:13.507715 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.507676    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-7bgsr" podUID="8f46a336-6162-4e1a-9ee8-86c83137f04b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
May 26 16:17:13.621427 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.621405    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-7bgsr"
May 26 16:17:13.655876 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.655841    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7fv8\" (UniqueName: \"kubernetes.io/projected/8f46a336-6162-4e1a-9ee8-86c83137f04b-kube-api-access-l7fv8\") pod \"8f46a336-6162-4e1a-9ee8-86c83137f04b\" (UID: \"8f46a336-6162-4e1a-9ee8-86c83137f04b\") "
May 26 16:17:13.656048 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.655901    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"message-dumper-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f46a336-6162-4e1a-9ee8-86c83137f04b-message-dumper-kube-rbac-proxy-sar-config\") pod \"8f46a336-6162-4e1a-9ee8-86c83137f04b\" (UID: \"8f46a336-6162-4e1a-9ee8-86c83137f04b\") "
May 26 16:17:13.656048 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.655958    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f46a336-6162-4e1a-9ee8-86c83137f04b-proxy-tls\") pod \"8f46a336-6162-4e1a-9ee8-86c83137f04b\" (UID: \"8f46a336-6162-4e1a-9ee8-86c83137f04b\") "
May 26 16:17:13.656238 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.656212    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f46a336-6162-4e1a-9ee8-86c83137f04b-message-dumper-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "message-dumper-kube-rbac-proxy-sar-config") pod "8f46a336-6162-4e1a-9ee8-86c83137f04b" (UID: "8f46a336-6162-4e1a-9ee8-86c83137f04b"). InnerVolumeSpecName "message-dumper-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:17:13.658001 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.657969    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f46a336-6162-4e1a-9ee8-86c83137f04b-kube-api-access-l7fv8" (OuterVolumeSpecName: "kube-api-access-l7fv8") pod "8f46a336-6162-4e1a-9ee8-86c83137f04b" (UID: "8f46a336-6162-4e1a-9ee8-86c83137f04b"). InnerVolumeSpecName "kube-api-access-l7fv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:17:13.658130 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.658054    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f46a336-6162-4e1a-9ee8-86c83137f04b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8f46a336-6162-4e1a-9ee8-86c83137f04b" (UID: "8f46a336-6162-4e1a-9ee8-86c83137f04b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:17:13.757333 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.757294    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f46a336-6162-4e1a-9ee8-86c83137f04b-proxy-tls\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:17:13.757333 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.757327    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7fv8\" (UniqueName: \"kubernetes.io/projected/8f46a336-6162-4e1a-9ee8-86c83137f04b-kube-api-access-l7fv8\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:17:13.757333 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.757338    2572 reconciler_common.go:299] "Volume detached for volume \"message-dumper-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f46a336-6162-4e1a-9ee8-86c83137f04b-message-dumper-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:17:13.803996 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.803964    2572 generic.go:358] "Generic (PLEG): container finished" podID="0463102e-e1e1-4cc6-9ff6-fd058cf08130" containerID="cb0b36f1f228f4386a75d0bb980fef5ed9c93b6b0b3919a35b0838fca700ce5b" exitCode=2
May 26 16:17:13.804152 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.804040    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv" event={"ID":"0463102e-e1e1-4cc6-9ff6-fd058cf08130","Type":"ContainerDied","Data":"cb0b36f1f228f4386a75d0bb980fef5ed9c93b6b0b3919a35b0838fca700ce5b"}
May 26 16:17:13.805312 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.805291    2572 generic.go:358] "Generic (PLEG): container finished" podID="8f46a336-6162-4e1a-9ee8-86c83137f04b" containerID="ae43ba558c83cc4dbc9239834aa30cb5aa06e4e0abe4fb5f987d465ee1ba2b36" exitCode=2
May 26 16:17:13.805312 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.805310    2572 generic.go:358] "Generic (PLEG): container finished" podID="8f46a336-6162-4e1a-9ee8-86c83137f04b" containerID="7bec13a917e2ccef6b7d8f37c76c8b1f7602373c6340470bcdc42d84323fa659" exitCode=2
May 26 16:17:13.805447 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.805365    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-7bgsr"
May 26 16:17:13.805447 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.805371    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-7bgsr" event={"ID":"8f46a336-6162-4e1a-9ee8-86c83137f04b","Type":"ContainerDied","Data":"ae43ba558c83cc4dbc9239834aa30cb5aa06e4e0abe4fb5f987d465ee1ba2b36"}
May 26 16:17:13.805447 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.805401    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-7bgsr" event={"ID":"8f46a336-6162-4e1a-9ee8-86c83137f04b","Type":"ContainerDied","Data":"7bec13a917e2ccef6b7d8f37c76c8b1f7602373c6340470bcdc42d84323fa659"}
May 26 16:17:13.805447 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.805413    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-7bgsr" event={"ID":"8f46a336-6162-4e1a-9ee8-86c83137f04b","Type":"ContainerDied","Data":"a30a4fd7d77d45d8a7ac7f17b89ec44cfbf63e4adfb6982b55b71eebe7545365"}
May 26 16:17:13.805447 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.805427    2572 scope.go:117] "RemoveContainer" containerID="ae43ba558c83cc4dbc9239834aa30cb5aa06e4e0abe4fb5f987d465ee1ba2b36"
May 26 16:17:13.813521 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.813506    2572 scope.go:117] "RemoveContainer" containerID="7bec13a917e2ccef6b7d8f37c76c8b1f7602373c6340470bcdc42d84323fa659"
May 26 16:17:13.820902 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.820885    2572 scope.go:117] "RemoveContainer" containerID="ae43ba558c83cc4dbc9239834aa30cb5aa06e4e0abe4fb5f987d465ee1ba2b36"
May 26 16:17:13.821144 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:17:13.821128    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae43ba558c83cc4dbc9239834aa30cb5aa06e4e0abe4fb5f987d465ee1ba2b36\": container with ID starting with ae43ba558c83cc4dbc9239834aa30cb5aa06e4e0abe4fb5f987d465ee1ba2b36 not found: ID does not exist" containerID="ae43ba558c83cc4dbc9239834aa30cb5aa06e4e0abe4fb5f987d465ee1ba2b36"
May 26 16:17:13.821194 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.821152    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae43ba558c83cc4dbc9239834aa30cb5aa06e4e0abe4fb5f987d465ee1ba2b36"} err="failed to get container status \"ae43ba558c83cc4dbc9239834aa30cb5aa06e4e0abe4fb5f987d465ee1ba2b36\": rpc error: code = NotFound desc = could not find container \"ae43ba558c83cc4dbc9239834aa30cb5aa06e4e0abe4fb5f987d465ee1ba2b36\": container with ID starting with ae43ba558c83cc4dbc9239834aa30cb5aa06e4e0abe4fb5f987d465ee1ba2b36 not found: ID does not exist"
May 26 16:17:13.821194 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.821169    2572 scope.go:117] "RemoveContainer" containerID="7bec13a917e2ccef6b7d8f37c76c8b1f7602373c6340470bcdc42d84323fa659"
May 26 16:17:13.821348 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:17:13.821333    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bec13a917e2ccef6b7d8f37c76c8b1f7602373c6340470bcdc42d84323fa659\": container with ID starting with 7bec13a917e2ccef6b7d8f37c76c8b1f7602373c6340470bcdc42d84323fa659 not found: ID does not exist" containerID="7bec13a917e2ccef6b7d8f37c76c8b1f7602373c6340470bcdc42d84323fa659"
May 26 16:17:13.821387 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.821353    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bec13a917e2ccef6b7d8f37c76c8b1f7602373c6340470bcdc42d84323fa659"} err="failed to get container status \"7bec13a917e2ccef6b7d8f37c76c8b1f7602373c6340470bcdc42d84323fa659\": rpc error: code = NotFound desc = could not find container \"7bec13a917e2ccef6b7d8f37c76c8b1f7602373c6340470bcdc42d84323fa659\": container with ID starting with 7bec13a917e2ccef6b7d8f37c76c8b1f7602373c6340470bcdc42d84323fa659 not found: ID does not exist"
May 26 16:17:13.821387 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.821367    2572 scope.go:117] "RemoveContainer" containerID="ae43ba558c83cc4dbc9239834aa30cb5aa06e4e0abe4fb5f987d465ee1ba2b36"
May 26 16:17:13.821641 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.821617    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae43ba558c83cc4dbc9239834aa30cb5aa06e4e0abe4fb5f987d465ee1ba2b36"} err="failed to get container status \"ae43ba558c83cc4dbc9239834aa30cb5aa06e4e0abe4fb5f987d465ee1ba2b36\": rpc error: code = NotFound desc = could not find container \"ae43ba558c83cc4dbc9239834aa30cb5aa06e4e0abe4fb5f987d465ee1ba2b36\": container with ID starting with ae43ba558c83cc4dbc9239834aa30cb5aa06e4e0abe4fb5f987d465ee1ba2b36 not found: ID does not exist"
May 26 16:17:13.821692 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.821646    2572 scope.go:117] "RemoveContainer" containerID="7bec13a917e2ccef6b7d8f37c76c8b1f7602373c6340470bcdc42d84323fa659"
May 26 16:17:13.821869 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.821846    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bec13a917e2ccef6b7d8f37c76c8b1f7602373c6340470bcdc42d84323fa659"} err="failed to get container status \"7bec13a917e2ccef6b7d8f37c76c8b1f7602373c6340470bcdc42d84323fa659\": rpc error: code = NotFound desc = could not find container \"7bec13a917e2ccef6b7d8f37c76c8b1f7602373c6340470bcdc42d84323fa659\": container with ID starting with 7bec13a917e2ccef6b7d8f37c76c8b1f7602373c6340470bcdc42d84323fa659 not found: ID does not exist"
May 26 16:17:13.828922 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.826708    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-7bgsr"]
May 26 16:17:13.831676 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.831652    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-c7d86bcbd-7bgsr"]
May 26 16:17:13.959527 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.959485    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4976badd-ed26-4af2-9205-21260aa36ef3-proxy-tls\") pod \"isvc-lightgbm-predictor-bdf964bd-r64vj\" (UID: \"4976badd-ed26-4af2-9205-21260aa36ef3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-r64vj"
May 26 16:17:13.961911 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:13.961874    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4976badd-ed26-4af2-9205-21260aa36ef3-proxy-tls\") pod \"isvc-lightgbm-predictor-bdf964bd-r64vj\" (UID: \"4976badd-ed26-4af2-9205-21260aa36ef3\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-r64vj"
May 26 16:17:14.079799 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:14.079767    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f46a336-6162-4e1a-9ee8-86c83137f04b" path="/var/lib/kubelet/pods/8f46a336-6162-4e1a-9ee8-86c83137f04b/volumes"
May 26 16:17:14.223953 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:14.223867    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-r64vj"
May 26 16:17:14.348169 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:14.348102    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-r64vj"]
May 26 16:17:14.351547 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:17:14.351519    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4976badd_ed26_4af2_9205_21260aa36ef3.slice/crio-d7c5752492ff5276860888bacbd62b2db3b04a835400914b4e16e15451e5dbfb WatchSource:0}: Error finding container d7c5752492ff5276860888bacbd62b2db3b04a835400914b4e16e15451e5dbfb: Status 404 returned error can't find the container with id d7c5752492ff5276860888bacbd62b2db3b04a835400914b4e16e15451e5dbfb
May 26 16:17:14.810217 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:14.810178    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-r64vj" event={"ID":"4976badd-ed26-4af2-9205-21260aa36ef3","Type":"ContainerStarted","Data":"d8650754bbab970428dc753eb0771c734230ae432b6367821508e5f81bfc6737"}
May 26 16:17:14.810217 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:14.810221    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-r64vj" event={"ID":"4976badd-ed26-4af2-9205-21260aa36ef3","Type":"ContainerStarted","Data":"d7c5752492ff5276860888bacbd62b2db3b04a835400914b4e16e15451e5dbfb"}
May 26 16:17:15.585554 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:15.585510    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv" podUID="0463102e-e1e1-4cc6-9ff6-fd058cf08130" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
May 26 16:17:17.820225 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:17.820186    2572 generic.go:358] "Generic (PLEG): container finished" podID="0463102e-e1e1-4cc6-9ff6-fd058cf08130" containerID="539ed5b2609581212f3a707481623949956534a06bd2e9f866aadfaade195127" exitCode=0
May 26 16:17:17.820225 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:17.820228    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv" event={"ID":"0463102e-e1e1-4cc6-9ff6-fd058cf08130","Type":"ContainerDied","Data":"539ed5b2609581212f3a707481623949956534a06bd2e9f866aadfaade195127"}
May 26 16:17:18.824813 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:18.824778    2572 generic.go:358] "Generic (PLEG): container finished" podID="4976badd-ed26-4af2-9205-21260aa36ef3" containerID="d8650754bbab970428dc753eb0771c734230ae432b6367821508e5f81bfc6737" exitCode=0
May 26 16:17:18.825265 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:18.824830    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-r64vj" event={"ID":"4976badd-ed26-4af2-9205-21260aa36ef3","Type":"ContainerDied","Data":"d8650754bbab970428dc753eb0771c734230ae432b6367821508e5f81bfc6737"}
May 26 16:17:20.585852 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:20.585759    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv" podUID="0463102e-e1e1-4cc6-9ff6-fd058cf08130" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
May 26 16:17:20.590271 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:20.590203    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv" podUID="0463102e-e1e1-4cc6-9ff6-fd058cf08130" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
May 26 16:17:20.590723 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:20.590669    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv" podUID="0463102e-e1e1-4cc6-9ff6-fd058cf08130" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:17:25.586077 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:25.585983    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv" podUID="0463102e-e1e1-4cc6-9ff6-fd058cf08130" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
May 26 16:17:25.586419 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:25.586142    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv"
May 26 16:17:25.849737 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:25.849659    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-r64vj" event={"ID":"4976badd-ed26-4af2-9205-21260aa36ef3","Type":"ContainerStarted","Data":"2af4d3f23db76912f20c120e32cbd13729baf4ee563a3f9e72d029b34254bc4e"}
May 26 16:17:25.849737 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:25.849699    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-r64vj" event={"ID":"4976badd-ed26-4af2-9205-21260aa36ef3","Type":"ContainerStarted","Data":"31798316da8fe38391aa5643cebecfb222b524fb11fb1bad74fcb8ab7b588ea1"}
May 26 16:17:25.849923 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:25.849904    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-r64vj"
May 26 16:17:25.868347 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:25.868286    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-r64vj" podStartSLOduration=6.536703688 podStartE2EDuration="12.868266397s" podCreationTimestamp="2026-05-26 16:17:13 +0000 UTC" firstStartedPulling="2026-05-26 16:17:18.826035739 +0000 UTC m=+955.301763067" lastFinishedPulling="2026-05-26 16:17:25.157598435 +0000 UTC m=+961.633325776" observedRunningTime="2026-05-26 16:17:25.866673104 +0000 UTC m=+962.342400455" watchObservedRunningTime="2026-05-26 16:17:25.868266397 +0000 UTC m=+962.343993748"
May 26 16:17:26.852850 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:26.852811    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-r64vj"
May 26 16:17:26.854142 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:26.854110    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-r64vj" podUID="4976badd-ed26-4af2-9205-21260aa36ef3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
May 26 16:17:27.855180 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:27.855147    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-r64vj" podUID="4976badd-ed26-4af2-9205-21260aa36ef3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
May 26 16:17:30.585783 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:30.585735    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv" podUID="0463102e-e1e1-4cc6-9ff6-fd058cf08130" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
May 26 16:17:30.590077 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:30.590047    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv" podUID="0463102e-e1e1-4cc6-9ff6-fd058cf08130" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
May 26 16:17:30.590439 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:30.590415    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv" podUID="0463102e-e1e1-4cc6-9ff6-fd058cf08130" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:17:32.859192 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:32.859160    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-r64vj"
May 26 16:17:32.859779 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:32.859750    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-r64vj" podUID="4976badd-ed26-4af2-9205-21260aa36ef3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
May 26 16:17:35.586031 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:35.585988    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv" podUID="0463102e-e1e1-4cc6-9ff6-fd058cf08130" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
May 26 16:17:40.585926 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:40.585867    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv" podUID="0463102e-e1e1-4cc6-9ff6-fd058cf08130" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
May 26 16:17:40.590256 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:40.590214    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv" podUID="0463102e-e1e1-4cc6-9ff6-fd058cf08130" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
May 26 16:17:40.590419 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:40.590396    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv"
May 26 16:17:40.590595 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:40.590572    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv" podUID="0463102e-e1e1-4cc6-9ff6-fd058cf08130" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:17:40.590700 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:40.590688    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv"
May 26 16:17:42.860008 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:42.859975    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-r64vj" podUID="4976badd-ed26-4af2-9205-21260aa36ef3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
May 26 16:17:43.441110 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:43.441083    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv"
May 26 16:17:43.507692 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:43.507654    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0463102e-e1e1-4cc6-9ff6-fd058cf08130-proxy-tls\") pod \"0463102e-e1e1-4cc6-9ff6-fd058cf08130\" (UID: \"0463102e-e1e1-4cc6-9ff6-fd058cf08130\") "
May 26 16:17:43.507904 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:43.507713    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0463102e-e1e1-4cc6-9ff6-fd058cf08130-kserve-provision-location\") pod \"0463102e-e1e1-4cc6-9ff6-fd058cf08130\" (UID: \"0463102e-e1e1-4cc6-9ff6-fd058cf08130\") "
May 26 16:17:43.507904 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:43.507770    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hnzps\" (UniqueName: \"kubernetes.io/projected/0463102e-e1e1-4cc6-9ff6-fd058cf08130-kube-api-access-hnzps\") pod \"0463102e-e1e1-4cc6-9ff6-fd058cf08130\" (UID: \"0463102e-e1e1-4cc6-9ff6-fd058cf08130\") "
May 26 16:17:43.507904 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:43.507812    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-logger-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0463102e-e1e1-4cc6-9ff6-fd058cf08130-isvc-logger-kube-rbac-proxy-sar-config\") pod \"0463102e-e1e1-4cc6-9ff6-fd058cf08130\" (UID: \"0463102e-e1e1-4cc6-9ff6-fd058cf08130\") "
May 26 16:17:43.508097 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:43.508075    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0463102e-e1e1-4cc6-9ff6-fd058cf08130-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0463102e-e1e1-4cc6-9ff6-fd058cf08130" (UID: "0463102e-e1e1-4cc6-9ff6-fd058cf08130"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:17:43.508208 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:43.508177    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0463102e-e1e1-4cc6-9ff6-fd058cf08130-isvc-logger-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-logger-kube-rbac-proxy-sar-config") pod "0463102e-e1e1-4cc6-9ff6-fd058cf08130" (UID: "0463102e-e1e1-4cc6-9ff6-fd058cf08130"). InnerVolumeSpecName "isvc-logger-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:17:43.509826 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:43.509804    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0463102e-e1e1-4cc6-9ff6-fd058cf08130-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0463102e-e1e1-4cc6-9ff6-fd058cf08130" (UID: "0463102e-e1e1-4cc6-9ff6-fd058cf08130"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:17:43.509920 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:43.509875    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0463102e-e1e1-4cc6-9ff6-fd058cf08130-kube-api-access-hnzps" (OuterVolumeSpecName: "kube-api-access-hnzps") pod "0463102e-e1e1-4cc6-9ff6-fd058cf08130" (UID: "0463102e-e1e1-4cc6-9ff6-fd058cf08130"). InnerVolumeSpecName "kube-api-access-hnzps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:17:43.608979 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:43.608903    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnzps\" (UniqueName: \"kubernetes.io/projected/0463102e-e1e1-4cc6-9ff6-fd058cf08130-kube-api-access-hnzps\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:17:43.608979 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:43.608932    2572 reconciler_common.go:299] "Volume detached for volume \"isvc-logger-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0463102e-e1e1-4cc6-9ff6-fd058cf08130-isvc-logger-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:17:43.608979 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:43.608943    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0463102e-e1e1-4cc6-9ff6-fd058cf08130-proxy-tls\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:17:43.608979 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:43.608953    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0463102e-e1e1-4cc6-9ff6-fd058cf08130-kserve-provision-location\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:17:43.901391 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:43.901359    2572 generic.go:358] "Generic (PLEG): container finished" podID="0463102e-e1e1-4cc6-9ff6-fd058cf08130" containerID="43e430ae16d1e0b6a3ae08019d0101914b55184ecc86921939f254ba47874769" exitCode=0
May 26 16:17:43.901772 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:43.901451    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv"
May 26 16:17:43.901772 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:43.901440    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv" event={"ID":"0463102e-e1e1-4cc6-9ff6-fd058cf08130","Type":"ContainerDied","Data":"43e430ae16d1e0b6a3ae08019d0101914b55184ecc86921939f254ba47874769"}
May 26 16:17:43.901772 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:43.901566    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv" event={"ID":"0463102e-e1e1-4cc6-9ff6-fd058cf08130","Type":"ContainerDied","Data":"5b50d1a44a16359a62d8b4d1832e5d99cd9dc0fc9e8300f0c7d823c71abb0c6d"}
May 26 16:17:43.901772 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:43.901589    2572 scope.go:117] "RemoveContainer" containerID="43e430ae16d1e0b6a3ae08019d0101914b55184ecc86921939f254ba47874769"
May 26 16:17:43.909346 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:43.909317    2572 scope.go:117] "RemoveContainer" containerID="cb0b36f1f228f4386a75d0bb980fef5ed9c93b6b0b3919a35b0838fca700ce5b"
May 26 16:17:43.916420 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:43.916402    2572 scope.go:117] "RemoveContainer" containerID="539ed5b2609581212f3a707481623949956534a06bd2e9f866aadfaade195127"
May 26 16:17:43.923101 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:43.923075    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv"]
May 26 16:17:43.925029 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:43.925014    2572 scope.go:117] "RemoveContainer" containerID="29b3c777e67a39b3983b8cc39c026ec22d922dd50f949f4725b889eb42fee277"
May 26 16:17:43.928365 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:43.928342    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6d94fc7bdd-jxjrv"]
May 26 16:17:43.932150 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:43.932128    2572 scope.go:117] "RemoveContainer" containerID="43e430ae16d1e0b6a3ae08019d0101914b55184ecc86921939f254ba47874769"
May 26 16:17:43.932410 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:17:43.932392    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43e430ae16d1e0b6a3ae08019d0101914b55184ecc86921939f254ba47874769\": container with ID starting with 43e430ae16d1e0b6a3ae08019d0101914b55184ecc86921939f254ba47874769 not found: ID does not exist" containerID="43e430ae16d1e0b6a3ae08019d0101914b55184ecc86921939f254ba47874769"
May 26 16:17:43.932474 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:43.932418    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43e430ae16d1e0b6a3ae08019d0101914b55184ecc86921939f254ba47874769"} err="failed to get container status \"43e430ae16d1e0b6a3ae08019d0101914b55184ecc86921939f254ba47874769\": rpc error: code = NotFound desc = could not find container \"43e430ae16d1e0b6a3ae08019d0101914b55184ecc86921939f254ba47874769\": container with ID starting with 43e430ae16d1e0b6a3ae08019d0101914b55184ecc86921939f254ba47874769 not found: ID does not exist"
May 26 16:17:43.932474 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:43.932438    2572 scope.go:117] "RemoveContainer" containerID="cb0b36f1f228f4386a75d0bb980fef5ed9c93b6b0b3919a35b0838fca700ce5b"
May 26 16:17:43.932679 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:17:43.932662    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb0b36f1f228f4386a75d0bb980fef5ed9c93b6b0b3919a35b0838fca700ce5b\": container with ID starting with cb0b36f1f228f4386a75d0bb980fef5ed9c93b6b0b3919a35b0838fca700ce5b not found: ID does not exist" containerID="cb0b36f1f228f4386a75d0bb980fef5ed9c93b6b0b3919a35b0838fca700ce5b"
May 26 16:17:43.932728 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:43.932685    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb0b36f1f228f4386a75d0bb980fef5ed9c93b6b0b3919a35b0838fca700ce5b"} err="failed to get container status \"cb0b36f1f228f4386a75d0bb980fef5ed9c93b6b0b3919a35b0838fca700ce5b\": rpc error: code = NotFound desc = could not find container \"cb0b36f1f228f4386a75d0bb980fef5ed9c93b6b0b3919a35b0838fca700ce5b\": container with ID starting with cb0b36f1f228f4386a75d0bb980fef5ed9c93b6b0b3919a35b0838fca700ce5b not found: ID does not exist"
May 26 16:17:43.932728 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:43.932703    2572 scope.go:117] "RemoveContainer" containerID="539ed5b2609581212f3a707481623949956534a06bd2e9f866aadfaade195127"
May 26 16:17:43.932910 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:17:43.932890    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"539ed5b2609581212f3a707481623949956534a06bd2e9f866aadfaade195127\": container with ID starting with 539ed5b2609581212f3a707481623949956534a06bd2e9f866aadfaade195127 not found: ID does not exist" containerID="539ed5b2609581212f3a707481623949956534a06bd2e9f866aadfaade195127"
May 26 16:17:43.932962 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:43.932918    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"539ed5b2609581212f3a707481623949956534a06bd2e9f866aadfaade195127"} err="failed to get container status \"539ed5b2609581212f3a707481623949956534a06bd2e9f866aadfaade195127\": rpc error: code = NotFound desc = could not find container \"539ed5b2609581212f3a707481623949956534a06bd2e9f866aadfaade195127\": container with ID starting with 539ed5b2609581212f3a707481623949956534a06bd2e9f866aadfaade195127 not found: ID does not exist"
May 26 16:17:43.932962 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:43.932933    2572 scope.go:117] "RemoveContainer" containerID="29b3c777e67a39b3983b8cc39c026ec22d922dd50f949f4725b889eb42fee277"
May 26 16:17:43.933158 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:17:43.933144    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29b3c777e67a39b3983b8cc39c026ec22d922dd50f949f4725b889eb42fee277\": container with ID starting with 29b3c777e67a39b3983b8cc39c026ec22d922dd50f949f4725b889eb42fee277 not found: ID does not exist" containerID="29b3c777e67a39b3983b8cc39c026ec22d922dd50f949f4725b889eb42fee277"
May 26 16:17:43.933208 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:43.933161    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29b3c777e67a39b3983b8cc39c026ec22d922dd50f949f4725b889eb42fee277"} err="failed to get container status \"29b3c777e67a39b3983b8cc39c026ec22d922dd50f949f4725b889eb42fee277\": rpc error: code = NotFound desc = could not find container \"29b3c777e67a39b3983b8cc39c026ec22d922dd50f949f4725b889eb42fee277\": container with ID starting with 29b3c777e67a39b3983b8cc39c026ec22d922dd50f949f4725b889eb42fee277 not found: ID does not exist"
May 26 16:17:44.079370 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:44.079338    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0463102e-e1e1-4cc6-9ff6-fd058cf08130" path="/var/lib/kubelet/pods/0463102e-e1e1-4cc6-9ff6-fd058cf08130/volumes"
May 26 16:17:52.859973 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:17:52.859931    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-r64vj" podUID="4976badd-ed26-4af2-9205-21260aa36ef3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
May 26 16:18:02.860124 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:02.860085    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-r64vj" podUID="4976badd-ed26-4af2-9205-21260aa36ef3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
May 26 16:18:12.859663 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:12.859597    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-r64vj" podUID="4976badd-ed26-4af2-9205-21260aa36ef3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
May 26 16:18:22.860129 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:22.860089    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-r64vj" podUID="4976badd-ed26-4af2-9205-21260aa36ef3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
May 26 16:18:32.860521 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:32.860481    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-r64vj" podUID="4976badd-ed26-4af2-9205-21260aa36ef3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
May 26 16:18:42.860406 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:42.860375    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-r64vj"
May 26 16:18:43.419048 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:43.419013    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-r64vj"]
May 26 16:18:43.419355 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:43.419321    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-r64vj" podUID="4976badd-ed26-4af2-9205-21260aa36ef3" containerName="kserve-container" containerID="cri-o://31798316da8fe38391aa5643cebecfb222b524fb11fb1bad74fcb8ab7b588ea1" gracePeriod=30
May 26 16:18:43.419506 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:43.419369    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-r64vj" podUID="4976badd-ed26-4af2-9205-21260aa36ef3" containerName="kube-rbac-proxy" containerID="cri-o://2af4d3f23db76912f20c120e32cbd13729baf4ee563a3f9e72d029b34254bc4e" gracePeriod=30
May 26 16:18:43.527582 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:43.527547    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt"]
May 26 16:18:43.527879 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:43.527862    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f46a336-6162-4e1a-9ee8-86c83137f04b" containerName="kube-rbac-proxy"
May 26 16:18:43.527879 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:43.527879    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f46a336-6162-4e1a-9ee8-86c83137f04b" containerName="kube-rbac-proxy"
May 26 16:18:43.528002 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:43.527887    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0463102e-e1e1-4cc6-9ff6-fd058cf08130" containerName="kube-rbac-proxy"
May 26 16:18:43.528002 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:43.527892    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0463102e-e1e1-4cc6-9ff6-fd058cf08130" containerName="kube-rbac-proxy"
May 26 16:18:43.528002 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:43.527899    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0463102e-e1e1-4cc6-9ff6-fd058cf08130" containerName="kserve-container"
May 26 16:18:43.528002 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:43.527905    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0463102e-e1e1-4cc6-9ff6-fd058cf08130" containerName="kserve-container"
May 26 16:18:43.528002 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:43.527921    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f46a336-6162-4e1a-9ee8-86c83137f04b" containerName="kserve-container"
May 26 16:18:43.528002 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:43.527928    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f46a336-6162-4e1a-9ee8-86c83137f04b" containerName="kserve-container"
May 26 16:18:43.528002 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:43.527934    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0463102e-e1e1-4cc6-9ff6-fd058cf08130" containerName="agent"
May 26 16:18:43.528002 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:43.527939    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0463102e-e1e1-4cc6-9ff6-fd058cf08130" containerName="agent"
May 26 16:18:43.528002 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:43.527946    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0463102e-e1e1-4cc6-9ff6-fd058cf08130" containerName="storage-initializer"
May 26 16:18:43.528002 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:43.527951    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0463102e-e1e1-4cc6-9ff6-fd058cf08130" containerName="storage-initializer"
May 26 16:18:43.528002 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:43.527991    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="8f46a336-6162-4e1a-9ee8-86c83137f04b" containerName="kube-rbac-proxy"
May 26 16:18:43.528002 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:43.527998    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="0463102e-e1e1-4cc6-9ff6-fd058cf08130" containerName="agent"
May 26 16:18:43.528002 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:43.528005    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="0463102e-e1e1-4cc6-9ff6-fd058cf08130" containerName="kserve-container"
May 26 16:18:43.528624 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:43.528013    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="0463102e-e1e1-4cc6-9ff6-fd058cf08130" containerName="kube-rbac-proxy"
May 26 16:18:43.528624 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:43.528019    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="8f46a336-6162-4e1a-9ee8-86c83137f04b" containerName="kserve-container"
May 26 16:18:43.530782 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:43.530762    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt"
May 26 16:18:43.532968 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:43.532942    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
May 26 16:18:43.533088 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:43.532966    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
May 26 16:18:43.538864 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:43.538842    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt"]
May 26 16:18:43.655363 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:43.655328    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a719d4f-01a0-45e3-952c-51786280d7cd-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt\" (UID: \"2a719d4f-01a0-45e3-952c-51786280d7cd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt"
May 26 16:18:43.655555 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:43.655430    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wvlg\" (UniqueName: \"kubernetes.io/projected/2a719d4f-01a0-45e3-952c-51786280d7cd-kube-api-access-8wvlg\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt\" (UID: \"2a719d4f-01a0-45e3-952c-51786280d7cd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt"
May 26 16:18:43.655555 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:43.655462    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a719d4f-01a0-45e3-952c-51786280d7cd-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt\" (UID: \"2a719d4f-01a0-45e3-952c-51786280d7cd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt"
May 26 16:18:43.655555 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:43.655528    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a719d4f-01a0-45e3-952c-51786280d7cd-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt\" (UID: \"2a719d4f-01a0-45e3-952c-51786280d7cd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt"
May 26 16:18:43.757670 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:43.757294    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8wvlg\" (UniqueName: \"kubernetes.io/projected/2a719d4f-01a0-45e3-952c-51786280d7cd-kube-api-access-8wvlg\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt\" (UID: \"2a719d4f-01a0-45e3-952c-51786280d7cd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt"
May 26 16:18:43.757670 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:43.757379    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a719d4f-01a0-45e3-952c-51786280d7cd-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt\" (UID: \"2a719d4f-01a0-45e3-952c-51786280d7cd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt"
May 26 16:18:43.757670 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:43.757412    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a719d4f-01a0-45e3-952c-51786280d7cd-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt\" (UID: \"2a719d4f-01a0-45e3-952c-51786280d7cd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt"
May 26 16:18:43.757670 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:43.757466    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a719d4f-01a0-45e3-952c-51786280d7cd-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt\" (UID: \"2a719d4f-01a0-45e3-952c-51786280d7cd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt"
May 26 16:18:43.761235 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:18:43.758077    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-serving-cert: secret "isvc-lightgbm-runtime-predictor-serving-cert" not found
May 26 16:18:43.761235 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:18:43.758183    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a719d4f-01a0-45e3-952c-51786280d7cd-proxy-tls podName:2a719d4f-01a0-45e3-952c-51786280d7cd nodeName:}" failed. No retries permitted until 2026-05-26 16:18:44.258161261 +0000 UTC m=+1040.733888591 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2a719d4f-01a0-45e3-952c-51786280d7cd-proxy-tls") pod "isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt" (UID: "2a719d4f-01a0-45e3-952c-51786280d7cd") : secret "isvc-lightgbm-runtime-predictor-serving-cert" not found
May 26 16:18:43.761235 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:43.758377    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a719d4f-01a0-45e3-952c-51786280d7cd-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt\" (UID: \"2a719d4f-01a0-45e3-952c-51786280d7cd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt"
May 26 16:18:43.761235 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:43.759033    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a719d4f-01a0-45e3-952c-51786280d7cd-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt\" (UID: \"2a719d4f-01a0-45e3-952c-51786280d7cd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt"
May 26 16:18:43.766448 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:43.766431    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wvlg\" (UniqueName: \"kubernetes.io/projected/2a719d4f-01a0-45e3-952c-51786280d7cd-kube-api-access-8wvlg\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt\" (UID: \"2a719d4f-01a0-45e3-952c-51786280d7cd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt"
May 26 16:18:44.074339 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:44.074254    2572 generic.go:358] "Generic (PLEG): container finished" podID="4976badd-ed26-4af2-9205-21260aa36ef3" containerID="2af4d3f23db76912f20c120e32cbd13729baf4ee563a3f9e72d029b34254bc4e" exitCode=2
May 26 16:18:44.074339 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:44.074327    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-r64vj" event={"ID":"4976badd-ed26-4af2-9205-21260aa36ef3","Type":"ContainerDied","Data":"2af4d3f23db76912f20c120e32cbd13729baf4ee563a3f9e72d029b34254bc4e"}
May 26 16:18:44.261795 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:44.261764    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a719d4f-01a0-45e3-952c-51786280d7cd-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt\" (UID: \"2a719d4f-01a0-45e3-952c-51786280d7cd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt"
May 26 16:18:44.264188 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:44.264157    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a719d4f-01a0-45e3-952c-51786280d7cd-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt\" (UID: \"2a719d4f-01a0-45e3-952c-51786280d7cd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt"
May 26 16:18:44.441950 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:44.441898    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt"
May 26 16:18:44.572924 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:44.572902    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt"]
May 26 16:18:44.575548 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:18:44.575515    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a719d4f_01a0_45e3_952c_51786280d7cd.slice/crio-24e7b346cf6693c15d17ec0d2012ec2716c64179f04631d23ece575fb5c76446 WatchSource:0}: Error finding container 24e7b346cf6693c15d17ec0d2012ec2716c64179f04631d23ece575fb5c76446: Status 404 returned error can't find the container with id 24e7b346cf6693c15d17ec0d2012ec2716c64179f04631d23ece575fb5c76446
May 26 16:18:45.078924 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:45.078883    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt" event={"ID":"2a719d4f-01a0-45e3-952c-51786280d7cd","Type":"ContainerStarted","Data":"494c2dde05199014d5023250741866ee73837c4dc0f191c902fd05a37962f363"}
May 26 16:18:45.079306 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:45.078934    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt" event={"ID":"2a719d4f-01a0-45e3-952c-51786280d7cd","Type":"ContainerStarted","Data":"24e7b346cf6693c15d17ec0d2012ec2716c64179f04631d23ece575fb5c76446"}
May 26 16:18:47.856243 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:47.856209    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-r64vj" podUID="4976badd-ed26-4af2-9205-21260aa36ef3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
May 26 16:18:48.058296 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:48.058271    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-r64vj"
May 26 16:18:48.090451 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:48.090378    2572 generic.go:358] "Generic (PLEG): container finished" podID="4976badd-ed26-4af2-9205-21260aa36ef3" containerID="31798316da8fe38391aa5643cebecfb222b524fb11fb1bad74fcb8ab7b588ea1" exitCode=0
May 26 16:18:48.090451 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:48.090437    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-r64vj" event={"ID":"4976badd-ed26-4af2-9205-21260aa36ef3","Type":"ContainerDied","Data":"31798316da8fe38391aa5643cebecfb222b524fb11fb1bad74fcb8ab7b588ea1"}
May 26 16:18:48.090688 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:48.090460    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-r64vj" event={"ID":"4976badd-ed26-4af2-9205-21260aa36ef3","Type":"ContainerDied","Data":"d7c5752492ff5276860888bacbd62b2db3b04a835400914b4e16e15451e5dbfb"}
May 26 16:18:48.090688 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:48.090476    2572 scope.go:117] "RemoveContainer" containerID="2af4d3f23db76912f20c120e32cbd13729baf4ee563a3f9e72d029b34254bc4e"
May 26 16:18:48.090688 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:48.090489    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-r64vj"
May 26 16:18:48.099322 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:48.099299    2572 scope.go:117] "RemoveContainer" containerID="31798316da8fe38391aa5643cebecfb222b524fb11fb1bad74fcb8ab7b588ea1"
May 26 16:18:48.106403 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:48.106380    2572 scope.go:117] "RemoveContainer" containerID="d8650754bbab970428dc753eb0771c734230ae432b6367821508e5f81bfc6737"
May 26 16:18:48.113218 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:48.113198    2572 scope.go:117] "RemoveContainer" containerID="2af4d3f23db76912f20c120e32cbd13729baf4ee563a3f9e72d029b34254bc4e"
May 26 16:18:48.113481 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:18:48.113460    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2af4d3f23db76912f20c120e32cbd13729baf4ee563a3f9e72d029b34254bc4e\": container with ID starting with 2af4d3f23db76912f20c120e32cbd13729baf4ee563a3f9e72d029b34254bc4e not found: ID does not exist" containerID="2af4d3f23db76912f20c120e32cbd13729baf4ee563a3f9e72d029b34254bc4e"
May 26 16:18:48.113522 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:48.113490    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2af4d3f23db76912f20c120e32cbd13729baf4ee563a3f9e72d029b34254bc4e"} err="failed to get container status \"2af4d3f23db76912f20c120e32cbd13729baf4ee563a3f9e72d029b34254bc4e\": rpc error: code = NotFound desc = could not find container \"2af4d3f23db76912f20c120e32cbd13729baf4ee563a3f9e72d029b34254bc4e\": container with ID starting with 2af4d3f23db76912f20c120e32cbd13729baf4ee563a3f9e72d029b34254bc4e not found: ID does not exist"
May 26 16:18:48.113522 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:48.113510    2572 scope.go:117] "RemoveContainer" containerID="31798316da8fe38391aa5643cebecfb222b524fb11fb1bad74fcb8ab7b588ea1"
May 26 16:18:48.113772 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:18:48.113749    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31798316da8fe38391aa5643cebecfb222b524fb11fb1bad74fcb8ab7b588ea1\": container with ID starting with 31798316da8fe38391aa5643cebecfb222b524fb11fb1bad74fcb8ab7b588ea1 not found: ID does not exist" containerID="31798316da8fe38391aa5643cebecfb222b524fb11fb1bad74fcb8ab7b588ea1"
May 26 16:18:48.113827 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:48.113787    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31798316da8fe38391aa5643cebecfb222b524fb11fb1bad74fcb8ab7b588ea1"} err="failed to get container status \"31798316da8fe38391aa5643cebecfb222b524fb11fb1bad74fcb8ab7b588ea1\": rpc error: code = NotFound desc = could not find container \"31798316da8fe38391aa5643cebecfb222b524fb11fb1bad74fcb8ab7b588ea1\": container with ID starting with 31798316da8fe38391aa5643cebecfb222b524fb11fb1bad74fcb8ab7b588ea1 not found: ID does not exist"
May 26 16:18:48.113827 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:48.113812    2572 scope.go:117] "RemoveContainer" containerID="d8650754bbab970428dc753eb0771c734230ae432b6367821508e5f81bfc6737"
May 26 16:18:48.114066 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:18:48.114048    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8650754bbab970428dc753eb0771c734230ae432b6367821508e5f81bfc6737\": container with ID starting with d8650754bbab970428dc753eb0771c734230ae432b6367821508e5f81bfc6737 not found: ID does not exist" containerID="d8650754bbab970428dc753eb0771c734230ae432b6367821508e5f81bfc6737"
May 26 16:18:48.114104 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:48.114073    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8650754bbab970428dc753eb0771c734230ae432b6367821508e5f81bfc6737"} err="failed to get container status \"d8650754bbab970428dc753eb0771c734230ae432b6367821508e5f81bfc6737\": rpc error: code = NotFound desc = could not find container \"d8650754bbab970428dc753eb0771c734230ae432b6367821508e5f81bfc6737\": container with ID starting with d8650754bbab970428dc753eb0771c734230ae432b6367821508e5f81bfc6737 not found: ID does not exist"
May 26 16:18:48.195836 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:48.195802    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4976badd-ed26-4af2-9205-21260aa36ef3-proxy-tls\") pod \"4976badd-ed26-4af2-9205-21260aa36ef3\" (UID: \"4976badd-ed26-4af2-9205-21260aa36ef3\") "
May 26 16:18:48.195969 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:48.195856    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxgbj\" (UniqueName: \"kubernetes.io/projected/4976badd-ed26-4af2-9205-21260aa36ef3-kube-api-access-zxgbj\") pod \"4976badd-ed26-4af2-9205-21260aa36ef3\" (UID: \"4976badd-ed26-4af2-9205-21260aa36ef3\") "
May 26 16:18:48.195969 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:48.195928    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4976badd-ed26-4af2-9205-21260aa36ef3-isvc-lightgbm-kube-rbac-proxy-sar-config\") pod \"4976badd-ed26-4af2-9205-21260aa36ef3\" (UID: \"4976badd-ed26-4af2-9205-21260aa36ef3\") "
May 26 16:18:48.195969 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:48.195962    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4976badd-ed26-4af2-9205-21260aa36ef3-kserve-provision-location\") pod \"4976badd-ed26-4af2-9205-21260aa36ef3\" (UID: \"4976badd-ed26-4af2-9205-21260aa36ef3\") "
May 26 16:18:48.196277 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:48.196250    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4976badd-ed26-4af2-9205-21260aa36ef3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4976badd-ed26-4af2-9205-21260aa36ef3" (UID: "4976badd-ed26-4af2-9205-21260aa36ef3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:18:48.196369 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:48.196280    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4976badd-ed26-4af2-9205-21260aa36ef3-isvc-lightgbm-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-lightgbm-kube-rbac-proxy-sar-config") pod "4976badd-ed26-4af2-9205-21260aa36ef3" (UID: "4976badd-ed26-4af2-9205-21260aa36ef3"). InnerVolumeSpecName "isvc-lightgbm-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:18:48.197964 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:48.197944    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4976badd-ed26-4af2-9205-21260aa36ef3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4976badd-ed26-4af2-9205-21260aa36ef3" (UID: "4976badd-ed26-4af2-9205-21260aa36ef3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:18:48.198033 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:48.197986    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4976badd-ed26-4af2-9205-21260aa36ef3-kube-api-access-zxgbj" (OuterVolumeSpecName: "kube-api-access-zxgbj") pod "4976badd-ed26-4af2-9205-21260aa36ef3" (UID: "4976badd-ed26-4af2-9205-21260aa36ef3"). InnerVolumeSpecName "kube-api-access-zxgbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:18:48.296792 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:48.296758    2572 reconciler_common.go:299] "Volume detached for volume \"isvc-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4976badd-ed26-4af2-9205-21260aa36ef3-isvc-lightgbm-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:18:48.296792 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:48.296786    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4976badd-ed26-4af2-9205-21260aa36ef3-kserve-provision-location\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:18:48.296792 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:48.296798    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4976badd-ed26-4af2-9205-21260aa36ef3-proxy-tls\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:18:48.297027 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:48.296809    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxgbj\" (UniqueName: \"kubernetes.io/projected/4976badd-ed26-4af2-9205-21260aa36ef3-kube-api-access-zxgbj\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:18:48.412490 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:48.412452    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-r64vj"]
May 26 16:18:48.424581 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:48.422623    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-r64vj"]
May 26 16:18:49.095099 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:49.095057    2572 generic.go:358] "Generic (PLEG): container finished" podID="2a719d4f-01a0-45e3-952c-51786280d7cd" containerID="494c2dde05199014d5023250741866ee73837c4dc0f191c902fd05a37962f363" exitCode=0
May 26 16:18:49.095099 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:49.095101    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt" event={"ID":"2a719d4f-01a0-45e3-952c-51786280d7cd","Type":"ContainerDied","Data":"494c2dde05199014d5023250741866ee73837c4dc0f191c902fd05a37962f363"}
May 26 16:18:50.079654 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:50.079619    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4976badd-ed26-4af2-9205-21260aa36ef3" path="/var/lib/kubelet/pods/4976badd-ed26-4af2-9205-21260aa36ef3/volumes"
May 26 16:18:50.099728 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:50.099696    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt" event={"ID":"2a719d4f-01a0-45e3-952c-51786280d7cd","Type":"ContainerStarted","Data":"b7c5532fc683689115637f50f9497b5134e8accfbdda3cd1a7f95a0bfef9d255"}
May 26 16:18:50.099728 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:50.099734    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt" event={"ID":"2a719d4f-01a0-45e3-952c-51786280d7cd","Type":"ContainerStarted","Data":"a76404df49542f96af9ca8526f19c89831a328f75f0210610bec4ccf4e82534a"}
May 26 16:18:50.100243 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:50.099965    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt"
May 26 16:18:50.118267 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:50.118223    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt" podStartSLOduration=7.11820834 podStartE2EDuration="7.11820834s" podCreationTimestamp="2026-05-26 16:18:43 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:18:50.116346865 +0000 UTC m=+1046.592074215" watchObservedRunningTime="2026-05-26 16:18:50.11820834 +0000 UTC m=+1046.593935691"
May 26 16:18:51.102869 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:51.102831    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt"
May 26 16:18:51.104024 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:51.103997    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt" podUID="2a719d4f-01a0-45e3-952c-51786280d7cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
May 26 16:18:52.105807 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:52.105767    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt" podUID="2a719d4f-01a0-45e3-952c-51786280d7cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
May 26 16:18:57.109780 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:57.109750    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt"
May 26 16:18:57.110378 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:18:57.110349    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt" podUID="2a719d4f-01a0-45e3-952c-51786280d7cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
May 26 16:19:07.110815 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:19:07.110777    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt" podUID="2a719d4f-01a0-45e3-952c-51786280d7cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
May 26 16:19:17.110287 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:19:17.110204    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt" podUID="2a719d4f-01a0-45e3-952c-51786280d7cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
May 26 16:19:27.110329 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:19:27.110285    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt" podUID="2a719d4f-01a0-45e3-952c-51786280d7cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
May 26 16:19:37.110506 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:19:37.110467    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt" podUID="2a719d4f-01a0-45e3-952c-51786280d7cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
May 26 16:19:47.110536 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:19:47.110497    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt" podUID="2a719d4f-01a0-45e3-952c-51786280d7cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
May 26 16:19:57.110481 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:19:57.110432    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt" podUID="2a719d4f-01a0-45e3-952c-51786280d7cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
May 26 16:20:07.110810 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:07.110781    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt"
May 26 16:20:14.000488 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:14.000451    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt"]
May 26 16:20:14.001002 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:14.000813    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt" podUID="2a719d4f-01a0-45e3-952c-51786280d7cd" containerName="kserve-container" containerID="cri-o://a76404df49542f96af9ca8526f19c89831a328f75f0210610bec4ccf4e82534a" gracePeriod=30
May 26 16:20:14.001002 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:14.000881    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt" podUID="2a719d4f-01a0-45e3-952c-51786280d7cd" containerName="kube-rbac-proxy" containerID="cri-o://b7c5532fc683689115637f50f9497b5134e8accfbdda3cd1a7f95a0bfef9d255" gracePeriod=30
May 26 16:20:14.100613 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:14.100564    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp"]
May 26 16:20:14.100845 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:14.100833    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4976badd-ed26-4af2-9205-21260aa36ef3" containerName="storage-initializer"
May 26 16:20:14.100889 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:14.100847    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4976badd-ed26-4af2-9205-21260aa36ef3" containerName="storage-initializer"
May 26 16:20:14.100889 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:14.100869    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4976badd-ed26-4af2-9205-21260aa36ef3" containerName="kserve-container"
May 26 16:20:14.100889 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:14.100875    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4976badd-ed26-4af2-9205-21260aa36ef3" containerName="kserve-container"
May 26 16:20:14.100889 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:14.100881    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4976badd-ed26-4af2-9205-21260aa36ef3" containerName="kube-rbac-proxy"
May 26 16:20:14.100889 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:14.100887    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4976badd-ed26-4af2-9205-21260aa36ef3" containerName="kube-rbac-proxy"
May 26 16:20:14.101034 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:14.100929    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="4976badd-ed26-4af2-9205-21260aa36ef3" containerName="kube-rbac-proxy"
May 26 16:20:14.101034 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:14.100937    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="4976badd-ed26-4af2-9205-21260aa36ef3" containerName="kserve-container"
May 26 16:20:14.102956 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:14.102930    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp"
May 26 16:20:14.105104 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:14.105082    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
May 26 16:20:14.105673 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:14.105652    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
May 26 16:20:14.117337 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:14.117310    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp"]
May 26 16:20:14.136969 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:14.136935    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/316f0cbe-f528-4781-80c7-faddf7ab1d8e-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp\" (UID: \"316f0cbe-f528-4781-80c7-faddf7ab1d8e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp"
May 26 16:20:14.137115 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:14.136979    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/316f0cbe-f528-4781-80c7-faddf7ab1d8e-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp\" (UID: \"316f0cbe-f528-4781-80c7-faddf7ab1d8e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp"
May 26 16:20:14.137115 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:14.137010    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28zcf\" (UniqueName: \"kubernetes.io/projected/316f0cbe-f528-4781-80c7-faddf7ab1d8e-kube-api-access-28zcf\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp\" (UID: \"316f0cbe-f528-4781-80c7-faddf7ab1d8e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp"
May 26 16:20:14.137115 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:14.137073    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/316f0cbe-f528-4781-80c7-faddf7ab1d8e-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp\" (UID: \"316f0cbe-f528-4781-80c7-faddf7ab1d8e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp"
May 26 16:20:14.237497 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:14.237456    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/316f0cbe-f528-4781-80c7-faddf7ab1d8e-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp\" (UID: \"316f0cbe-f528-4781-80c7-faddf7ab1d8e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp"
May 26 16:20:14.237675 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:14.237624    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/316f0cbe-f528-4781-80c7-faddf7ab1d8e-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp\" (UID: \"316f0cbe-f528-4781-80c7-faddf7ab1d8e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp"
May 26 16:20:14.237747 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:14.237688    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-28zcf\" (UniqueName: \"kubernetes.io/projected/316f0cbe-f528-4781-80c7-faddf7ab1d8e-kube-api-access-28zcf\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp\" (UID: \"316f0cbe-f528-4781-80c7-faddf7ab1d8e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp"
May 26 16:20:14.237747 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:14.237737    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/316f0cbe-f528-4781-80c7-faddf7ab1d8e-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp\" (UID: \"316f0cbe-f528-4781-80c7-faddf7ab1d8e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp"
May 26 16:20:14.237898 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:20:14.237882    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-serving-cert: secret "isvc-lightgbm-v2-runtime-predictor-serving-cert" not found
May 26 16:20:14.237969 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:20:14.237957    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/316f0cbe-f528-4781-80c7-faddf7ab1d8e-proxy-tls podName:316f0cbe-f528-4781-80c7-faddf7ab1d8e nodeName:}" failed. No retries permitted until 2026-05-26 16:20:14.737940517 +0000 UTC m=+1131.213667849 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/316f0cbe-f528-4781-80c7-faddf7ab1d8e-proxy-tls") pod "isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp" (UID: "316f0cbe-f528-4781-80c7-faddf7ab1d8e") : secret "isvc-lightgbm-v2-runtime-predictor-serving-cert" not found
May 26 16:20:14.237969 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:14.237885    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/316f0cbe-f528-4781-80c7-faddf7ab1d8e-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp\" (UID: \"316f0cbe-f528-4781-80c7-faddf7ab1d8e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp"
May 26 16:20:14.238227 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:14.238209    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/316f0cbe-f528-4781-80c7-faddf7ab1d8e-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp\" (UID: \"316f0cbe-f528-4781-80c7-faddf7ab1d8e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp"
May 26 16:20:14.247021 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:14.247001    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28zcf\" (UniqueName: \"kubernetes.io/projected/316f0cbe-f528-4781-80c7-faddf7ab1d8e-kube-api-access-28zcf\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp\" (UID: \"316f0cbe-f528-4781-80c7-faddf7ab1d8e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp"
May 26 16:20:14.330086 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:14.330002    2572 generic.go:358] "Generic (PLEG): container finished" podID="2a719d4f-01a0-45e3-952c-51786280d7cd" containerID="b7c5532fc683689115637f50f9497b5134e8accfbdda3cd1a7f95a0bfef9d255" exitCode=2
May 26 16:20:14.330086 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:14.330040    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt" event={"ID":"2a719d4f-01a0-45e3-952c-51786280d7cd","Type":"ContainerDied","Data":"b7c5532fc683689115637f50f9497b5134e8accfbdda3cd1a7f95a0bfef9d255"}
May 26 16:20:14.741727 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:14.741667    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/316f0cbe-f528-4781-80c7-faddf7ab1d8e-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp\" (UID: \"316f0cbe-f528-4781-80c7-faddf7ab1d8e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp"
May 26 16:20:14.744117 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:14.744092    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/316f0cbe-f528-4781-80c7-faddf7ab1d8e-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp\" (UID: \"316f0cbe-f528-4781-80c7-faddf7ab1d8e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp"
May 26 16:20:15.014056 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:15.013962    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp"
May 26 16:20:15.136373 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:15.136343    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp"]
May 26 16:20:15.140946 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:20:15.140913    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod316f0cbe_f528_4781_80c7_faddf7ab1d8e.slice/crio-ff5058ca68e519436c22b5287384acfea291642d017a0fd58b4ee2711ea916f5 WatchSource:0}: Error finding container ff5058ca68e519436c22b5287384acfea291642d017a0fd58b4ee2711ea916f5: Status 404 returned error can't find the container with id ff5058ca68e519436c22b5287384acfea291642d017a0fd58b4ee2711ea916f5
May 26 16:20:15.335265 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:15.335181    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp" event={"ID":"316f0cbe-f528-4781-80c7-faddf7ab1d8e","Type":"ContainerStarted","Data":"7bf80c19dbf37a5285c882f484f251e2747632059469ed59e7c1aff234a616c4"}
May 26 16:20:15.335265 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:15.335220    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp" event={"ID":"316f0cbe-f528-4781-80c7-faddf7ab1d8e","Type":"ContainerStarted","Data":"ff5058ca68e519436c22b5287384acfea291642d017a0fd58b4ee2711ea916f5"}
May 26 16:20:17.106981 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:17.106936    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt" podUID="2a719d4f-01a0-45e3-952c-51786280d7cd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
May 26 16:20:17.110263 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:17.110229    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt" podUID="2a719d4f-01a0-45e3-952c-51786280d7cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
May 26 16:20:18.749990 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:18.749965    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt"
May 26 16:20:18.876700 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:18.876669    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wvlg\" (UniqueName: \"kubernetes.io/projected/2a719d4f-01a0-45e3-952c-51786280d7cd-kube-api-access-8wvlg\") pod \"2a719d4f-01a0-45e3-952c-51786280d7cd\" (UID: \"2a719d4f-01a0-45e3-952c-51786280d7cd\") "
May 26 16:20:18.876873 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:18.876709    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a719d4f-01a0-45e3-952c-51786280d7cd-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\") pod \"2a719d4f-01a0-45e3-952c-51786280d7cd\" (UID: \"2a719d4f-01a0-45e3-952c-51786280d7cd\") "
May 26 16:20:18.876873 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:18.876753    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a719d4f-01a0-45e3-952c-51786280d7cd-kserve-provision-location\") pod \"2a719d4f-01a0-45e3-952c-51786280d7cd\" (UID: \"2a719d4f-01a0-45e3-952c-51786280d7cd\") "
May 26 16:20:18.876873 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:18.876802    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a719d4f-01a0-45e3-952c-51786280d7cd-proxy-tls\") pod \"2a719d4f-01a0-45e3-952c-51786280d7cd\" (UID: \"2a719d4f-01a0-45e3-952c-51786280d7cd\") "
May 26 16:20:18.877132 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:18.877110    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a719d4f-01a0-45e3-952c-51786280d7cd-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-lightgbm-runtime-kube-rbac-proxy-sar-config") pod "2a719d4f-01a0-45e3-952c-51786280d7cd" (UID: "2a719d4f-01a0-45e3-952c-51786280d7cd"). InnerVolumeSpecName "isvc-lightgbm-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:20:18.877198 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:18.877125    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a719d4f-01a0-45e3-952c-51786280d7cd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2a719d4f-01a0-45e3-952c-51786280d7cd" (UID: "2a719d4f-01a0-45e3-952c-51786280d7cd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:20:18.878814 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:18.878784    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a719d4f-01a0-45e3-952c-51786280d7cd-kube-api-access-8wvlg" (OuterVolumeSpecName: "kube-api-access-8wvlg") pod "2a719d4f-01a0-45e3-952c-51786280d7cd" (UID: "2a719d4f-01a0-45e3-952c-51786280d7cd"). InnerVolumeSpecName "kube-api-access-8wvlg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:20:18.878912 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:18.878829    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a719d4f-01a0-45e3-952c-51786280d7cd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2a719d4f-01a0-45e3-952c-51786280d7cd" (UID: "2a719d4f-01a0-45e3-952c-51786280d7cd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:20:18.978333 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:18.978299    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wvlg\" (UniqueName: \"kubernetes.io/projected/2a719d4f-01a0-45e3-952c-51786280d7cd-kube-api-access-8wvlg\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:20:18.978333 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:18.978333    2572 reconciler_common.go:299] "Volume detached for volume \"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a719d4f-01a0-45e3-952c-51786280d7cd-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:20:18.978551 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:18.978350    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a719d4f-01a0-45e3-952c-51786280d7cd-kserve-provision-location\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:20:18.978551 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:18.978365    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a719d4f-01a0-45e3-952c-51786280d7cd-proxy-tls\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:20:19.346973 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:19.346934    2572 generic.go:358] "Generic (PLEG): container finished" podID="2a719d4f-01a0-45e3-952c-51786280d7cd" containerID="a76404df49542f96af9ca8526f19c89831a328f75f0210610bec4ccf4e82534a" exitCode=0
May 26 16:20:19.347153 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:19.347012    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt"
May 26 16:20:19.347153 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:19.347016    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt" event={"ID":"2a719d4f-01a0-45e3-952c-51786280d7cd","Type":"ContainerDied","Data":"a76404df49542f96af9ca8526f19c89831a328f75f0210610bec4ccf4e82534a"}
May 26 16:20:19.347153 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:19.347048    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt" event={"ID":"2a719d4f-01a0-45e3-952c-51786280d7cd","Type":"ContainerDied","Data":"24e7b346cf6693c15d17ec0d2012ec2716c64179f04631d23ece575fb5c76446"}
May 26 16:20:19.347153 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:19.347064    2572 scope.go:117] "RemoveContainer" containerID="b7c5532fc683689115637f50f9497b5134e8accfbdda3cd1a7f95a0bfef9d255"
May 26 16:20:19.348430 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:19.348407    2572 generic.go:358] "Generic (PLEG): container finished" podID="316f0cbe-f528-4781-80c7-faddf7ab1d8e" containerID="7bf80c19dbf37a5285c882f484f251e2747632059469ed59e7c1aff234a616c4" exitCode=0
May 26 16:20:19.348546 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:19.348467    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp" event={"ID":"316f0cbe-f528-4781-80c7-faddf7ab1d8e","Type":"ContainerDied","Data":"7bf80c19dbf37a5285c882f484f251e2747632059469ed59e7c1aff234a616c4"}
May 26 16:20:19.358364 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:19.358346    2572 scope.go:117] "RemoveContainer" containerID="a76404df49542f96af9ca8526f19c89831a328f75f0210610bec4ccf4e82534a"
May 26 16:20:19.365260 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:19.365241    2572 scope.go:117] "RemoveContainer" containerID="494c2dde05199014d5023250741866ee73837c4dc0f191c902fd05a37962f363"
May 26 16:20:19.372621 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:19.372582    2572 scope.go:117] "RemoveContainer" containerID="b7c5532fc683689115637f50f9497b5134e8accfbdda3cd1a7f95a0bfef9d255"
May 26 16:20:19.372859 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:20:19.372843    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7c5532fc683689115637f50f9497b5134e8accfbdda3cd1a7f95a0bfef9d255\": container with ID starting with b7c5532fc683689115637f50f9497b5134e8accfbdda3cd1a7f95a0bfef9d255 not found: ID does not exist" containerID="b7c5532fc683689115637f50f9497b5134e8accfbdda3cd1a7f95a0bfef9d255"
May 26 16:20:19.372915 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:19.372869    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7c5532fc683689115637f50f9497b5134e8accfbdda3cd1a7f95a0bfef9d255"} err="failed to get container status \"b7c5532fc683689115637f50f9497b5134e8accfbdda3cd1a7f95a0bfef9d255\": rpc error: code = NotFound desc = could not find container \"b7c5532fc683689115637f50f9497b5134e8accfbdda3cd1a7f95a0bfef9d255\": container with ID starting with b7c5532fc683689115637f50f9497b5134e8accfbdda3cd1a7f95a0bfef9d255 not found: ID does not exist"
May 26 16:20:19.372915 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:19.372886    2572 scope.go:117] "RemoveContainer" containerID="a76404df49542f96af9ca8526f19c89831a328f75f0210610bec4ccf4e82534a"
May 26 16:20:19.373076 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:20:19.373059    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a76404df49542f96af9ca8526f19c89831a328f75f0210610bec4ccf4e82534a\": container with ID starting with a76404df49542f96af9ca8526f19c89831a328f75f0210610bec4ccf4e82534a not found: ID does not exist" containerID="a76404df49542f96af9ca8526f19c89831a328f75f0210610bec4ccf4e82534a"
May 26 16:20:19.373118 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:19.373083    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a76404df49542f96af9ca8526f19c89831a328f75f0210610bec4ccf4e82534a"} err="failed to get container status \"a76404df49542f96af9ca8526f19c89831a328f75f0210610bec4ccf4e82534a\": rpc error: code = NotFound desc = could not find container \"a76404df49542f96af9ca8526f19c89831a328f75f0210610bec4ccf4e82534a\": container with ID starting with a76404df49542f96af9ca8526f19c89831a328f75f0210610bec4ccf4e82534a not found: ID does not exist"
May 26 16:20:19.373118 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:19.373103    2572 scope.go:117] "RemoveContainer" containerID="494c2dde05199014d5023250741866ee73837c4dc0f191c902fd05a37962f363"
May 26 16:20:19.373290 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:20:19.373273    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"494c2dde05199014d5023250741866ee73837c4dc0f191c902fd05a37962f363\": container with ID starting with 494c2dde05199014d5023250741866ee73837c4dc0f191c902fd05a37962f363 not found: ID does not exist" containerID="494c2dde05199014d5023250741866ee73837c4dc0f191c902fd05a37962f363"
May 26 16:20:19.373333 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:19.373295    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"494c2dde05199014d5023250741866ee73837c4dc0f191c902fd05a37962f363"} err="failed to get container status \"494c2dde05199014d5023250741866ee73837c4dc0f191c902fd05a37962f363\": rpc error: code = NotFound desc = could not find container \"494c2dde05199014d5023250741866ee73837c4dc0f191c902fd05a37962f363\": container with ID starting with 494c2dde05199014d5023250741866ee73837c4dc0f191c902fd05a37962f363 not found: ID does not exist"
May 26 16:20:19.377572 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:19.377550    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt"]
May 26 16:20:19.383152 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:19.383130    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-nmnbt"]
May 26 16:20:20.082962 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:20:20.082421    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a719d4f-01a0-45e3-952c-51786280d7cd" path="/var/lib/kubelet/pods/2a719d4f-01a0-45e3-952c-51786280d7cd/volumes"
May 26 16:22:37.160521 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:22:37.160500    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 16:22:37.780364 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:22:37.780330    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp" event={"ID":"316f0cbe-f528-4781-80c7-faddf7ab1d8e","Type":"ContainerStarted","Data":"81b1c024178199f3c806cec97e539be3939e08567ff34ceff48442aa903f7c53"}
May 26 16:22:37.780364 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:22:37.780364    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp" event={"ID":"316f0cbe-f528-4781-80c7-faddf7ab1d8e","Type":"ContainerStarted","Data":"513508ac08295b0f16df9dec30bad69e67cbfeacb75f71e7b365e521fc2f4c15"}
May 26 16:22:37.780691 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:22:37.780518    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp"
May 26 16:22:37.802572 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:22:37.802508    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp" podStartSLOduration=6.117216773 podStartE2EDuration="2m23.802489036s" podCreationTimestamp="2026-05-26 16:20:14 +0000 UTC" firstStartedPulling="2026-05-26 16:20:19.349538775 +0000 UTC m=+1135.825266103" lastFinishedPulling="2026-05-26 16:22:37.034811024 +0000 UTC m=+1273.510538366" observedRunningTime="2026-05-26 16:22:37.801558551 +0000 UTC m=+1274.277285902" watchObservedRunningTime="2026-05-26 16:22:37.802489036 +0000 UTC m=+1274.278216387"
May 26 16:22:38.783137 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:22:38.783103    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp"
May 26 16:22:44.791795 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:22:44.791764    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp"
May 26 16:23:14.796195 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:14.796166    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp"
May 26 16:23:24.223319 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:24.223284    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp"]
May 26 16:23:24.223826 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:24.223616    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp" podUID="316f0cbe-f528-4781-80c7-faddf7ab1d8e" containerName="kserve-container" containerID="cri-o://513508ac08295b0f16df9dec30bad69e67cbfeacb75f71e7b365e521fc2f4c15" gracePeriod=30
May 26 16:23:24.223826 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:24.223674    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp" podUID="316f0cbe-f528-4781-80c7-faddf7ab1d8e" containerName="kube-rbac-proxy" containerID="cri-o://81b1c024178199f3c806cec97e539be3939e08567ff34ceff48442aa903f7c53" gracePeriod=30
May 26 16:23:24.345281 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:24.345245    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv"]
May 26 16:23:24.345646 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:24.345629    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a719d4f-01a0-45e3-952c-51786280d7cd" containerName="storage-initializer"
May 26 16:23:24.345708 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:24.345650    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a719d4f-01a0-45e3-952c-51786280d7cd" containerName="storage-initializer"
May 26 16:23:24.345708 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:24.345670    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a719d4f-01a0-45e3-952c-51786280d7cd" containerName="kube-rbac-proxy"
May 26 16:23:24.345708 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:24.345679    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a719d4f-01a0-45e3-952c-51786280d7cd" containerName="kube-rbac-proxy"
May 26 16:23:24.345708 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:24.345694    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a719d4f-01a0-45e3-952c-51786280d7cd" containerName="kserve-container"
May 26 16:23:24.345708 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:24.345703    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a719d4f-01a0-45e3-952c-51786280d7cd" containerName="kserve-container"
May 26 16:23:24.345900 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:24.345807    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="2a719d4f-01a0-45e3-952c-51786280d7cd" containerName="kserve-container"
May 26 16:23:24.345900 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:24.345819    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="2a719d4f-01a0-45e3-952c-51786280d7cd" containerName="kube-rbac-proxy"
May 26 16:23:24.348105 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:24.348087    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv"
May 26 16:23:24.350246 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:24.350221    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
May 26 16:23:24.350367 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:24.350242    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
May 26 16:23:24.356989 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:24.356965    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv"]
May 26 16:23:24.399615 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:24.399572    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a31452b4-6879-42fe-968e-30d96044ad36-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv\" (UID: \"a31452b4-6879-42fe-968e-30d96044ad36\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv"
May 26 16:23:24.399791 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:24.399635    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a31452b4-6879-42fe-968e-30d96044ad36-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv\" (UID: \"a31452b4-6879-42fe-968e-30d96044ad36\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv"
May 26 16:23:24.399791 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:24.399665    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a31452b4-6879-42fe-968e-30d96044ad36-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv\" (UID: \"a31452b4-6879-42fe-968e-30d96044ad36\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv"
May 26 16:23:24.399791 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:24.399718    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz49m\" (UniqueName: \"kubernetes.io/projected/a31452b4-6879-42fe-968e-30d96044ad36-kube-api-access-gz49m\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv\" (UID: \"a31452b4-6879-42fe-968e-30d96044ad36\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv"
May 26 16:23:24.501062 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:24.500967    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a31452b4-6879-42fe-968e-30d96044ad36-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv\" (UID: \"a31452b4-6879-42fe-968e-30d96044ad36\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv"
May 26 16:23:24.501062 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:24.501013    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a31452b4-6879-42fe-968e-30d96044ad36-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv\" (UID: \"a31452b4-6879-42fe-968e-30d96044ad36\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv"
May 26 16:23:24.501062 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:24.501058    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gz49m\" (UniqueName: \"kubernetes.io/projected/a31452b4-6879-42fe-968e-30d96044ad36-kube-api-access-gz49m\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv\" (UID: \"a31452b4-6879-42fe-968e-30d96044ad36\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv"
May 26 16:23:24.501348 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:24.501104    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a31452b4-6879-42fe-968e-30d96044ad36-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv\" (UID: \"a31452b4-6879-42fe-968e-30d96044ad36\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv"
May 26 16:23:24.501348 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:23:24.501213    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-serving-cert: secret "isvc-lightgbm-v2-kserve-predictor-serving-cert" not found
May 26 16:23:24.501348 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:23:24.501294    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a31452b4-6879-42fe-968e-30d96044ad36-proxy-tls podName:a31452b4-6879-42fe-968e-30d96044ad36 nodeName:}" failed. No retries permitted until 2026-05-26 16:23:25.001271368 +0000 UTC m=+1321.476998700 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a31452b4-6879-42fe-968e-30d96044ad36-proxy-tls") pod "isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv" (UID: "a31452b4-6879-42fe-968e-30d96044ad36") : secret "isvc-lightgbm-v2-kserve-predictor-serving-cert" not found
May 26 16:23:24.501505 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:24.501450    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a31452b4-6879-42fe-968e-30d96044ad36-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv\" (UID: \"a31452b4-6879-42fe-968e-30d96044ad36\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv"
May 26 16:23:24.501837 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:24.501817    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a31452b4-6879-42fe-968e-30d96044ad36-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv\" (UID: \"a31452b4-6879-42fe-968e-30d96044ad36\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv"
May 26 16:23:24.509446 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:24.509424    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz49m\" (UniqueName: \"kubernetes.io/projected/a31452b4-6879-42fe-968e-30d96044ad36-kube-api-access-gz49m\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv\" (UID: \"a31452b4-6879-42fe-968e-30d96044ad36\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv"
May 26 16:23:24.786980 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:24.786892    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp" podUID="316f0cbe-f528-4781-80c7-faddf7ab1d8e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
May 26 16:23:24.792395 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:24.792353    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp" podUID="316f0cbe-f528-4781-80c7-faddf7ab1d8e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.27:8080/v2/models/isvc-lightgbm-v2-runtime/ready\": dial tcp 10.132.0.27:8080: connect: connection refused"
May 26 16:23:24.906028 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:24.905991    2572 generic.go:358] "Generic (PLEG): container finished" podID="316f0cbe-f528-4781-80c7-faddf7ab1d8e" containerID="81b1c024178199f3c806cec97e539be3939e08567ff34ceff48442aa903f7c53" exitCode=2
May 26 16:23:24.906203 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:24.906057    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp" event={"ID":"316f0cbe-f528-4781-80c7-faddf7ab1d8e","Type":"ContainerDied","Data":"81b1c024178199f3c806cec97e539be3939e08567ff34ceff48442aa903f7c53"}
May 26 16:23:25.004991 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:25.004947    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a31452b4-6879-42fe-968e-30d96044ad36-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv\" (UID: \"a31452b4-6879-42fe-968e-30d96044ad36\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv"
May 26 16:23:25.007354 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:25.007326    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a31452b4-6879-42fe-968e-30d96044ad36-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv\" (UID: \"a31452b4-6879-42fe-968e-30d96044ad36\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv"
May 26 16:23:25.259655 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:25.259620    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv"
May 26 16:23:25.371410 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:25.371385    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp"
May 26 16:23:25.393277 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:25.393205    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv"]
May 26 16:23:25.397501 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:23:25.397476    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda31452b4_6879_42fe_968e_30d96044ad36.slice/crio-cd78570f30d4ec02b75055a77e517e7cf50aff8bbe10570e8250c117e454ab39 WatchSource:0}: Error finding container cd78570f30d4ec02b75055a77e517e7cf50aff8bbe10570e8250c117e454ab39: Status 404 returned error can't find the container with id cd78570f30d4ec02b75055a77e517e7cf50aff8bbe10570e8250c117e454ab39
May 26 16:23:25.408148 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:25.408125    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/316f0cbe-f528-4781-80c7-faddf7ab1d8e-proxy-tls\") pod \"316f0cbe-f528-4781-80c7-faddf7ab1d8e\" (UID: \"316f0cbe-f528-4781-80c7-faddf7ab1d8e\") "
May 26 16:23:25.408265 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:25.408178    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28zcf\" (UniqueName: \"kubernetes.io/projected/316f0cbe-f528-4781-80c7-faddf7ab1d8e-kube-api-access-28zcf\") pod \"316f0cbe-f528-4781-80c7-faddf7ab1d8e\" (UID: \"316f0cbe-f528-4781-80c7-faddf7ab1d8e\") "
May 26 16:23:25.408332 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:25.408311    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/316f0cbe-f528-4781-80c7-faddf7ab1d8e-kserve-provision-location\") pod \"316f0cbe-f528-4781-80c7-faddf7ab1d8e\" (UID: \"316f0cbe-f528-4781-80c7-faddf7ab1d8e\") "
May 26 16:23:25.408395 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:25.408352    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/316f0cbe-f528-4781-80c7-faddf7ab1d8e-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\") pod \"316f0cbe-f528-4781-80c7-faddf7ab1d8e\" (UID: \"316f0cbe-f528-4781-80c7-faddf7ab1d8e\") "
May 26 16:23:25.408700 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:25.408672    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/316f0cbe-f528-4781-80c7-faddf7ab1d8e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "316f0cbe-f528-4781-80c7-faddf7ab1d8e" (UID: "316f0cbe-f528-4781-80c7-faddf7ab1d8e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:23:25.408803 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:25.408776    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/316f0cbe-f528-4781-80c7-faddf7ab1d8e-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config") pod "316f0cbe-f528-4781-80c7-faddf7ab1d8e" (UID: "316f0cbe-f528-4781-80c7-faddf7ab1d8e"). InnerVolumeSpecName "isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:23:25.410296 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:25.410270    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/316f0cbe-f528-4781-80c7-faddf7ab1d8e-kube-api-access-28zcf" (OuterVolumeSpecName: "kube-api-access-28zcf") pod "316f0cbe-f528-4781-80c7-faddf7ab1d8e" (UID: "316f0cbe-f528-4781-80c7-faddf7ab1d8e"). InnerVolumeSpecName "kube-api-access-28zcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:23:25.410558 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:25.410539    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/316f0cbe-f528-4781-80c7-faddf7ab1d8e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "316f0cbe-f528-4781-80c7-faddf7ab1d8e" (UID: "316f0cbe-f528-4781-80c7-faddf7ab1d8e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:23:25.508950 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:25.508910    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/316f0cbe-f528-4781-80c7-faddf7ab1d8e-proxy-tls\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:23:25.508950 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:25.508944    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28zcf\" (UniqueName: \"kubernetes.io/projected/316f0cbe-f528-4781-80c7-faddf7ab1d8e-kube-api-access-28zcf\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:23:25.508950 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:25.508954    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/316f0cbe-f528-4781-80c7-faddf7ab1d8e-kserve-provision-location\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:23:25.509162 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:25.508964    2572 reconciler_common.go:299] "Volume detached for volume \"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/316f0cbe-f528-4781-80c7-faddf7ab1d8e-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:23:25.911045 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:25.911005    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv" event={"ID":"a31452b4-6879-42fe-968e-30d96044ad36","Type":"ContainerStarted","Data":"01b7208869af8773aaf6dd55022f0f09437941369d21fe2895914a461571a3cc"}
May 26 16:23:25.911251 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:25.911054    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv" event={"ID":"a31452b4-6879-42fe-968e-30d96044ad36","Type":"ContainerStarted","Data":"cd78570f30d4ec02b75055a77e517e7cf50aff8bbe10570e8250c117e454ab39"}
May 26 16:23:25.912781 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:25.912751    2572 generic.go:358] "Generic (PLEG): container finished" podID="316f0cbe-f528-4781-80c7-faddf7ab1d8e" containerID="513508ac08295b0f16df9dec30bad69e67cbfeacb75f71e7b365e521fc2f4c15" exitCode=0
May 26 16:23:25.912925 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:25.912825    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp" event={"ID":"316f0cbe-f528-4781-80c7-faddf7ab1d8e","Type":"ContainerDied","Data":"513508ac08295b0f16df9dec30bad69e67cbfeacb75f71e7b365e521fc2f4c15"}
May 26 16:23:25.912925 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:25.912846    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp"
May 26 16:23:25.912925 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:25.912861    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp" event={"ID":"316f0cbe-f528-4781-80c7-faddf7ab1d8e","Type":"ContainerDied","Data":"ff5058ca68e519436c22b5287384acfea291642d017a0fd58b4ee2711ea916f5"}
May 26 16:23:25.912925 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:25.912877    2572 scope.go:117] "RemoveContainer" containerID="81b1c024178199f3c806cec97e539be3939e08567ff34ceff48442aa903f7c53"
May 26 16:23:25.920805 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:25.920695    2572 scope.go:117] "RemoveContainer" containerID="513508ac08295b0f16df9dec30bad69e67cbfeacb75f71e7b365e521fc2f4c15"
May 26 16:23:25.928973 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:25.928948    2572 scope.go:117] "RemoveContainer" containerID="7bf80c19dbf37a5285c882f484f251e2747632059469ed59e7c1aff234a616c4"
May 26 16:23:25.936407 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:25.936384    2572 scope.go:117] "RemoveContainer" containerID="81b1c024178199f3c806cec97e539be3939e08567ff34ceff48442aa903f7c53"
May 26 16:23:25.936736 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:23:25.936717    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81b1c024178199f3c806cec97e539be3939e08567ff34ceff48442aa903f7c53\": container with ID starting with 81b1c024178199f3c806cec97e539be3939e08567ff34ceff48442aa903f7c53 not found: ID does not exist" containerID="81b1c024178199f3c806cec97e539be3939e08567ff34ceff48442aa903f7c53"
May 26 16:23:25.936809 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:25.936746    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81b1c024178199f3c806cec97e539be3939e08567ff34ceff48442aa903f7c53"} err="failed to get container status \"81b1c024178199f3c806cec97e539be3939e08567ff34ceff48442aa903f7c53\": rpc error: code = NotFound desc = could not find container \"81b1c024178199f3c806cec97e539be3939e08567ff34ceff48442aa903f7c53\": container with ID starting with 81b1c024178199f3c806cec97e539be3939e08567ff34ceff48442aa903f7c53 not found: ID does not exist"
May 26 16:23:25.936809 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:25.936764    2572 scope.go:117] "RemoveContainer" containerID="513508ac08295b0f16df9dec30bad69e67cbfeacb75f71e7b365e521fc2f4c15"
May 26 16:23:25.937019 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:23:25.936999    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"513508ac08295b0f16df9dec30bad69e67cbfeacb75f71e7b365e521fc2f4c15\": container with ID starting with 513508ac08295b0f16df9dec30bad69e67cbfeacb75f71e7b365e521fc2f4c15 not found: ID does not exist" containerID="513508ac08295b0f16df9dec30bad69e67cbfeacb75f71e7b365e521fc2f4c15"
May 26 16:23:25.937056 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:25.937034    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"513508ac08295b0f16df9dec30bad69e67cbfeacb75f71e7b365e521fc2f4c15"} err="failed to get container status \"513508ac08295b0f16df9dec30bad69e67cbfeacb75f71e7b365e521fc2f4c15\": rpc error: code = NotFound desc = could not find container \"513508ac08295b0f16df9dec30bad69e67cbfeacb75f71e7b365e521fc2f4c15\": container with ID starting with 513508ac08295b0f16df9dec30bad69e67cbfeacb75f71e7b365e521fc2f4c15 not found: ID does not exist"
May 26 16:23:25.937056 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:25.937053    2572 scope.go:117] "RemoveContainer" containerID="7bf80c19dbf37a5285c882f484f251e2747632059469ed59e7c1aff234a616c4"
May 26 16:23:25.937331 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:23:25.937311    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bf80c19dbf37a5285c882f484f251e2747632059469ed59e7c1aff234a616c4\": container with ID starting with 7bf80c19dbf37a5285c882f484f251e2747632059469ed59e7c1aff234a616c4 not found: ID does not exist" containerID="7bf80c19dbf37a5285c882f484f251e2747632059469ed59e7c1aff234a616c4"
May 26 16:23:25.937427 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:25.937334    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bf80c19dbf37a5285c882f484f251e2747632059469ed59e7c1aff234a616c4"} err="failed to get container status \"7bf80c19dbf37a5285c882f484f251e2747632059469ed59e7c1aff234a616c4\": rpc error: code = NotFound desc = could not find container \"7bf80c19dbf37a5285c882f484f251e2747632059469ed59e7c1aff234a616c4\": container with ID starting with 7bf80c19dbf37a5285c882f484f251e2747632059469ed59e7c1aff234a616c4 not found: ID does not exist"
May 26 16:23:25.941934 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:25.941910    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp"]
May 26 16:23:25.945934 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:25.945913    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-6nfdp"]
May 26 16:23:26.079152 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:26.079121    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="316f0cbe-f528-4781-80c7-faddf7ab1d8e" path="/var/lib/kubelet/pods/316f0cbe-f528-4781-80c7-faddf7ab1d8e/volumes"
May 26 16:23:29.925508 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:29.925473    2572 generic.go:358] "Generic (PLEG): container finished" podID="a31452b4-6879-42fe-968e-30d96044ad36" containerID="01b7208869af8773aaf6dd55022f0f09437941369d21fe2895914a461571a3cc" exitCode=0
May 26 16:23:29.925983 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:29.925551    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv" event={"ID":"a31452b4-6879-42fe-968e-30d96044ad36","Type":"ContainerDied","Data":"01b7208869af8773aaf6dd55022f0f09437941369d21fe2895914a461571a3cc"}
May 26 16:23:30.930690 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:30.930654    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv" event={"ID":"a31452b4-6879-42fe-968e-30d96044ad36","Type":"ContainerStarted","Data":"e2b9adc924d7286e293225121270f9a3e3f4bfb7a36c447f0175bb1e358e3f26"}
May 26 16:23:30.930690 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:30.930694    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv" event={"ID":"a31452b4-6879-42fe-968e-30d96044ad36","Type":"ContainerStarted","Data":"b887fd04c5810f7852827cbb4f39b434a208882508d2cf953b50678cfd48f1c8"}
May 26 16:23:30.931266 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:30.931045    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv"
May 26 16:23:30.931266 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:30.931136    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv"
May 26 16:23:30.932480 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:30.932442    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv" podUID="a31452b4-6879-42fe-968e-30d96044ad36" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
May 26 16:23:30.949316 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:30.949263    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv" podStartSLOduration=6.949243631 podStartE2EDuration="6.949243631s" podCreationTimestamp="2026-05-26 16:23:24 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:23:30.947063207 +0000 UTC m=+1327.422790584" watchObservedRunningTime="2026-05-26 16:23:30.949243631 +0000 UTC m=+1327.424970984"
May 26 16:23:31.933498 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:31.933459    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv" podUID="a31452b4-6879-42fe-968e-30d96044ad36" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
May 26 16:23:36.938714 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:36.938684    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv"
May 26 16:23:36.939188 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:36.939154    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv" podUID="a31452b4-6879-42fe-968e-30d96044ad36" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
May 26 16:23:46.940569 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:46.940495    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv"
May 26 16:23:54.373934 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:54.373901    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv"]
May 26 16:23:54.374347 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:54.374252    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv" podUID="a31452b4-6879-42fe-968e-30d96044ad36" containerName="kube-rbac-proxy" containerID="cri-o://e2b9adc924d7286e293225121270f9a3e3f4bfb7a36c447f0175bb1e358e3f26" gracePeriod=30
May 26 16:23:54.374417 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:54.374245    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv" podUID="a31452b4-6879-42fe-968e-30d96044ad36" containerName="kserve-container" containerID="cri-o://b887fd04c5810f7852827cbb4f39b434a208882508d2cf953b50678cfd48f1c8" gracePeriod=30
May 26 16:23:54.472876 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:54.472832    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq"]
May 26 16:23:54.473244 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:54.473226    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="316f0cbe-f528-4781-80c7-faddf7ab1d8e" containerName="kserve-container"
May 26 16:23:54.473301 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:54.473248    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="316f0cbe-f528-4781-80c7-faddf7ab1d8e" containerName="kserve-container"
May 26 16:23:54.473301 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:54.473274    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="316f0cbe-f528-4781-80c7-faddf7ab1d8e" containerName="kube-rbac-proxy"
May 26 16:23:54.473301 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:54.473284    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="316f0cbe-f528-4781-80c7-faddf7ab1d8e" containerName="kube-rbac-proxy"
May 26 16:23:54.473301 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:54.473295    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="316f0cbe-f528-4781-80c7-faddf7ab1d8e" containerName="storage-initializer"
May 26 16:23:54.473439 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:54.473305    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="316f0cbe-f528-4781-80c7-faddf7ab1d8e" containerName="storage-initializer"
May 26 16:23:54.473439 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:54.473377    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="316f0cbe-f528-4781-80c7-faddf7ab1d8e" containerName="kserve-container"
May 26 16:23:54.473439 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:54.473390    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="316f0cbe-f528-4781-80c7-faddf7ab1d8e" containerName="kube-rbac-proxy"
May 26 16:23:54.475505 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:54.475480    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq"
May 26 16:23:54.477471 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:54.477451    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
May 26 16:23:54.477593 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:54.477482    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
May 26 16:23:54.484708 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:54.484679    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq"]
May 26 16:23:54.632494 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:54.632396    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99d997d5-89c5-4f34-a429-8c82fd08fa34-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq\" (UID: \"99d997d5-89c5-4f34-a429-8c82fd08fa34\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq"
May 26 16:23:54.632494 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:54.632444    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99d997d5-89c5-4f34-a429-8c82fd08fa34-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq\" (UID: \"99d997d5-89c5-4f34-a429-8c82fd08fa34\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq"
May 26 16:23:54.632494 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:54.632465    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99d997d5-89c5-4f34-a429-8c82fd08fa34-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq\" (UID: \"99d997d5-89c5-4f34-a429-8c82fd08fa34\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq"
May 26 16:23:54.632795 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:54.632550    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8cgk\" (UniqueName: \"kubernetes.io/projected/99d997d5-89c5-4f34-a429-8c82fd08fa34-kube-api-access-v8cgk\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq\" (UID: \"99d997d5-89c5-4f34-a429-8c82fd08fa34\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq"
May 26 16:23:54.733492 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:54.733439    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99d997d5-89c5-4f34-a429-8c82fd08fa34-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq\" (UID: \"99d997d5-89c5-4f34-a429-8c82fd08fa34\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq"
May 26 16:23:54.733492 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:54.733498    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99d997d5-89c5-4f34-a429-8c82fd08fa34-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq\" (UID: \"99d997d5-89c5-4f34-a429-8c82fd08fa34\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq"
May 26 16:23:54.733781 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:54.733532    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99d997d5-89c5-4f34-a429-8c82fd08fa34-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq\" (UID: \"99d997d5-89c5-4f34-a429-8c82fd08fa34\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq"
May 26 16:23:54.733781 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:54.733563    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-v8cgk\" (UniqueName: \"kubernetes.io/projected/99d997d5-89c5-4f34-a429-8c82fd08fa34-kube-api-access-v8cgk\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq\" (UID: \"99d997d5-89c5-4f34-a429-8c82fd08fa34\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq"
May 26 16:23:54.734145 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:54.734116    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99d997d5-89c5-4f34-a429-8c82fd08fa34-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq\" (UID: \"99d997d5-89c5-4f34-a429-8c82fd08fa34\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq"
May 26 16:23:54.734254 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:54.734209    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99d997d5-89c5-4f34-a429-8c82fd08fa34-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq\" (UID: \"99d997d5-89c5-4f34-a429-8c82fd08fa34\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq"
May 26 16:23:54.736184 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:54.736154    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99d997d5-89c5-4f34-a429-8c82fd08fa34-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq\" (UID: \"99d997d5-89c5-4f34-a429-8c82fd08fa34\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq"
May 26 16:23:54.742885 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:54.742859    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8cgk\" (UniqueName: \"kubernetes.io/projected/99d997d5-89c5-4f34-a429-8c82fd08fa34-kube-api-access-v8cgk\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq\" (UID: \"99d997d5-89c5-4f34-a429-8c82fd08fa34\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq"
May 26 16:23:54.787424 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:54.787384    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq"
May 26 16:23:54.921173 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:54.921055    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq"]
May 26 16:23:54.942726 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:23:54.942691    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99d997d5_89c5_4f34_a429_8c82fd08fa34.slice/crio-fcd56fc89b73145115fcf2a2cc16c633bc630d64cd462251233f4d482078fc65 WatchSource:0}: Error finding container fcd56fc89b73145115fcf2a2cc16c633bc630d64cd462251233f4d482078fc65: Status 404 returned error can't find the container with id fcd56fc89b73145115fcf2a2cc16c633bc630d64cd462251233f4d482078fc65
May 26 16:23:55.003214 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:55.003189    2572 generic.go:358] "Generic (PLEG): container finished" podID="a31452b4-6879-42fe-968e-30d96044ad36" containerID="e2b9adc924d7286e293225121270f9a3e3f4bfb7a36c447f0175bb1e358e3f26" exitCode=2
May 26 16:23:55.003214 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:55.003210    2572 generic.go:358] "Generic (PLEG): container finished" podID="a31452b4-6879-42fe-968e-30d96044ad36" containerID="b887fd04c5810f7852827cbb4f39b434a208882508d2cf953b50678cfd48f1c8" exitCode=0
May 26 16:23:55.003409 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:55.003277    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv" event={"ID":"a31452b4-6879-42fe-968e-30d96044ad36","Type":"ContainerDied","Data":"e2b9adc924d7286e293225121270f9a3e3f4bfb7a36c447f0175bb1e358e3f26"}
May 26 16:23:55.003409 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:55.003323    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv" event={"ID":"a31452b4-6879-42fe-968e-30d96044ad36","Type":"ContainerDied","Data":"b887fd04c5810f7852827cbb4f39b434a208882508d2cf953b50678cfd48f1c8"}
May 26 16:23:55.004447 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:55.004426    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq" event={"ID":"99d997d5-89c5-4f34-a429-8c82fd08fa34","Type":"ContainerStarted","Data":"fcd56fc89b73145115fcf2a2cc16c633bc630d64cd462251233f4d482078fc65"}
May 26 16:23:55.007667 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:55.007650    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv"
May 26 16:23:55.136717 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:55.136632    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a31452b4-6879-42fe-968e-30d96044ad36-kserve-provision-location\") pod \"a31452b4-6879-42fe-968e-30d96044ad36\" (UID: \"a31452b4-6879-42fe-968e-30d96044ad36\") "
May 26 16:23:55.136717 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:55.136693    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a31452b4-6879-42fe-968e-30d96044ad36-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\") pod \"a31452b4-6879-42fe-968e-30d96044ad36\" (UID: \"a31452b4-6879-42fe-968e-30d96044ad36\") "
May 26 16:23:55.136940 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:55.136750    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a31452b4-6879-42fe-968e-30d96044ad36-proxy-tls\") pod \"a31452b4-6879-42fe-968e-30d96044ad36\" (UID: \"a31452b4-6879-42fe-968e-30d96044ad36\") "
May 26 16:23:55.136940 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:55.136769    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gz49m\" (UniqueName: \"kubernetes.io/projected/a31452b4-6879-42fe-968e-30d96044ad36-kube-api-access-gz49m\") pod \"a31452b4-6879-42fe-968e-30d96044ad36\" (UID: \"a31452b4-6879-42fe-968e-30d96044ad36\") "
May 26 16:23:55.137049 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:55.136998    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a31452b4-6879-42fe-968e-30d96044ad36-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a31452b4-6879-42fe-968e-30d96044ad36" (UID: "a31452b4-6879-42fe-968e-30d96044ad36"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:23:55.137166 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:55.137129    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a31452b4-6879-42fe-968e-30d96044ad36-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config") pod "a31452b4-6879-42fe-968e-30d96044ad36" (UID: "a31452b4-6879-42fe-968e-30d96044ad36"). InnerVolumeSpecName "isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:23:55.138887 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:55.138865    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a31452b4-6879-42fe-968e-30d96044ad36-kube-api-access-gz49m" (OuterVolumeSpecName: "kube-api-access-gz49m") pod "a31452b4-6879-42fe-968e-30d96044ad36" (UID: "a31452b4-6879-42fe-968e-30d96044ad36"). InnerVolumeSpecName "kube-api-access-gz49m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:23:55.138887 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:55.138875    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a31452b4-6879-42fe-968e-30d96044ad36-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a31452b4-6879-42fe-968e-30d96044ad36" (UID: "a31452b4-6879-42fe-968e-30d96044ad36"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:23:55.238158 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:55.238112    2572 reconciler_common.go:299] "Volume detached for volume \"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a31452b4-6879-42fe-968e-30d96044ad36-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:23:55.238158 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:55.238145    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a31452b4-6879-42fe-968e-30d96044ad36-proxy-tls\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:23:55.238158 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:55.238162    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz49m\" (UniqueName: \"kubernetes.io/projected/a31452b4-6879-42fe-968e-30d96044ad36-kube-api-access-gz49m\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:23:55.238158 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:55.238175    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a31452b4-6879-42fe-968e-30d96044ad36-kserve-provision-location\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:23:56.009268 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:56.009227    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv" event={"ID":"a31452b4-6879-42fe-968e-30d96044ad36","Type":"ContainerDied","Data":"cd78570f30d4ec02b75055a77e517e7cf50aff8bbe10570e8250c117e454ab39"}
May 26 16:23:56.009771 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:56.009277    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv"
May 26 16:23:56.009771 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:56.009282    2572 scope.go:117] "RemoveContainer" containerID="e2b9adc924d7286e293225121270f9a3e3f4bfb7a36c447f0175bb1e358e3f26"
May 26 16:23:56.010704 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:56.010659    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq" event={"ID":"99d997d5-89c5-4f34-a429-8c82fd08fa34","Type":"ContainerStarted","Data":"73d40daa8ac9071a1b5a8c14fcf83e17f1607aa4314e9a0121856ab5663a185e"}
May 26 16:23:56.019347 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:56.019326    2572 scope.go:117] "RemoveContainer" containerID="b887fd04c5810f7852827cbb4f39b434a208882508d2cf953b50678cfd48f1c8"
May 26 16:23:56.026195 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:56.026177    2572 scope.go:117] "RemoveContainer" containerID="01b7208869af8773aaf6dd55022f0f09437941369d21fe2895914a461571a3cc"
May 26 16:23:56.042837 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:56.042807    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv"]
May 26 16:23:56.053583 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:56.053557    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-559bf6989-75rxv"]
May 26 16:23:56.079991 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:56.079963    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a31452b4-6879-42fe-968e-30d96044ad36" path="/var/lib/kubelet/pods/a31452b4-6879-42fe-968e-30d96044ad36/volumes"
May 26 16:23:59.021471 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:59.021429    2572 generic.go:358] "Generic (PLEG): container finished" podID="99d997d5-89c5-4f34-a429-8c82fd08fa34" containerID="73d40daa8ac9071a1b5a8c14fcf83e17f1607aa4314e9a0121856ab5663a185e" exitCode=0
May 26 16:23:59.021859 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:23:59.021488    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq" event={"ID":"99d997d5-89c5-4f34-a429-8c82fd08fa34","Type":"ContainerDied","Data":"73d40daa8ac9071a1b5a8c14fcf83e17f1607aa4314e9a0121856ab5663a185e"}
May 26 16:24:00.027431 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:00.027396    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq" event={"ID":"99d997d5-89c5-4f34-a429-8c82fd08fa34","Type":"ContainerStarted","Data":"c32b491ea92367d43db3a178b945189987ae9fe03fe72cf90dbea525a828a5b3"}
May 26 16:24:00.027841 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:00.027437    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq" event={"ID":"99d997d5-89c5-4f34-a429-8c82fd08fa34","Type":"ContainerStarted","Data":"e1f2670c802c40663a71b1968d3aebf1e9088b922582c43a3d8beb9aa931f427"}
May 26 16:24:00.027841 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:00.027664    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq"
May 26 16:24:00.046152 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:00.046100    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq" podStartSLOduration=6.046085708 podStartE2EDuration="6.046085708s" podCreationTimestamp="2026-05-26 16:23:54 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:24:00.044829159 +0000 UTC m=+1356.520556509" watchObservedRunningTime="2026-05-26 16:24:00.046085708 +0000 UTC m=+1356.521813057"
May 26 16:24:01.031100 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:01.031071    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq"
May 26 16:24:07.040792 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:07.040761    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq"
May 26 16:24:37.044287 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:37.044252    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq"
May 26 16:24:44.543070 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:44.543036    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq"]
May 26 16:24:44.543560 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:44.543330    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq" podUID="99d997d5-89c5-4f34-a429-8c82fd08fa34" containerName="kserve-container" containerID="cri-o://e1f2670c802c40663a71b1968d3aebf1e9088b922582c43a3d8beb9aa931f427" gracePeriod=30
May 26 16:24:44.543560 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:44.543396    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq" podUID="99d997d5-89c5-4f34-a429-8c82fd08fa34" containerName="kube-rbac-proxy" containerID="cri-o://c32b491ea92367d43db3a178b945189987ae9fe03fe72cf90dbea525a828a5b3" gracePeriod=30
May 26 16:24:44.627862 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:44.627823    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92"]
May 26 16:24:44.628170 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:44.628151    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a31452b4-6879-42fe-968e-30d96044ad36" containerName="storage-initializer"
May 26 16:24:44.628249 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:44.628174    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a31452b4-6879-42fe-968e-30d96044ad36" containerName="storage-initializer"
May 26 16:24:44.628249 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:44.628202    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a31452b4-6879-42fe-968e-30d96044ad36" containerName="kube-rbac-proxy"
May 26 16:24:44.628249 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:44.628211    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a31452b4-6879-42fe-968e-30d96044ad36" containerName="kube-rbac-proxy"
May 26 16:24:44.628249 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:44.628229    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a31452b4-6879-42fe-968e-30d96044ad36" containerName="kserve-container"
May 26 16:24:44.628249 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:44.628238    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a31452b4-6879-42fe-968e-30d96044ad36" containerName="kserve-container"
May 26 16:24:44.628557 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:44.628307    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="a31452b4-6879-42fe-968e-30d96044ad36" containerName="kube-rbac-proxy"
May 26 16:24:44.628557 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:44.628320    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="a31452b4-6879-42fe-968e-30d96044ad36" containerName="kserve-container"
May 26 16:24:44.631155 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:44.631132    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92"
May 26 16:24:44.633299 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:44.633269    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
May 26 16:24:44.633395 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:44.633310    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
May 26 16:24:44.641763 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:44.641733    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92"]
May 26 16:24:44.707023 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:44.706981    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/baa8854a-095a-43c1-b906-d13a55336c01-isvc-sklearn-mcp-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92\" (UID: \"baa8854a-095a-43c1-b906-d13a55336c01\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92"
May 26 16:24:44.707023 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:44.707028    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/baa8854a-095a-43c1-b906-d13a55336c01-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92\" (UID: \"baa8854a-095a-43c1-b906-d13a55336c01\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92"
May 26 16:24:44.707231 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:44.707154    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/baa8854a-095a-43c1-b906-d13a55336c01-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92\" (UID: \"baa8854a-095a-43c1-b906-d13a55336c01\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92"
May 26 16:24:44.707231 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:44.707185    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llkfc\" (UniqueName: \"kubernetes.io/projected/baa8854a-095a-43c1-b906-d13a55336c01-kube-api-access-llkfc\") pod \"isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92\" (UID: \"baa8854a-095a-43c1-b906-d13a55336c01\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92"
May 26 16:24:44.808080 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:44.807984    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/baa8854a-095a-43c1-b906-d13a55336c01-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92\" (UID: \"baa8854a-095a-43c1-b906-d13a55336c01\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92"
May 26 16:24:44.808080 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:44.808024    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-llkfc\" (UniqueName: \"kubernetes.io/projected/baa8854a-095a-43c1-b906-d13a55336c01-kube-api-access-llkfc\") pod \"isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92\" (UID: \"baa8854a-095a-43c1-b906-d13a55336c01\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92"
May 26 16:24:44.808080 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:44.808067    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/baa8854a-095a-43c1-b906-d13a55336c01-isvc-sklearn-mcp-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92\" (UID: \"baa8854a-095a-43c1-b906-d13a55336c01\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92"
May 26 16:24:44.808332 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:44.808090    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/baa8854a-095a-43c1-b906-d13a55336c01-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92\" (UID: \"baa8854a-095a-43c1-b906-d13a55336c01\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92"
May 26 16:24:44.808459 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:44.808437    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/baa8854a-095a-43c1-b906-d13a55336c01-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92\" (UID: \"baa8854a-095a-43c1-b906-d13a55336c01\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92"
May 26 16:24:44.808895 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:44.808867    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/baa8854a-095a-43c1-b906-d13a55336c01-isvc-sklearn-mcp-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92\" (UID: \"baa8854a-095a-43c1-b906-d13a55336c01\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92"
May 26 16:24:44.810685 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:44.810663    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/baa8854a-095a-43c1-b906-d13a55336c01-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92\" (UID: \"baa8854a-095a-43c1-b906-d13a55336c01\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92"
May 26 16:24:44.817353 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:44.817325    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llkfc\" (UniqueName: \"kubernetes.io/projected/baa8854a-095a-43c1-b906-d13a55336c01-kube-api-access-llkfc\") pod \"isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92\" (UID: \"baa8854a-095a-43c1-b906-d13a55336c01\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92"
May 26 16:24:44.943635 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:44.943568    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92"
May 26 16:24:45.075504 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:45.075408    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92"]
May 26 16:24:45.078746 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:24:45.078720    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaa8854a_095a_43c1_b906_d13a55336c01.slice/crio-23037fad74b049c549c277e5b52ed7b8503cd37bb1762301c685701914598bba WatchSource:0}: Error finding container 23037fad74b049c549c277e5b52ed7b8503cd37bb1762301c685701914598bba: Status 404 returned error can't find the container with id 23037fad74b049c549c277e5b52ed7b8503cd37bb1762301c685701914598bba
May 26 16:24:45.157595 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:45.157563    2572 generic.go:358] "Generic (PLEG): container finished" podID="99d997d5-89c5-4f34-a429-8c82fd08fa34" containerID="c32b491ea92367d43db3a178b945189987ae9fe03fe72cf90dbea525a828a5b3" exitCode=2
May 26 16:24:45.157775 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:45.157633    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq" event={"ID":"99d997d5-89c5-4f34-a429-8c82fd08fa34","Type":"ContainerDied","Data":"c32b491ea92367d43db3a178b945189987ae9fe03fe72cf90dbea525a828a5b3"}
May 26 16:24:45.159024 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:45.159002    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92" event={"ID":"baa8854a-095a-43c1-b906-d13a55336c01","Type":"ContainerStarted","Data":"117db7777d902ca8b975a9d90bf222be5df31b099e48f30b51e4c4efce5995d0"}
May 26 16:24:45.159131 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:45.159029    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92" event={"ID":"baa8854a-095a-43c1-b906-d13a55336c01","Type":"ContainerStarted","Data":"23037fad74b049c549c277e5b52ed7b8503cd37bb1762301c685701914598bba"}
May 26 16:24:45.881051 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:45.881023    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq"
May 26 16:24:45.915637 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:45.915588    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99d997d5-89c5-4f34-a429-8c82fd08fa34-kserve-provision-location\") pod \"99d997d5-89c5-4f34-a429-8c82fd08fa34\" (UID: \"99d997d5-89c5-4f34-a429-8c82fd08fa34\") "
May 26 16:24:45.915793 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:45.915642    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8cgk\" (UniqueName: \"kubernetes.io/projected/99d997d5-89c5-4f34-a429-8c82fd08fa34-kube-api-access-v8cgk\") pod \"99d997d5-89c5-4f34-a429-8c82fd08fa34\" (UID: \"99d997d5-89c5-4f34-a429-8c82fd08fa34\") "
May 26 16:24:45.915793 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:45.915684    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99d997d5-89c5-4f34-a429-8c82fd08fa34-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\") pod \"99d997d5-89c5-4f34-a429-8c82fd08fa34\" (UID: \"99d997d5-89c5-4f34-a429-8c82fd08fa34\") "
May 26 16:24:45.915793 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:45.915709    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99d997d5-89c5-4f34-a429-8c82fd08fa34-proxy-tls\") pod \"99d997d5-89c5-4f34-a429-8c82fd08fa34\" (UID: \"99d997d5-89c5-4f34-a429-8c82fd08fa34\") "
May 26 16:24:45.915977 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:45.915943    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99d997d5-89c5-4f34-a429-8c82fd08fa34-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "99d997d5-89c5-4f34-a429-8c82fd08fa34" (UID: "99d997d5-89c5-4f34-a429-8c82fd08fa34"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:24:45.916103 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:45.915987    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99d997d5-89c5-4f34-a429-8c82fd08fa34-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config") pod "99d997d5-89c5-4f34-a429-8c82fd08fa34" (UID: "99d997d5-89c5-4f34-a429-8c82fd08fa34"). InnerVolumeSpecName "isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:24:45.917744 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:45.917714    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99d997d5-89c5-4f34-a429-8c82fd08fa34-kube-api-access-v8cgk" (OuterVolumeSpecName: "kube-api-access-v8cgk") pod "99d997d5-89c5-4f34-a429-8c82fd08fa34" (UID: "99d997d5-89c5-4f34-a429-8c82fd08fa34"). InnerVolumeSpecName "kube-api-access-v8cgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:24:45.917744 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:45.917728    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99d997d5-89c5-4f34-a429-8c82fd08fa34-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "99d997d5-89c5-4f34-a429-8c82fd08fa34" (UID: "99d997d5-89c5-4f34-a429-8c82fd08fa34"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:24:46.016581 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:46.016479    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99d997d5-89c5-4f34-a429-8c82fd08fa34-kserve-provision-location\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:24:46.016581 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:46.016523    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8cgk\" (UniqueName: \"kubernetes.io/projected/99d997d5-89c5-4f34-a429-8c82fd08fa34-kube-api-access-v8cgk\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:24:46.016581 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:46.016534    2572 reconciler_common.go:299] "Volume detached for volume \"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99d997d5-89c5-4f34-a429-8c82fd08fa34-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:24:46.016581 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:46.016544    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99d997d5-89c5-4f34-a429-8c82fd08fa34-proxy-tls\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:24:46.163855 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:46.163818    2572 generic.go:358] "Generic (PLEG): container finished" podID="99d997d5-89c5-4f34-a429-8c82fd08fa34" containerID="e1f2670c802c40663a71b1968d3aebf1e9088b922582c43a3d8beb9aa931f427" exitCode=0
May 26 16:24:46.164044 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:46.163907    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq"
May 26 16:24:46.164044 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:46.163907    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq" event={"ID":"99d997d5-89c5-4f34-a429-8c82fd08fa34","Type":"ContainerDied","Data":"e1f2670c802c40663a71b1968d3aebf1e9088b922582c43a3d8beb9aa931f427"}
May 26 16:24:46.164044 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:46.163948    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq" event={"ID":"99d997d5-89c5-4f34-a429-8c82fd08fa34","Type":"ContainerDied","Data":"fcd56fc89b73145115fcf2a2cc16c633bc630d64cd462251233f4d482078fc65"}
May 26 16:24:46.164044 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:46.163964    2572 scope.go:117] "RemoveContainer" containerID="c32b491ea92367d43db3a178b945189987ae9fe03fe72cf90dbea525a828a5b3"
May 26 16:24:46.172561 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:46.172465    2572 scope.go:117] "RemoveContainer" containerID="e1f2670c802c40663a71b1968d3aebf1e9088b922582c43a3d8beb9aa931f427"
May 26 16:24:46.178543 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:46.178515    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq"]
May 26 16:24:46.180558 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:46.180542    2572 scope.go:117] "RemoveContainer" containerID="73d40daa8ac9071a1b5a8c14fcf83e17f1607aa4314e9a0121856ab5663a185e"
May 26 16:24:46.184027 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:46.184004    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-x6fjq"]
May 26 16:24:46.187982 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:46.187963    2572 scope.go:117] "RemoveContainer" containerID="c32b491ea92367d43db3a178b945189987ae9fe03fe72cf90dbea525a828a5b3"
May 26 16:24:46.188245 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:24:46.188226    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c32b491ea92367d43db3a178b945189987ae9fe03fe72cf90dbea525a828a5b3\": container with ID starting with c32b491ea92367d43db3a178b945189987ae9fe03fe72cf90dbea525a828a5b3 not found: ID does not exist" containerID="c32b491ea92367d43db3a178b945189987ae9fe03fe72cf90dbea525a828a5b3"
May 26 16:24:46.188331 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:46.188259    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c32b491ea92367d43db3a178b945189987ae9fe03fe72cf90dbea525a828a5b3"} err="failed to get container status \"c32b491ea92367d43db3a178b945189987ae9fe03fe72cf90dbea525a828a5b3\": rpc error: code = NotFound desc = could not find container \"c32b491ea92367d43db3a178b945189987ae9fe03fe72cf90dbea525a828a5b3\": container with ID starting with c32b491ea92367d43db3a178b945189987ae9fe03fe72cf90dbea525a828a5b3 not found: ID does not exist"
May 26 16:24:46.188331 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:46.188286    2572 scope.go:117] "RemoveContainer" containerID="e1f2670c802c40663a71b1968d3aebf1e9088b922582c43a3d8beb9aa931f427"
May 26 16:24:46.188509 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:24:46.188493    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1f2670c802c40663a71b1968d3aebf1e9088b922582c43a3d8beb9aa931f427\": container with ID starting with e1f2670c802c40663a71b1968d3aebf1e9088b922582c43a3d8beb9aa931f427 not found: ID does not exist" containerID="e1f2670c802c40663a71b1968d3aebf1e9088b922582c43a3d8beb9aa931f427"
May 26 16:24:46.188547 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:46.188515    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1f2670c802c40663a71b1968d3aebf1e9088b922582c43a3d8beb9aa931f427"} err="failed to get container status \"e1f2670c802c40663a71b1968d3aebf1e9088b922582c43a3d8beb9aa931f427\": rpc error: code = NotFound desc = could not find container \"e1f2670c802c40663a71b1968d3aebf1e9088b922582c43a3d8beb9aa931f427\": container with ID starting with e1f2670c802c40663a71b1968d3aebf1e9088b922582c43a3d8beb9aa931f427 not found: ID does not exist"
May 26 16:24:46.188547 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:46.188540    2572 scope.go:117] "RemoveContainer" containerID="73d40daa8ac9071a1b5a8c14fcf83e17f1607aa4314e9a0121856ab5663a185e"
May 26 16:24:46.188786 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:24:46.188767    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73d40daa8ac9071a1b5a8c14fcf83e17f1607aa4314e9a0121856ab5663a185e\": container with ID starting with 73d40daa8ac9071a1b5a8c14fcf83e17f1607aa4314e9a0121856ab5663a185e not found: ID does not exist" containerID="73d40daa8ac9071a1b5a8c14fcf83e17f1607aa4314e9a0121856ab5663a185e"
May 26 16:24:46.188874 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:46.188789    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73d40daa8ac9071a1b5a8c14fcf83e17f1607aa4314e9a0121856ab5663a185e"} err="failed to get container status \"73d40daa8ac9071a1b5a8c14fcf83e17f1607aa4314e9a0121856ab5663a185e\": rpc error: code = NotFound desc = could not find container \"73d40daa8ac9071a1b5a8c14fcf83e17f1607aa4314e9a0121856ab5663a185e\": container with ID starting with 73d40daa8ac9071a1b5a8c14fcf83e17f1607aa4314e9a0121856ab5663a185e not found: ID does not exist"
May 26 16:24:48.079026 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:48.078992    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99d997d5-89c5-4f34-a429-8c82fd08fa34" path="/var/lib/kubelet/pods/99d997d5-89c5-4f34-a429-8c82fd08fa34/volumes"
May 26 16:24:49.175304 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:49.175216    2572 generic.go:358] "Generic (PLEG): container finished" podID="baa8854a-095a-43c1-b906-d13a55336c01" containerID="117db7777d902ca8b975a9d90bf222be5df31b099e48f30b51e4c4efce5995d0" exitCode=0
May 26 16:24:49.175696 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:49.175296    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92" event={"ID":"baa8854a-095a-43c1-b906-d13a55336c01","Type":"ContainerDied","Data":"117db7777d902ca8b975a9d90bf222be5df31b099e48f30b51e4c4efce5995d0"}
May 26 16:24:50.180520 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:50.180485    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92" event={"ID":"baa8854a-095a-43c1-b906-d13a55336c01","Type":"ContainerStarted","Data":"0bc13be0215258934fdd7199f5fb86d2ef5ae77641dd43b5e835525805997932"}
May 26 16:24:52.189250 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:52.189222    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92" event={"ID":"baa8854a-095a-43c1-b906-d13a55336c01","Type":"ContainerStarted","Data":"b89cd782beb1394153db4320bdbc41ea4e5c02e2d70c5388a8a2429b1a15968c"}
May 26 16:24:53.194451 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:53.194415    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92" event={"ID":"baa8854a-095a-43c1-b906-d13a55336c01","Type":"ContainerStarted","Data":"a2ababd01c04d8a2c19dc2ccf243aa07c0cef5b9455d87426445abb938e57019"}
May 26 16:24:53.194843 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:53.194645    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92"
May 26 16:24:53.217198 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:53.217154    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92" podStartSLOduration=6.349257204 podStartE2EDuration="9.217138976s" podCreationTimestamp="2026-05-26 16:24:44 +0000 UTC" firstStartedPulling="2026-05-26 16:24:49.236857558 +0000 UTC m=+1405.712584890" lastFinishedPulling="2026-05-26 16:24:52.104739334 +0000 UTC m=+1408.580466662" observedRunningTime="2026-05-26 16:24:53.216593128 +0000 UTC m=+1409.692320475" watchObservedRunningTime="2026-05-26 16:24:53.217138976 +0000 UTC m=+1409.692866327"
May 26 16:24:54.197110 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:54.197078    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92"
May 26 16:24:54.197110 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:24:54.197115    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92"
May 26 16:25:00.205655 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:25:00.205625    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92"
May 26 16:25:20.207212 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:25:20.207126    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92" podUID="baa8854a-095a-43c1-b906-d13a55336c01" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.30:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.30:8080: connect: connection refused"
May 26 16:25:30.207892 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:25:30.207858    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92"
May 26 16:26:00.209114 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:00.209084    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92"
May 26 16:26:04.676732 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:04.676699    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92"]
May 26 16:26:04.677100 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:04.677058    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92" podUID="baa8854a-095a-43c1-b906-d13a55336c01" containerName="kserve-container" containerID="cri-o://0bc13be0215258934fdd7199f5fb86d2ef5ae77641dd43b5e835525805997932" gracePeriod=30
May 26 16:26:04.677175 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:04.677122    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92" podUID="baa8854a-095a-43c1-b906-d13a55336c01" containerName="kube-rbac-proxy" containerID="cri-o://a2ababd01c04d8a2c19dc2ccf243aa07c0cef5b9455d87426445abb938e57019" gracePeriod=30
May 26 16:26:04.677228 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:04.677166    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92" podUID="baa8854a-095a-43c1-b906-d13a55336c01" containerName="kserve-agent" containerID="cri-o://b89cd782beb1394153db4320bdbc41ea4e5c02e2d70c5388a8a2429b1a15968c" gracePeriod=30
May 26 16:26:04.763093 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:04.763052    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-r58lh"]
May 26 16:26:04.763406 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:04.763389    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99d997d5-89c5-4f34-a429-8c82fd08fa34" containerName="kube-rbac-proxy"
May 26 16:26:04.763490 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:04.763409    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="99d997d5-89c5-4f34-a429-8c82fd08fa34" containerName="kube-rbac-proxy"
May 26 16:26:04.763490 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:04.763446    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99d997d5-89c5-4f34-a429-8c82fd08fa34" containerName="kserve-container"
May 26 16:26:04.763490 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:04.763456    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="99d997d5-89c5-4f34-a429-8c82fd08fa34" containerName="kserve-container"
May 26 16:26:04.763490 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:04.763470    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99d997d5-89c5-4f34-a429-8c82fd08fa34" containerName="storage-initializer"
May 26 16:26:04.763490 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:04.763479    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="99d997d5-89c5-4f34-a429-8c82fd08fa34" containerName="storage-initializer"
May 26 16:26:04.763763 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:04.763545    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="99d997d5-89c5-4f34-a429-8c82fd08fa34" containerName="kserve-container"
May 26 16:26:04.763763 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:04.763559    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="99d997d5-89c5-4f34-a429-8c82fd08fa34" containerName="kube-rbac-proxy"
May 26 16:26:04.766732 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:04.766710    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-r58lh"
May 26 16:26:04.768769 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:04.768748    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
May 26 16:26:04.768892 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:04.768874    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-predictor-serving-cert\""
May 26 16:26:04.777463 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:04.777435    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-r58lh"]
May 26 16:26:04.826651 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:04.826570    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b5428e3-eff7-486d-9d38-163aa8c258df-kserve-provision-location\") pod \"isvc-paddle-predictor-6b8b7cfb4b-r58lh\" (UID: \"5b5428e3-eff7-486d-9d38-163aa8c258df\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-r58lh"
May 26 16:26:04.826651 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:04.826636    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b5428e3-eff7-486d-9d38-163aa8c258df-proxy-tls\") pod \"isvc-paddle-predictor-6b8b7cfb4b-r58lh\" (UID: \"5b5428e3-eff7-486d-9d38-163aa8c258df\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-r58lh"
May 26 16:26:04.826651 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:04.826656    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kzsz\" (UniqueName: \"kubernetes.io/projected/5b5428e3-eff7-486d-9d38-163aa8c258df-kube-api-access-8kzsz\") pod \"isvc-paddle-predictor-6b8b7cfb4b-r58lh\" (UID: \"5b5428e3-eff7-486d-9d38-163aa8c258df\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-r58lh"
May 26 16:26:04.826869 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:04.826696    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-paddle-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b5428e3-eff7-486d-9d38-163aa8c258df-isvc-paddle-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6b8b7cfb4b-r58lh\" (UID: \"5b5428e3-eff7-486d-9d38-163aa8c258df\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-r58lh"
May 26 16:26:04.928160 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:04.928069    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b5428e3-eff7-486d-9d38-163aa8c258df-kserve-provision-location\") pod \"isvc-paddle-predictor-6b8b7cfb4b-r58lh\" (UID: \"5b5428e3-eff7-486d-9d38-163aa8c258df\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-r58lh"
May 26 16:26:04.928160 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:04.928123    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b5428e3-eff7-486d-9d38-163aa8c258df-proxy-tls\") pod \"isvc-paddle-predictor-6b8b7cfb4b-r58lh\" (UID: \"5b5428e3-eff7-486d-9d38-163aa8c258df\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-r58lh"
May 26 16:26:04.928160 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:04.928151    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8kzsz\" (UniqueName: \"kubernetes.io/projected/5b5428e3-eff7-486d-9d38-163aa8c258df-kube-api-access-8kzsz\") pod \"isvc-paddle-predictor-6b8b7cfb4b-r58lh\" (UID: \"5b5428e3-eff7-486d-9d38-163aa8c258df\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-r58lh"
May 26 16:26:04.928401 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:04.928206    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-paddle-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b5428e3-eff7-486d-9d38-163aa8c258df-isvc-paddle-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6b8b7cfb4b-r58lh\" (UID: \"5b5428e3-eff7-486d-9d38-163aa8c258df\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-r58lh"
May 26 16:26:04.928401 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:26:04.928289    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-predictor-serving-cert: secret "isvc-paddle-predictor-serving-cert" not found
May 26 16:26:04.928401 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:26:04.928355    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b5428e3-eff7-486d-9d38-163aa8c258df-proxy-tls podName:5b5428e3-eff7-486d-9d38-163aa8c258df nodeName:}" failed. No retries permitted until 2026-05-26 16:26:05.428331534 +0000 UTC m=+1481.904058879 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5b5428e3-eff7-486d-9d38-163aa8c258df-proxy-tls") pod "isvc-paddle-predictor-6b8b7cfb4b-r58lh" (UID: "5b5428e3-eff7-486d-9d38-163aa8c258df") : secret "isvc-paddle-predictor-serving-cert" not found
May 26 16:26:04.928540 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:04.928521    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b5428e3-eff7-486d-9d38-163aa8c258df-kserve-provision-location\") pod \"isvc-paddle-predictor-6b8b7cfb4b-r58lh\" (UID: \"5b5428e3-eff7-486d-9d38-163aa8c258df\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-r58lh"
May 26 16:26:04.928860 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:04.928843    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-paddle-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b5428e3-eff7-486d-9d38-163aa8c258df-isvc-paddle-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6b8b7cfb4b-r58lh\" (UID: \"5b5428e3-eff7-486d-9d38-163aa8c258df\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-r58lh"
May 26 16:26:04.936959 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:04.936934    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kzsz\" (UniqueName: \"kubernetes.io/projected/5b5428e3-eff7-486d-9d38-163aa8c258df-kube-api-access-8kzsz\") pod \"isvc-paddle-predictor-6b8b7cfb4b-r58lh\" (UID: \"5b5428e3-eff7-486d-9d38-163aa8c258df\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-r58lh"
May 26 16:26:05.200311 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:05.200205    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92" podUID="baa8854a-095a-43c1-b906-d13a55336c01" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
May 26 16:26:05.396977 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:05.396945    2572 generic.go:358] "Generic (PLEG): container finished" podID="baa8854a-095a-43c1-b906-d13a55336c01" containerID="a2ababd01c04d8a2c19dc2ccf243aa07c0cef5b9455d87426445abb938e57019" exitCode=2
May 26 16:26:05.397144 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:05.396988    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92" event={"ID":"baa8854a-095a-43c1-b906-d13a55336c01","Type":"ContainerDied","Data":"a2ababd01c04d8a2c19dc2ccf243aa07c0cef5b9455d87426445abb938e57019"}
May 26 16:26:05.432666 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:05.432596    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b5428e3-eff7-486d-9d38-163aa8c258df-proxy-tls\") pod \"isvc-paddle-predictor-6b8b7cfb4b-r58lh\" (UID: \"5b5428e3-eff7-486d-9d38-163aa8c258df\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-r58lh"
May 26 16:26:05.435138 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:05.435110    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b5428e3-eff7-486d-9d38-163aa8c258df-proxy-tls\") pod \"isvc-paddle-predictor-6b8b7cfb4b-r58lh\" (UID: \"5b5428e3-eff7-486d-9d38-163aa8c258df\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-r58lh"
May 26 16:26:05.676978 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:05.676933    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-r58lh"
May 26 16:26:05.804772 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:05.804736    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-r58lh"]
May 26 16:26:05.808123 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:26:05.808090    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b5428e3_eff7_486d_9d38_163aa8c258df.slice/crio-c0b1797942dc1402319ab6273c52b02933f1a0b3e58a8f6e7b5bb8d8bf39ad64 WatchSource:0}: Error finding container c0b1797942dc1402319ab6273c52b02933f1a0b3e58a8f6e7b5bb8d8bf39ad64: Status 404 returned error can't find the container with id c0b1797942dc1402319ab6273c52b02933f1a0b3e58a8f6e7b5bb8d8bf39ad64
May 26 16:26:06.400566 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:06.400529    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-r58lh" event={"ID":"5b5428e3-eff7-486d-9d38-163aa8c258df","Type":"ContainerStarted","Data":"142f86222bfe0537fa8218f1b194bc9fdf1aead11b460f4043ed94a54392a4fa"}
May 26 16:26:06.400566 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:06.400570    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-r58lh" event={"ID":"5b5428e3-eff7-486d-9d38-163aa8c258df","Type":"ContainerStarted","Data":"c0b1797942dc1402319ab6273c52b02933f1a0b3e58a8f6e7b5bb8d8bf39ad64"}
May 26 16:26:07.405619 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:07.405582    2572 generic.go:358] "Generic (PLEG): container finished" podID="baa8854a-095a-43c1-b906-d13a55336c01" containerID="0bc13be0215258934fdd7199f5fb86d2ef5ae77641dd43b5e835525805997932" exitCode=0
May 26 16:26:07.405971 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:07.405655    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92" event={"ID":"baa8854a-095a-43c1-b906-d13a55336c01","Type":"ContainerDied","Data":"0bc13be0215258934fdd7199f5fb86d2ef5ae77641dd43b5e835525805997932"}
May 26 16:26:10.200220 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:10.200171    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92" podUID="baa8854a-095a-43c1-b906-d13a55336c01" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
May 26 16:26:10.206462 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:10.206426    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92" podUID="baa8854a-095a-43c1-b906-d13a55336c01" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.30:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.30:8080: connect: connection refused"
May 26 16:26:11.418214 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:11.418171    2572 generic.go:358] "Generic (PLEG): container finished" podID="5b5428e3-eff7-486d-9d38-163aa8c258df" containerID="142f86222bfe0537fa8218f1b194bc9fdf1aead11b460f4043ed94a54392a4fa" exitCode=0
May 26 16:26:11.418654 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:11.418245    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-r58lh" event={"ID":"5b5428e3-eff7-486d-9d38-163aa8c258df","Type":"ContainerDied","Data":"142f86222bfe0537fa8218f1b194bc9fdf1aead11b460f4043ed94a54392a4fa"}
May 26 16:26:15.200645 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:15.200589    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92" podUID="baa8854a-095a-43c1-b906-d13a55336c01" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
May 26 16:26:15.201088 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:15.200738    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92"
May 26 16:26:20.200756 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:20.200705    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92" podUID="baa8854a-095a-43c1-b906-d13a55336c01" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
May 26 16:26:20.206310 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:20.206272    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92" podUID="baa8854a-095a-43c1-b906-d13a55336c01" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.30:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.30:8080: connect: connection refused"
May 26 16:26:23.459041 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:23.459006    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-r58lh" event={"ID":"5b5428e3-eff7-486d-9d38-163aa8c258df","Type":"ContainerStarted","Data":"b0eb8aa6e259ffe6197d7dc7c4ffd1050205055a6c95c216cb8d8b5fddb12c77"}
May 26 16:26:23.459419 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:23.459051    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-r58lh" event={"ID":"5b5428e3-eff7-486d-9d38-163aa8c258df","Type":"ContainerStarted","Data":"e5d37ec92403204fe03b75741300608f027021bb5d4cf5286fc52307501bae8b"}
May 26 16:26:23.459419 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:23.459302    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-r58lh"
May 26 16:26:23.459541 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:23.459458    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-r58lh"
May 26 16:26:23.460620 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:23.460583    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-r58lh" podUID="5b5428e3-eff7-486d-9d38-163aa8c258df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
May 26 16:26:23.476657 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:23.476614    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-r58lh" podStartSLOduration=7.582267529 podStartE2EDuration="19.476587101s" podCreationTimestamp="2026-05-26 16:26:04 +0000 UTC" firstStartedPulling="2026-05-26 16:26:11.419431602 +0000 UTC m=+1487.895158930" lastFinishedPulling="2026-05-26 16:26:23.313751167 +0000 UTC m=+1499.789478502" observedRunningTime="2026-05-26 16:26:23.474406472 +0000 UTC m=+1499.950133823" watchObservedRunningTime="2026-05-26 16:26:23.476587101 +0000 UTC m=+1499.952314450"
May 26 16:26:24.462635 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:24.462575    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-r58lh" podUID="5b5428e3-eff7-486d-9d38-163aa8c258df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
May 26 16:26:25.200770 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:25.200729    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92" podUID="baa8854a-095a-43c1-b906-d13a55336c01" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
May 26 16:26:29.466679 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:29.466647    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-r58lh"
May 26 16:26:29.467236 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:29.467209    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-r58lh" podUID="5b5428e3-eff7-486d-9d38-163aa8c258df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
May 26 16:26:30.201220 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:30.201176    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92" podUID="baa8854a-095a-43c1-b906-d13a55336c01" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
May 26 16:26:30.206614 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:30.206571    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92" podUID="baa8854a-095a-43c1-b906-d13a55336c01" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.30:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.30:8080: connect: connection refused"
May 26 16:26:30.206718 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:30.206705    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92"
May 26 16:26:34.868401 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:34.868376    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92"
May 26 16:26:34.987158 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:34.987073    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/baa8854a-095a-43c1-b906-d13a55336c01-isvc-sklearn-mcp-kube-rbac-proxy-sar-config\") pod \"baa8854a-095a-43c1-b906-d13a55336c01\" (UID: \"baa8854a-095a-43c1-b906-d13a55336c01\") "
May 26 16:26:34.987158 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:34.987123    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/baa8854a-095a-43c1-b906-d13a55336c01-kserve-provision-location\") pod \"baa8854a-095a-43c1-b906-d13a55336c01\" (UID: \"baa8854a-095a-43c1-b906-d13a55336c01\") "
May 26 16:26:34.987158 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:34.987160    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/baa8854a-095a-43c1-b906-d13a55336c01-proxy-tls\") pod \"baa8854a-095a-43c1-b906-d13a55336c01\" (UID: \"baa8854a-095a-43c1-b906-d13a55336c01\") "
May 26 16:26:34.987422 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:34.987176    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-llkfc\" (UniqueName: \"kubernetes.io/projected/baa8854a-095a-43c1-b906-d13a55336c01-kube-api-access-llkfc\") pod \"baa8854a-095a-43c1-b906-d13a55336c01\" (UID: \"baa8854a-095a-43c1-b906-d13a55336c01\") "
May 26 16:26:34.987514 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:34.987484    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baa8854a-095a-43c1-b906-d13a55336c01-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "baa8854a-095a-43c1-b906-d13a55336c01" (UID: "baa8854a-095a-43c1-b906-d13a55336c01"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:26:34.987578 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:34.987520    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baa8854a-095a-43c1-b906-d13a55336c01-isvc-sklearn-mcp-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-mcp-kube-rbac-proxy-sar-config") pod "baa8854a-095a-43c1-b906-d13a55336c01" (UID: "baa8854a-095a-43c1-b906-d13a55336c01"). InnerVolumeSpecName "isvc-sklearn-mcp-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:26:34.989225 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:34.989202    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baa8854a-095a-43c1-b906-d13a55336c01-kube-api-access-llkfc" (OuterVolumeSpecName: "kube-api-access-llkfc") pod "baa8854a-095a-43c1-b906-d13a55336c01" (UID: "baa8854a-095a-43c1-b906-d13a55336c01"). InnerVolumeSpecName "kube-api-access-llkfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:26:34.989269 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:34.989230    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/baa8854a-095a-43c1-b906-d13a55336c01-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "baa8854a-095a-43c1-b906-d13a55336c01" (UID: "baa8854a-095a-43c1-b906-d13a55336c01"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:26:35.087919 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:35.087884    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/baa8854a-095a-43c1-b906-d13a55336c01-proxy-tls\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:26:35.087919 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:35.087917    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llkfc\" (UniqueName: \"kubernetes.io/projected/baa8854a-095a-43c1-b906-d13a55336c01-kube-api-access-llkfc\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:26:35.088121 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:35.087933    2572 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/baa8854a-095a-43c1-b906-d13a55336c01-isvc-sklearn-mcp-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:26:35.088121 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:35.087947    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/baa8854a-095a-43c1-b906-d13a55336c01-kserve-provision-location\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:26:35.494556 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:35.494520    2572 generic.go:358] "Generic (PLEG): container finished" podID="baa8854a-095a-43c1-b906-d13a55336c01" containerID="b89cd782beb1394153db4320bdbc41ea4e5c02e2d70c5388a8a2429b1a15968c" exitCode=137
May 26 16:26:35.494759 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:35.494588    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92" event={"ID":"baa8854a-095a-43c1-b906-d13a55336c01","Type":"ContainerDied","Data":"b89cd782beb1394153db4320bdbc41ea4e5c02e2d70c5388a8a2429b1a15968c"}
May 26 16:26:35.494759 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:35.494632    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92" event={"ID":"baa8854a-095a-43c1-b906-d13a55336c01","Type":"ContainerDied","Data":"23037fad74b049c549c277e5b52ed7b8503cd37bb1762301c685701914598bba"}
May 26 16:26:35.494759 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:35.494648    2572 scope.go:117] "RemoveContainer" containerID="a2ababd01c04d8a2c19dc2ccf243aa07c0cef5b9455d87426445abb938e57019"
May 26 16:26:35.494759 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:35.494645    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92"
May 26 16:26:35.502496 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:35.502341    2572 scope.go:117] "RemoveContainer" containerID="b89cd782beb1394153db4320bdbc41ea4e5c02e2d70c5388a8a2429b1a15968c"
May 26 16:26:35.509427 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:35.509411    2572 scope.go:117] "RemoveContainer" containerID="0bc13be0215258934fdd7199f5fb86d2ef5ae77641dd43b5e835525805997932"
May 26 16:26:35.515080 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:35.515056    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92"]
May 26 16:26:35.516246 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:35.516226    2572 scope.go:117] "RemoveContainer" containerID="117db7777d902ca8b975a9d90bf222be5df31b099e48f30b51e4c4efce5995d0"
May 26 16:26:35.518906 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:35.518882    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6b9b4445f6-cnw92"]
May 26 16:26:35.523523 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:35.523507    2572 scope.go:117] "RemoveContainer" containerID="a2ababd01c04d8a2c19dc2ccf243aa07c0cef5b9455d87426445abb938e57019"
May 26 16:26:35.523794 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:26:35.523773    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2ababd01c04d8a2c19dc2ccf243aa07c0cef5b9455d87426445abb938e57019\": container with ID starting with a2ababd01c04d8a2c19dc2ccf243aa07c0cef5b9455d87426445abb938e57019 not found: ID does not exist" containerID="a2ababd01c04d8a2c19dc2ccf243aa07c0cef5b9455d87426445abb938e57019"
May 26 16:26:35.523851 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:35.523803    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2ababd01c04d8a2c19dc2ccf243aa07c0cef5b9455d87426445abb938e57019"} err="failed to get container status \"a2ababd01c04d8a2c19dc2ccf243aa07c0cef5b9455d87426445abb938e57019\": rpc error: code = NotFound desc = could not find container \"a2ababd01c04d8a2c19dc2ccf243aa07c0cef5b9455d87426445abb938e57019\": container with ID starting with a2ababd01c04d8a2c19dc2ccf243aa07c0cef5b9455d87426445abb938e57019 not found: ID does not exist"
May 26 16:26:35.523851 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:35.523823    2572 scope.go:117] "RemoveContainer" containerID="b89cd782beb1394153db4320bdbc41ea4e5c02e2d70c5388a8a2429b1a15968c"
May 26 16:26:35.524066 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:26:35.524049    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b89cd782beb1394153db4320bdbc41ea4e5c02e2d70c5388a8a2429b1a15968c\": container with ID starting with b89cd782beb1394153db4320bdbc41ea4e5c02e2d70c5388a8a2429b1a15968c not found: ID does not exist" containerID="b89cd782beb1394153db4320bdbc41ea4e5c02e2d70c5388a8a2429b1a15968c"
May 26 16:26:35.524131 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:35.524077    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b89cd782beb1394153db4320bdbc41ea4e5c02e2d70c5388a8a2429b1a15968c"} err="failed to get container status \"b89cd782beb1394153db4320bdbc41ea4e5c02e2d70c5388a8a2429b1a15968c\": rpc error: code = NotFound desc = could not find container \"b89cd782beb1394153db4320bdbc41ea4e5c02e2d70c5388a8a2429b1a15968c\": container with ID starting with b89cd782beb1394153db4320bdbc41ea4e5c02e2d70c5388a8a2429b1a15968c not found: ID does not exist"
May 26 16:26:35.524131 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:35.524101    2572 scope.go:117] "RemoveContainer" containerID="0bc13be0215258934fdd7199f5fb86d2ef5ae77641dd43b5e835525805997932"
May 26 16:26:35.524341 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:26:35.524321    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bc13be0215258934fdd7199f5fb86d2ef5ae77641dd43b5e835525805997932\": container with ID starting with 0bc13be0215258934fdd7199f5fb86d2ef5ae77641dd43b5e835525805997932 not found: ID does not exist" containerID="0bc13be0215258934fdd7199f5fb86d2ef5ae77641dd43b5e835525805997932"
May 26 16:26:35.524386 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:35.524348    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bc13be0215258934fdd7199f5fb86d2ef5ae77641dd43b5e835525805997932"} err="failed to get container status \"0bc13be0215258934fdd7199f5fb86d2ef5ae77641dd43b5e835525805997932\": rpc error: code = NotFound desc = could not find container \"0bc13be0215258934fdd7199f5fb86d2ef5ae77641dd43b5e835525805997932\": container with ID starting with 0bc13be0215258934fdd7199f5fb86d2ef5ae77641dd43b5e835525805997932 not found: ID does not exist"
May 26 16:26:35.524386 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:35.524364    2572 scope.go:117] "RemoveContainer" containerID="117db7777d902ca8b975a9d90bf222be5df31b099e48f30b51e4c4efce5995d0"
May 26 16:26:35.524563 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:26:35.524548    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"117db7777d902ca8b975a9d90bf222be5df31b099e48f30b51e4c4efce5995d0\": container with ID starting with 117db7777d902ca8b975a9d90bf222be5df31b099e48f30b51e4c4efce5995d0 not found: ID does not exist" containerID="117db7777d902ca8b975a9d90bf222be5df31b099e48f30b51e4c4efce5995d0"
May 26 16:26:35.524637 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:35.524569    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"117db7777d902ca8b975a9d90bf222be5df31b099e48f30b51e4c4efce5995d0"} err="failed to get container status \"117db7777d902ca8b975a9d90bf222be5df31b099e48f30b51e4c4efce5995d0\": rpc error: code = NotFound desc = could not find container \"117db7777d902ca8b975a9d90bf222be5df31b099e48f30b51e4c4efce5995d0\": container with ID starting with 117db7777d902ca8b975a9d90bf222be5df31b099e48f30b51e4c4efce5995d0 not found: ID does not exist"
May 26 16:26:36.080126 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:36.080092    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="baa8854a-095a-43c1-b906-d13a55336c01" path="/var/lib/kubelet/pods/baa8854a-095a-43c1-b906-d13a55336c01/volumes"
May 26 16:26:39.468133 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:39.468090    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-r58lh" podUID="5b5428e3-eff7-486d-9d38-163aa8c258df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
May 26 16:26:49.468066 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:49.467980    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-r58lh" podUID="5b5428e3-eff7-486d-9d38-163aa8c258df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
May 26 16:26:59.467428 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:26:59.467388    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-r58lh" podUID="5b5428e3-eff7-486d-9d38-163aa8c258df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
May 26 16:27:09.467799 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:09.467764    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-r58lh"
May 26 16:27:16.383482 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:16.383447    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-r58lh"]
May 26 16:27:16.383911 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:16.383854    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-r58lh" podUID="5b5428e3-eff7-486d-9d38-163aa8c258df" containerName="kserve-container" containerID="cri-o://e5d37ec92403204fe03b75741300608f027021bb5d4cf5286fc52307501bae8b" gracePeriod=30
May 26 16:27:16.383911 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:16.383892    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-r58lh" podUID="5b5428e3-eff7-486d-9d38-163aa8c258df" containerName="kube-rbac-proxy" containerID="cri-o://b0eb8aa6e259ffe6197d7dc7c4ffd1050205055a6c95c216cb8d8b5fddb12c77" gracePeriod=30
May 26 16:27:16.483145 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:16.483103    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm"]
May 26 16:27:16.483469 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:16.483452    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="baa8854a-095a-43c1-b906-d13a55336c01" containerName="storage-initializer"
May 26 16:27:16.483556 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:16.483472    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="baa8854a-095a-43c1-b906-d13a55336c01" containerName="storage-initializer"
May 26 16:27:16.483556 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:16.483485    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="baa8854a-095a-43c1-b906-d13a55336c01" containerName="kserve-container"
May 26 16:27:16.483556 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:16.483493    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="baa8854a-095a-43c1-b906-d13a55336c01" containerName="kserve-container"
May 26 16:27:16.483556 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:16.483503    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="baa8854a-095a-43c1-b906-d13a55336c01" containerName="kserve-agent"
May 26 16:27:16.483556 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:16.483512    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="baa8854a-095a-43c1-b906-d13a55336c01" containerName="kserve-agent"
May 26 16:27:16.483556 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:16.483521    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="baa8854a-095a-43c1-b906-d13a55336c01" containerName="kube-rbac-proxy"
May 26 16:27:16.483556 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:16.483529    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="baa8854a-095a-43c1-b906-d13a55336c01" containerName="kube-rbac-proxy"
May 26 16:27:16.483936 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:16.483623    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="baa8854a-095a-43c1-b906-d13a55336c01" containerName="kube-rbac-proxy"
May 26 16:27:16.483936 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:16.483638    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="baa8854a-095a-43c1-b906-d13a55336c01" containerName="kserve-agent"
May 26 16:27:16.483936 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:16.483651    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="baa8854a-095a-43c1-b906-d13a55336c01" containerName="kserve-container"
May 26 16:27:16.497770 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:16.497738    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm"]
May 26 16:27:16.497930 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:16.497871    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm"
May 26 16:27:16.499920 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:16.499897    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-predictor-serving-cert\""
May 26 16:27:16.500101 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:16.500082    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
May 26 16:27:16.576705 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:16.576665    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804-proxy-tls\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm\" (UID: \"2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm"
May 26 16:27:16.576705 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:16.576710    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj2rh\" (UniqueName: \"kubernetes.io/projected/2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804-kube-api-access-fj2rh\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm\" (UID: \"2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm"
May 26 16:27:16.576916 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:16.576742    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-paddle-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804-isvc-paddle-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm\" (UID: \"2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm"
May 26 16:27:16.576916 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:16.576760    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm\" (UID: \"2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm"
May 26 16:27:16.613556 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:16.613510    2572 generic.go:358] "Generic (PLEG): container finished" podID="5b5428e3-eff7-486d-9d38-163aa8c258df" containerID="b0eb8aa6e259ffe6197d7dc7c4ffd1050205055a6c95c216cb8d8b5fddb12c77" exitCode=2
May 26 16:27:16.613738 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:16.613587    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-r58lh" event={"ID":"5b5428e3-eff7-486d-9d38-163aa8c258df","Type":"ContainerDied","Data":"b0eb8aa6e259ffe6197d7dc7c4ffd1050205055a6c95c216cb8d8b5fddb12c77"}
May 26 16:27:16.677573 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:16.677485    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804-proxy-tls\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm\" (UID: \"2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm"
May 26 16:27:16.677573 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:16.677533    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fj2rh\" (UniqueName: \"kubernetes.io/projected/2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804-kube-api-access-fj2rh\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm\" (UID: \"2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm"
May 26 16:27:16.677573 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:16.677564    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-paddle-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804-isvc-paddle-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm\" (UID: \"2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm"
May 26 16:27:16.677844 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:16.677584    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm\" (UID: \"2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm"
May 26 16:27:16.678090 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:16.678068    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm\" (UID: \"2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm"
May 26 16:27:16.678327 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:16.678308    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-paddle-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804-isvc-paddle-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm\" (UID: \"2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm"
May 26 16:27:16.679986 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:16.679970    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804-proxy-tls\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm\" (UID: \"2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm"
May 26 16:27:16.685134 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:16.685109    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj2rh\" (UniqueName: \"kubernetes.io/projected/2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804-kube-api-access-fj2rh\") pod \"isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm\" (UID: \"2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm"
May 26 16:27:16.808919 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:16.808877    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm"
May 26 16:27:16.938012 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:16.937900    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm"]
May 26 16:27:16.941014 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:27:16.940984    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2251b0fe_b8e1_4ba1_9b5b_14a70f1bd804.slice/crio-8be353ee2df68e488a0ecfe359e0f7f6bc68b026f9a22e4eb372bf0be1bc0d18 WatchSource:0}: Error finding container 8be353ee2df68e488a0ecfe359e0f7f6bc68b026f9a22e4eb372bf0be1bc0d18: Status 404 returned error can't find the container with id 8be353ee2df68e488a0ecfe359e0f7f6bc68b026f9a22e4eb372bf0be1bc0d18
May 26 16:27:17.617779 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:17.617745    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm" event={"ID":"2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804","Type":"ContainerStarted","Data":"1821adc2272977db2ab54e5dbb195922495d94a1a761f21093b88a1d3fee3388"}
May 26 16:27:17.617779 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:17.617779    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm" event={"ID":"2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804","Type":"ContainerStarted","Data":"8be353ee2df68e488a0ecfe359e0f7f6bc68b026f9a22e4eb372bf0be1bc0d18"}
May 26 16:27:19.233542 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:19.233513    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-r58lh"
May 26 16:27:19.296547 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:19.296453    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-paddle-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b5428e3-eff7-486d-9d38-163aa8c258df-isvc-paddle-kube-rbac-proxy-sar-config\") pod \"5b5428e3-eff7-486d-9d38-163aa8c258df\" (UID: \"5b5428e3-eff7-486d-9d38-163aa8c258df\") "
May 26 16:27:19.296547 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:19.296505    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8kzsz\" (UniqueName: \"kubernetes.io/projected/5b5428e3-eff7-486d-9d38-163aa8c258df-kube-api-access-8kzsz\") pod \"5b5428e3-eff7-486d-9d38-163aa8c258df\" (UID: \"5b5428e3-eff7-486d-9d38-163aa8c258df\") "
May 26 16:27:19.296547 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:19.296527    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b5428e3-eff7-486d-9d38-163aa8c258df-kserve-provision-location\") pod \"5b5428e3-eff7-486d-9d38-163aa8c258df\" (UID: \"5b5428e3-eff7-486d-9d38-163aa8c258df\") "
May 26 16:27:19.296935 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:19.296553    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b5428e3-eff7-486d-9d38-163aa8c258df-proxy-tls\") pod \"5b5428e3-eff7-486d-9d38-163aa8c258df\" (UID: \"5b5428e3-eff7-486d-9d38-163aa8c258df\") "
May 26 16:27:19.296935 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:19.296834    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b5428e3-eff7-486d-9d38-163aa8c258df-isvc-paddle-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-paddle-kube-rbac-proxy-sar-config") pod "5b5428e3-eff7-486d-9d38-163aa8c258df" (UID: "5b5428e3-eff7-486d-9d38-163aa8c258df"). InnerVolumeSpecName "isvc-paddle-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:27:19.298612 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:19.298583    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b5428e3-eff7-486d-9d38-163aa8c258df-kube-api-access-8kzsz" (OuterVolumeSpecName: "kube-api-access-8kzsz") pod "5b5428e3-eff7-486d-9d38-163aa8c258df" (UID: "5b5428e3-eff7-486d-9d38-163aa8c258df"). InnerVolumeSpecName "kube-api-access-8kzsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:27:19.298714 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:19.298649    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b5428e3-eff7-486d-9d38-163aa8c258df-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5b5428e3-eff7-486d-9d38-163aa8c258df" (UID: "5b5428e3-eff7-486d-9d38-163aa8c258df"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:27:19.309685 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:19.309656    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b5428e3-eff7-486d-9d38-163aa8c258df-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5b5428e3-eff7-486d-9d38-163aa8c258df" (UID: "5b5428e3-eff7-486d-9d38-163aa8c258df"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:27:19.397589 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:19.397548    2572 reconciler_common.go:299] "Volume detached for volume \"isvc-paddle-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b5428e3-eff7-486d-9d38-163aa8c258df-isvc-paddle-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:27:19.397589 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:19.397581    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kzsz\" (UniqueName: \"kubernetes.io/projected/5b5428e3-eff7-486d-9d38-163aa8c258df-kube-api-access-8kzsz\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:27:19.397589 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:19.397592    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5b5428e3-eff7-486d-9d38-163aa8c258df-kserve-provision-location\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:27:19.397837 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:19.397623    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b5428e3-eff7-486d-9d38-163aa8c258df-proxy-tls\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:27:19.625494 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:19.625461    2572 generic.go:358] "Generic (PLEG): container finished" podID="5b5428e3-eff7-486d-9d38-163aa8c258df" containerID="e5d37ec92403204fe03b75741300608f027021bb5d4cf5286fc52307501bae8b" exitCode=0
May 26 16:27:19.625687 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:19.625526    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-r58lh" event={"ID":"5b5428e3-eff7-486d-9d38-163aa8c258df","Type":"ContainerDied","Data":"e5d37ec92403204fe03b75741300608f027021bb5d4cf5286fc52307501bae8b"}
May 26 16:27:19.625687 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:19.625536    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-r58lh"
May 26 16:27:19.625687 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:19.625554    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-r58lh" event={"ID":"5b5428e3-eff7-486d-9d38-163aa8c258df","Type":"ContainerDied","Data":"c0b1797942dc1402319ab6273c52b02933f1a0b3e58a8f6e7b5bb8d8bf39ad64"}
May 26 16:27:19.625687 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:19.625569    2572 scope.go:117] "RemoveContainer" containerID="b0eb8aa6e259ffe6197d7dc7c4ffd1050205055a6c95c216cb8d8b5fddb12c77"
May 26 16:27:19.633420 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:19.633394    2572 scope.go:117] "RemoveContainer" containerID="e5d37ec92403204fe03b75741300608f027021bb5d4cf5286fc52307501bae8b"
May 26 16:27:19.640680 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:19.640660    2572 scope.go:117] "RemoveContainer" containerID="142f86222bfe0537fa8218f1b194bc9fdf1aead11b460f4043ed94a54392a4fa"
May 26 16:27:19.645653 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:19.645627    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-r58lh"]
May 26 16:27:19.648176 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:19.648160    2572 scope.go:117] "RemoveContainer" containerID="b0eb8aa6e259ffe6197d7dc7c4ffd1050205055a6c95c216cb8d8b5fddb12c77"
May 26 16:27:19.648469 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:27:19.648445    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0eb8aa6e259ffe6197d7dc7c4ffd1050205055a6c95c216cb8d8b5fddb12c77\": container with ID starting with b0eb8aa6e259ffe6197d7dc7c4ffd1050205055a6c95c216cb8d8b5fddb12c77 not found: ID does not exist" containerID="b0eb8aa6e259ffe6197d7dc7c4ffd1050205055a6c95c216cb8d8b5fddb12c77"
May 26 16:27:19.648556 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:19.648474    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0eb8aa6e259ffe6197d7dc7c4ffd1050205055a6c95c216cb8d8b5fddb12c77"} err="failed to get container status \"b0eb8aa6e259ffe6197d7dc7c4ffd1050205055a6c95c216cb8d8b5fddb12c77\": rpc error: code = NotFound desc = could not find container \"b0eb8aa6e259ffe6197d7dc7c4ffd1050205055a6c95c216cb8d8b5fddb12c77\": container with ID starting with b0eb8aa6e259ffe6197d7dc7c4ffd1050205055a6c95c216cb8d8b5fddb12c77 not found: ID does not exist"
May 26 16:27:19.648556 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:19.648495    2572 scope.go:117] "RemoveContainer" containerID="e5d37ec92403204fe03b75741300608f027021bb5d4cf5286fc52307501bae8b"
May 26 16:27:19.648873 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:27:19.648798    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5d37ec92403204fe03b75741300608f027021bb5d4cf5286fc52307501bae8b\": container with ID starting with e5d37ec92403204fe03b75741300608f027021bb5d4cf5286fc52307501bae8b not found: ID does not exist" containerID="e5d37ec92403204fe03b75741300608f027021bb5d4cf5286fc52307501bae8b"
May 26 16:27:19.648873 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:19.648829    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5d37ec92403204fe03b75741300608f027021bb5d4cf5286fc52307501bae8b"} err="failed to get container status \"e5d37ec92403204fe03b75741300608f027021bb5d4cf5286fc52307501bae8b\": rpc error: code = NotFound desc = could not find container \"e5d37ec92403204fe03b75741300608f027021bb5d4cf5286fc52307501bae8b\": container with ID starting with e5d37ec92403204fe03b75741300608f027021bb5d4cf5286fc52307501bae8b not found: ID does not exist"
May 26 16:27:19.648873 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:19.648850    2572 scope.go:117] "RemoveContainer" containerID="142f86222bfe0537fa8218f1b194bc9fdf1aead11b460f4043ed94a54392a4fa"
May 26 16:27:19.649196 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:27:19.649170    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"142f86222bfe0537fa8218f1b194bc9fdf1aead11b460f4043ed94a54392a4fa\": container with ID starting with 142f86222bfe0537fa8218f1b194bc9fdf1aead11b460f4043ed94a54392a4fa not found: ID does not exist" containerID="142f86222bfe0537fa8218f1b194bc9fdf1aead11b460f4043ed94a54392a4fa"
May 26 16:27:19.649320 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:19.649205    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"142f86222bfe0537fa8218f1b194bc9fdf1aead11b460f4043ed94a54392a4fa"} err="failed to get container status \"142f86222bfe0537fa8218f1b194bc9fdf1aead11b460f4043ed94a54392a4fa\": rpc error: code = NotFound desc = could not find container \"142f86222bfe0537fa8218f1b194bc9fdf1aead11b460f4043ed94a54392a4fa\": container with ID starting with 142f86222bfe0537fa8218f1b194bc9fdf1aead11b460f4043ed94a54392a4fa not found: ID does not exist"
May 26 16:27:19.650849 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:19.650829    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6b8b7cfb4b-r58lh"]
May 26 16:27:20.079417 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:20.079336    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b5428e3-eff7-486d-9d38-163aa8c258df" path="/var/lib/kubelet/pods/5b5428e3-eff7-486d-9d38-163aa8c258df/volumes"
May 26 16:27:22.636692 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:22.636658    2572 generic.go:358] "Generic (PLEG): container finished" podID="2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804" containerID="1821adc2272977db2ab54e5dbb195922495d94a1a761f21093b88a1d3fee3388" exitCode=0
May 26 16:27:22.637173 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:22.636735    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm" event={"ID":"2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804","Type":"ContainerDied","Data":"1821adc2272977db2ab54e5dbb195922495d94a1a761f21093b88a1d3fee3388"}
May 26 16:27:23.641301 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:23.641265    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm" event={"ID":"2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804","Type":"ContainerStarted","Data":"2ef0e35fe549812a14c6947612a1caa0fde1bb9d3c83a80b173c51eb176c85b3"}
May 26 16:27:23.641301 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:23.641305    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm" event={"ID":"2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804","Type":"ContainerStarted","Data":"999c8ccb401d45a369964c4d530913930b8ab7d5460359aa3cc2dd1594c364cf"}
May 26 16:27:23.641795 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:23.641624    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm"
May 26 16:27:23.641795 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:23.641751    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm"
May 26 16:27:23.643199 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:23.643173    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm" podUID="2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
May 26 16:27:23.658717 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:23.658670    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm" podStartSLOduration=7.6586559770000004 podStartE2EDuration="7.658655977s" podCreationTimestamp="2026-05-26 16:27:16 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:27:23.657681387 +0000 UTC m=+1560.133408737" watchObservedRunningTime="2026-05-26 16:27:23.658655977 +0000 UTC m=+1560.134383326"
May 26 16:27:24.645031 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:24.644994    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm" podUID="2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
May 26 16:27:29.649580 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:29.649548    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm"
May 26 16:27:29.650258 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:29.650228    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm" podUID="2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
May 26 16:27:39.650572 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:39.650527    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm" podUID="2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
May 26 16:27:49.650937 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:49.650894    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm" podUID="2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
May 26 16:27:59.650324 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:27:59.650278    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm" podUID="2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
May 26 16:28:09.650746 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:09.650715    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm"
May 26 16:28:17.967200 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:17.967111    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm"]
May 26 16:28:17.967658 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:17.967450    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm" podUID="2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804" containerName="kserve-container" containerID="cri-o://999c8ccb401d45a369964c4d530913930b8ab7d5460359aa3cc2dd1594c364cf" gracePeriod=30
May 26 16:28:17.967658 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:17.967506    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm" podUID="2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804" containerName="kube-rbac-proxy" containerID="cri-o://2ef0e35fe549812a14c6947612a1caa0fde1bb9d3c83a80b173c51eb176c85b3" gracePeriod=30
May 26 16:28:18.060890 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:18.060834    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z"]
May 26 16:28:18.061154 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:18.061142    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b5428e3-eff7-486d-9d38-163aa8c258df" containerName="kserve-container"
May 26 16:28:18.061197 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:18.061156    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b5428e3-eff7-486d-9d38-163aa8c258df" containerName="kserve-container"
May 26 16:28:18.061197 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:18.061169    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b5428e3-eff7-486d-9d38-163aa8c258df" containerName="storage-initializer"
May 26 16:28:18.061197 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:18.061176    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b5428e3-eff7-486d-9d38-163aa8c258df" containerName="storage-initializer"
May 26 16:28:18.061197 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:18.061188    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b5428e3-eff7-486d-9d38-163aa8c258df" containerName="kube-rbac-proxy"
May 26 16:28:18.061197 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:18.061193    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b5428e3-eff7-486d-9d38-163aa8c258df" containerName="kube-rbac-proxy"
May 26 16:28:18.061347 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:18.061236    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="5b5428e3-eff7-486d-9d38-163aa8c258df" containerName="kube-rbac-proxy"
May 26 16:28:18.061347 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:18.061245    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="5b5428e3-eff7-486d-9d38-163aa8c258df" containerName="kserve-container"
May 26 16:28:18.064317 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:18.064298    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z"
May 26 16:28:18.066366 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:18.066343    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
May 26 16:28:18.066366 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:18.066361    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
May 26 16:28:18.074103 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:18.074062    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z"]
May 26 16:28:18.144297 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:18.144256    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca27d6ba-772e-4abd-9c1e-dd842ba317c0-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z\" (UID: \"ca27d6ba-772e-4abd-9c1e-dd842ba317c0\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z"
May 26 16:28:18.144479 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:18.144346    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zplkg\" (UniqueName: \"kubernetes.io/projected/ca27d6ba-772e-4abd-9c1e-dd842ba317c0-kube-api-access-zplkg\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z\" (UID: \"ca27d6ba-772e-4abd-9c1e-dd842ba317c0\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z"
May 26 16:28:18.144479 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:18.144393    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca27d6ba-772e-4abd-9c1e-dd842ba317c0-isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z\" (UID: \"ca27d6ba-772e-4abd-9c1e-dd842ba317c0\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z"
May 26 16:28:18.144479 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:18.144449    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca27d6ba-772e-4abd-9c1e-dd842ba317c0-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z\" (UID: \"ca27d6ba-772e-4abd-9c1e-dd842ba317c0\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z"
May 26 16:28:18.245688 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:18.245545    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca27d6ba-772e-4abd-9c1e-dd842ba317c0-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z\" (UID: \"ca27d6ba-772e-4abd-9c1e-dd842ba317c0\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z"
May 26 16:28:18.245688 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:18.245668    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca27d6ba-772e-4abd-9c1e-dd842ba317c0-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z\" (UID: \"ca27d6ba-772e-4abd-9c1e-dd842ba317c0\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z"
May 26 16:28:18.245931 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:18.245704    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zplkg\" (UniqueName: \"kubernetes.io/projected/ca27d6ba-772e-4abd-9c1e-dd842ba317c0-kube-api-access-zplkg\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z\" (UID: \"ca27d6ba-772e-4abd-9c1e-dd842ba317c0\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z"
May 26 16:28:18.245931 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:18.245769    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca27d6ba-772e-4abd-9c1e-dd842ba317c0-isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z\" (UID: \"ca27d6ba-772e-4abd-9c1e-dd842ba317c0\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z"
May 26 16:28:18.246173 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:18.246153    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca27d6ba-772e-4abd-9c1e-dd842ba317c0-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z\" (UID: \"ca27d6ba-772e-4abd-9c1e-dd842ba317c0\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z"
May 26 16:28:18.246453 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:18.246436    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca27d6ba-772e-4abd-9c1e-dd842ba317c0-isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z\" (UID: \"ca27d6ba-772e-4abd-9c1e-dd842ba317c0\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z"
May 26 16:28:18.248022 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:18.248003    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca27d6ba-772e-4abd-9c1e-dd842ba317c0-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z\" (UID: \"ca27d6ba-772e-4abd-9c1e-dd842ba317c0\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z"
May 26 16:28:18.255512 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:18.255492    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zplkg\" (UniqueName: \"kubernetes.io/projected/ca27d6ba-772e-4abd-9c1e-dd842ba317c0-kube-api-access-zplkg\") pod \"isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z\" (UID: \"ca27d6ba-772e-4abd-9c1e-dd842ba317c0\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z"
May 26 16:28:18.375341 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:18.375298    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z"
May 26 16:28:18.521131 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:18.521037    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z"]
May 26 16:28:18.523673 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:28:18.523648    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca27d6ba_772e_4abd_9c1e_dd842ba317c0.slice/crio-6c84a17812bd84289eeefdc4a195d7d590551a550149e0fba412bdc265d20493 WatchSource:0}: Error finding container 6c84a17812bd84289eeefdc4a195d7d590551a550149e0fba412bdc265d20493: Status 404 returned error can't find the container with id 6c84a17812bd84289eeefdc4a195d7d590551a550149e0fba412bdc265d20493
May 26 16:28:18.525401 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:18.525384    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 16:28:18.794797 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:18.794720    2572 generic.go:358] "Generic (PLEG): container finished" podID="2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804" containerID="2ef0e35fe549812a14c6947612a1caa0fde1bb9d3c83a80b173c51eb176c85b3" exitCode=2
May 26 16:28:18.794797 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:18.794780    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm" event={"ID":"2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804","Type":"ContainerDied","Data":"2ef0e35fe549812a14c6947612a1caa0fde1bb9d3c83a80b173c51eb176c85b3"}
May 26 16:28:18.796022 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:18.795994    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z" event={"ID":"ca27d6ba-772e-4abd-9c1e-dd842ba317c0","Type":"ContainerStarted","Data":"7329926b287bda1c6ef7e862ee676b733ed26861869acec9fe47d143b53209d3"}
May 26 16:28:18.796127 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:18.796033    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z" event={"ID":"ca27d6ba-772e-4abd-9c1e-dd842ba317c0","Type":"ContainerStarted","Data":"6c84a17812bd84289eeefdc4a195d7d590551a550149e0fba412bdc265d20493"}
May 26 16:28:19.646256 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:19.646196    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm" podUID="2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
May 26 16:28:19.650365 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:19.650334    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm" podUID="2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
May 26 16:28:20.711397 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:20.711371    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm"
May 26 16:28:20.802942 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:20.802855    2572 generic.go:358] "Generic (PLEG): container finished" podID="2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804" containerID="999c8ccb401d45a369964c4d530913930b8ab7d5460359aa3cc2dd1594c364cf" exitCode=0
May 26 16:28:20.802942 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:20.802908    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm" event={"ID":"2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804","Type":"ContainerDied","Data":"999c8ccb401d45a369964c4d530913930b8ab7d5460359aa3cc2dd1594c364cf"}
May 26 16:28:20.802942 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:20.802935    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm" event={"ID":"2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804","Type":"ContainerDied","Data":"8be353ee2df68e488a0ecfe359e0f7f6bc68b026f9a22e4eb372bf0be1bc0d18"}
May 26 16:28:20.802942 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:20.802939    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm"
May 26 16:28:20.803231 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:20.802950    2572 scope.go:117] "RemoveContainer" containerID="2ef0e35fe549812a14c6947612a1caa0fde1bb9d3c83a80b173c51eb176c85b3"
May 26 16:28:20.810470 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:20.810449    2572 scope.go:117] "RemoveContainer" containerID="999c8ccb401d45a369964c4d530913930b8ab7d5460359aa3cc2dd1594c364cf"
May 26 16:28:20.817084 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:20.817068    2572 scope.go:117] "RemoveContainer" containerID="1821adc2272977db2ab54e5dbb195922495d94a1a761f21093b88a1d3fee3388"
May 26 16:28:20.823501 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:20.823484    2572 scope.go:117] "RemoveContainer" containerID="2ef0e35fe549812a14c6947612a1caa0fde1bb9d3c83a80b173c51eb176c85b3"
May 26 16:28:20.823787 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:28:20.823768    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ef0e35fe549812a14c6947612a1caa0fde1bb9d3c83a80b173c51eb176c85b3\": container with ID starting with 2ef0e35fe549812a14c6947612a1caa0fde1bb9d3c83a80b173c51eb176c85b3 not found: ID does not exist" containerID="2ef0e35fe549812a14c6947612a1caa0fde1bb9d3c83a80b173c51eb176c85b3"
May 26 16:28:20.823832 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:20.823798    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ef0e35fe549812a14c6947612a1caa0fde1bb9d3c83a80b173c51eb176c85b3"} err="failed to get container status \"2ef0e35fe549812a14c6947612a1caa0fde1bb9d3c83a80b173c51eb176c85b3\": rpc error: code = NotFound desc = could not find container \"2ef0e35fe549812a14c6947612a1caa0fde1bb9d3c83a80b173c51eb176c85b3\": container with ID starting with 2ef0e35fe549812a14c6947612a1caa0fde1bb9d3c83a80b173c51eb176c85b3 not found: ID does not exist"
May 26 16:28:20.823832 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:20.823816    2572 scope.go:117] "RemoveContainer" containerID="999c8ccb401d45a369964c4d530913930b8ab7d5460359aa3cc2dd1594c364cf"
May 26 16:28:20.824044 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:28:20.824025    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"999c8ccb401d45a369964c4d530913930b8ab7d5460359aa3cc2dd1594c364cf\": container with ID starting with 999c8ccb401d45a369964c4d530913930b8ab7d5460359aa3cc2dd1594c364cf not found: ID does not exist" containerID="999c8ccb401d45a369964c4d530913930b8ab7d5460359aa3cc2dd1594c364cf"
May 26 16:28:20.824086 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:20.824051    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"999c8ccb401d45a369964c4d530913930b8ab7d5460359aa3cc2dd1594c364cf"} err="failed to get container status \"999c8ccb401d45a369964c4d530913930b8ab7d5460359aa3cc2dd1594c364cf\": rpc error: code = NotFound desc = could not find container \"999c8ccb401d45a369964c4d530913930b8ab7d5460359aa3cc2dd1594c364cf\": container with ID starting with 999c8ccb401d45a369964c4d530913930b8ab7d5460359aa3cc2dd1594c364cf not found: ID does not exist"
May 26 16:28:20.824086 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:20.824066    2572 scope.go:117] "RemoveContainer" containerID="1821adc2272977db2ab54e5dbb195922495d94a1a761f21093b88a1d3fee3388"
May 26 16:28:20.824300 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:28:20.824284    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1821adc2272977db2ab54e5dbb195922495d94a1a761f21093b88a1d3fee3388\": container with ID starting with 1821adc2272977db2ab54e5dbb195922495d94a1a761f21093b88a1d3fee3388 not found: ID does not exist" containerID="1821adc2272977db2ab54e5dbb195922495d94a1a761f21093b88a1d3fee3388"
May 26 16:28:20.824354 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:20.824303    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1821adc2272977db2ab54e5dbb195922495d94a1a761f21093b88a1d3fee3388"} err="failed to get container status \"1821adc2272977db2ab54e5dbb195922495d94a1a761f21093b88a1d3fee3388\": rpc error: code = NotFound desc = could not find container \"1821adc2272977db2ab54e5dbb195922495d94a1a761f21093b88a1d3fee3388\": container with ID starting with 1821adc2272977db2ab54e5dbb195922495d94a1a761f21093b88a1d3fee3388 not found: ID does not exist"
May 26 16:28:20.868644 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:20.868592    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804-proxy-tls\") pod \"2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804\" (UID: \"2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804\") "
May 26 16:28:20.868728 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:20.868658    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fj2rh\" (UniqueName: \"kubernetes.io/projected/2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804-kube-api-access-fj2rh\") pod \"2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804\" (UID: \"2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804\") "
May 26 16:28:20.868728 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:20.868717    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-paddle-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804-isvc-paddle-runtime-kube-rbac-proxy-sar-config\") pod \"2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804\" (UID: \"2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804\") "
May 26 16:28:20.868801 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:20.868741    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804-kserve-provision-location\") pod \"2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804\" (UID: \"2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804\") "
May 26 16:28:20.869113 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:20.869091    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804-isvc-paddle-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-paddle-runtime-kube-rbac-proxy-sar-config") pod "2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804" (UID: "2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804"). InnerVolumeSpecName "isvc-paddle-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:28:20.870748 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:20.870726    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804" (UID: "2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:28:20.870852 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:20.870839    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804-kube-api-access-fj2rh" (OuterVolumeSpecName: "kube-api-access-fj2rh") pod "2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804" (UID: "2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804"). InnerVolumeSpecName "kube-api-access-fj2rh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:28:20.878467 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:20.878440    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804" (UID: "2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:28:20.969982 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:20.969942    2572 reconciler_common.go:299] "Volume detached for volume \"isvc-paddle-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804-isvc-paddle-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:28:20.969982 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:20.969978    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804-kserve-provision-location\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:28:20.970156 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:20.969995    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804-proxy-tls\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:28:20.970156 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:20.970009    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj2rh\" (UniqueName: \"kubernetes.io/projected/2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804-kube-api-access-fj2rh\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:28:21.125217 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:21.125184    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm"]
May 26 16:28:21.128973 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:21.128946    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-7f4d4f9dc8-8x9sm"]
May 26 16:28:22.079142 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:22.079103    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804" path="/var/lib/kubelet/pods/2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804/volumes"
May 26 16:28:23.813902 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:23.813859    2572 generic.go:358] "Generic (PLEG): container finished" podID="ca27d6ba-772e-4abd-9c1e-dd842ba317c0" containerID="7329926b287bda1c6ef7e862ee676b733ed26861869acec9fe47d143b53209d3" exitCode=0
May 26 16:28:23.814263 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:23.813932    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z" event={"ID":"ca27d6ba-772e-4abd-9c1e-dd842ba317c0","Type":"ContainerDied","Data":"7329926b287bda1c6ef7e862ee676b733ed26861869acec9fe47d143b53209d3"}
May 26 16:28:24.818382 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:24.818348    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z" event={"ID":"ca27d6ba-772e-4abd-9c1e-dd842ba317c0","Type":"ContainerStarted","Data":"37fe96f4055ece73e530811ebb2c3cf409b5677bbcc7fdbd283f49510d7db383"}
May 26 16:28:24.818382 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:24.818389    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z" event={"ID":"ca27d6ba-772e-4abd-9c1e-dd842ba317c0","Type":"ContainerStarted","Data":"843a1b2f679e5fc9683ed245ecb725a337656ce42d1978ffc3a2708f175d870c"}
May 26 16:28:24.818820 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:24.818583    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z"
May 26 16:28:24.834649 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:24.834582    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z" podStartSLOduration=6.834568144 podStartE2EDuration="6.834568144s" podCreationTimestamp="2026-05-26 16:28:18 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:28:24.833730443 +0000 UTC m=+1621.309457794" watchObservedRunningTime="2026-05-26 16:28:24.834568144 +0000 UTC m=+1621.310295494"
May 26 16:28:25.821577 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:25.821550    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z"
May 26 16:28:25.822654 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:25.822627    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z" podUID="ca27d6ba-772e-4abd-9c1e-dd842ba317c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
May 26 16:28:26.824011 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:26.823972    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z" podUID="ca27d6ba-772e-4abd-9c1e-dd842ba317c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
May 26 16:28:31.828265 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:31.828239    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z"
May 26 16:28:31.828735 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:31.828709    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z" podUID="ca27d6ba-772e-4abd-9c1e-dd842ba317c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
May 26 16:28:41.829169 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:41.829129    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z" podUID="ca27d6ba-772e-4abd-9c1e-dd842ba317c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
May 26 16:28:51.829504 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:28:51.829460    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z" podUID="ca27d6ba-772e-4abd-9c1e-dd842ba317c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
May 26 16:29:01.829689 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:01.829642    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z" podUID="ca27d6ba-772e-4abd-9c1e-dd842ba317c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
May 26 16:29:11.830264 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:11.830235    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z"
May 26 16:29:19.685904 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:19.685862    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z"]
May 26 16:29:19.686400 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:19.686191    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z" podUID="ca27d6ba-772e-4abd-9c1e-dd842ba317c0" containerName="kserve-container" containerID="cri-o://843a1b2f679e5fc9683ed245ecb725a337656ce42d1978ffc3a2708f175d870c" gracePeriod=30
May 26 16:29:19.686400 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:19.686269    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z" podUID="ca27d6ba-772e-4abd-9c1e-dd842ba317c0" containerName="kube-rbac-proxy" containerID="cri-o://37fe96f4055ece73e530811ebb2c3cf409b5677bbcc7fdbd283f49510d7db383" gracePeriod=30
May 26 16:29:19.795084 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:19.795047    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-4mm66"]
May 26 16:29:19.795330 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:19.795317    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804" containerName="kserve-container"
May 26 16:29:19.795376 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:19.795332    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804" containerName="kserve-container"
May 26 16:29:19.795376 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:19.795349    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804" containerName="kube-rbac-proxy"
May 26 16:29:19.795376 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:19.795355    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804" containerName="kube-rbac-proxy"
May 26 16:29:19.795376 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:19.795362    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804" containerName="storage-initializer"
May 26 16:29:19.795376 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:19.795367    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804" containerName="storage-initializer"
May 26 16:29:19.795557 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:19.795413    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804" containerName="kube-rbac-proxy"
May 26 16:29:19.795557 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:19.795421    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="2251b0fe-b8e1-4ba1-9b5b-14a70f1bd804" containerName="kserve-container"
May 26 16:29:19.800409 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:19.800379    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-4mm66"
May 26 16:29:19.803466 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:19.803438    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-predictor-serving-cert\""
May 26 16:29:19.803717 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:19.803694    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
May 26 16:29:19.816734 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:19.816711    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-4mm66"]
May 26 16:29:19.919217 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:19.919177    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-pmml-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9dbd5351-b06e-4957-a0c5-41a266c71fbc-isvc-pmml-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8bb578669-4mm66\" (UID: \"9dbd5351-b06e-4957-a0c5-41a266c71fbc\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-4mm66"
May 26 16:29:19.919217 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:19.919215    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9dbd5351-b06e-4957-a0c5-41a266c71fbc-kserve-provision-location\") pod \"isvc-pmml-predictor-8bb578669-4mm66\" (UID: \"9dbd5351-b06e-4957-a0c5-41a266c71fbc\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-4mm66"
May 26 16:29:19.919446 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:19.919304    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9dbd5351-b06e-4957-a0c5-41a266c71fbc-proxy-tls\") pod \"isvc-pmml-predictor-8bb578669-4mm66\" (UID: \"9dbd5351-b06e-4957-a0c5-41a266c71fbc\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-4mm66"
May 26 16:29:19.919446 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:19.919360    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrr7n\" (UniqueName: \"kubernetes.io/projected/9dbd5351-b06e-4957-a0c5-41a266c71fbc-kube-api-access-wrr7n\") pod \"isvc-pmml-predictor-8bb578669-4mm66\" (UID: \"9dbd5351-b06e-4957-a0c5-41a266c71fbc\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-4mm66"
May 26 16:29:19.967326 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:19.967236    2572 generic.go:358] "Generic (PLEG): container finished" podID="ca27d6ba-772e-4abd-9c1e-dd842ba317c0" containerID="37fe96f4055ece73e530811ebb2c3cf409b5677bbcc7fdbd283f49510d7db383" exitCode=2
May 26 16:29:19.967326 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:19.967290    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z" event={"ID":"ca27d6ba-772e-4abd-9c1e-dd842ba317c0","Type":"ContainerDied","Data":"37fe96f4055ece73e530811ebb2c3cf409b5677bbcc7fdbd283f49510d7db383"}
May 26 16:29:20.020133 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:20.020100    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9dbd5351-b06e-4957-a0c5-41a266c71fbc-proxy-tls\") pod \"isvc-pmml-predictor-8bb578669-4mm66\" (UID: \"9dbd5351-b06e-4957-a0c5-41a266c71fbc\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-4mm66"
May 26 16:29:20.020267 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:20.020147    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wrr7n\" (UniqueName: \"kubernetes.io/projected/9dbd5351-b06e-4957-a0c5-41a266c71fbc-kube-api-access-wrr7n\") pod \"isvc-pmml-predictor-8bb578669-4mm66\" (UID: \"9dbd5351-b06e-4957-a0c5-41a266c71fbc\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-4mm66"
May 26 16:29:20.020315 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:20.020262    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-pmml-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9dbd5351-b06e-4957-a0c5-41a266c71fbc-isvc-pmml-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8bb578669-4mm66\" (UID: \"9dbd5351-b06e-4957-a0c5-41a266c71fbc\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-4mm66"
May 26 16:29:20.020315 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:20.020295    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9dbd5351-b06e-4957-a0c5-41a266c71fbc-kserve-provision-location\") pod \"isvc-pmml-predictor-8bb578669-4mm66\" (UID: \"9dbd5351-b06e-4957-a0c5-41a266c71fbc\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-4mm66"
May 26 16:29:20.020796 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:20.020720    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9dbd5351-b06e-4957-a0c5-41a266c71fbc-kserve-provision-location\") pod \"isvc-pmml-predictor-8bb578669-4mm66\" (UID: \"9dbd5351-b06e-4957-a0c5-41a266c71fbc\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-4mm66"
May 26 16:29:20.021013 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:20.020967    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-pmml-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9dbd5351-b06e-4957-a0c5-41a266c71fbc-isvc-pmml-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8bb578669-4mm66\" (UID: \"9dbd5351-b06e-4957-a0c5-41a266c71fbc\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-4mm66"
May 26 16:29:20.022690 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:20.022668    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9dbd5351-b06e-4957-a0c5-41a266c71fbc-proxy-tls\") pod \"isvc-pmml-predictor-8bb578669-4mm66\" (UID: \"9dbd5351-b06e-4957-a0c5-41a266c71fbc\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-4mm66"
May 26 16:29:20.027597 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:20.027579    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrr7n\" (UniqueName: \"kubernetes.io/projected/9dbd5351-b06e-4957-a0c5-41a266c71fbc-kube-api-access-wrr7n\") pod \"isvc-pmml-predictor-8bb578669-4mm66\" (UID: \"9dbd5351-b06e-4957-a0c5-41a266c71fbc\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-4mm66"
May 26 16:29:20.111698 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:20.111656    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-4mm66"
May 26 16:29:20.244356 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:20.244213    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-4mm66"]
May 26 16:29:20.246880 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:29:20.246851    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dbd5351_b06e_4957_a0c5_41a266c71fbc.slice/crio-856cc776ad0236557b4db93b68a57879d2f997f4b28cdd21084950b648751b6a WatchSource:0}: Error finding container 856cc776ad0236557b4db93b68a57879d2f997f4b28cdd21084950b648751b6a: Status 404 returned error can't find the container with id 856cc776ad0236557b4db93b68a57879d2f997f4b28cdd21084950b648751b6a
May 26 16:29:20.971756 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:20.971714    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-4mm66" event={"ID":"9dbd5351-b06e-4957-a0c5-41a266c71fbc","Type":"ContainerStarted","Data":"a838c0f61bf03a0aee8967aa84d4cffb69656f890a7435c7c9fe35833a53925f"}
May 26 16:29:20.971756 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:20.971761    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-4mm66" event={"ID":"9dbd5351-b06e-4957-a0c5-41a266c71fbc","Type":"ContainerStarted","Data":"856cc776ad0236557b4db93b68a57879d2f997f4b28cdd21084950b648751b6a"}
May 26 16:29:21.824370 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:21.824323    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z" podUID="ca27d6ba-772e-4abd-9c1e-dd842ba317c0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
May 26 16:29:21.829480 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:21.829451    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z" podUID="ca27d6ba-772e-4abd-9c1e-dd842ba317c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
May 26 16:29:22.630924 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:22.630900    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z"
May 26 16:29:22.742150 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:22.742047    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca27d6ba-772e-4abd-9c1e-dd842ba317c0-proxy-tls\") pod \"ca27d6ba-772e-4abd-9c1e-dd842ba317c0\" (UID: \"ca27d6ba-772e-4abd-9c1e-dd842ba317c0\") "
May 26 16:29:22.742150 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:22.742093    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca27d6ba-772e-4abd-9c1e-dd842ba317c0-kserve-provision-location\") pod \"ca27d6ba-772e-4abd-9c1e-dd842ba317c0\" (UID: \"ca27d6ba-772e-4abd-9c1e-dd842ba317c0\") "
May 26 16:29:22.742150 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:22.742119    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca27d6ba-772e-4abd-9c1e-dd842ba317c0-isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\") pod \"ca27d6ba-772e-4abd-9c1e-dd842ba317c0\" (UID: \"ca27d6ba-772e-4abd-9c1e-dd842ba317c0\") "
May 26 16:29:22.742445 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:22.742172    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zplkg\" (UniqueName: \"kubernetes.io/projected/ca27d6ba-772e-4abd-9c1e-dd842ba317c0-kube-api-access-zplkg\") pod \"ca27d6ba-772e-4abd-9c1e-dd842ba317c0\" (UID: \"ca27d6ba-772e-4abd-9c1e-dd842ba317c0\") "
May 26 16:29:22.742527 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:22.742503    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca27d6ba-772e-4abd-9c1e-dd842ba317c0-isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config") pod "ca27d6ba-772e-4abd-9c1e-dd842ba317c0" (UID: "ca27d6ba-772e-4abd-9c1e-dd842ba317c0"). InnerVolumeSpecName "isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:29:22.744267 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:22.744237    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca27d6ba-772e-4abd-9c1e-dd842ba317c0-kube-api-access-zplkg" (OuterVolumeSpecName: "kube-api-access-zplkg") pod "ca27d6ba-772e-4abd-9c1e-dd842ba317c0" (UID: "ca27d6ba-772e-4abd-9c1e-dd842ba317c0"). InnerVolumeSpecName "kube-api-access-zplkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:29:22.744267 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:22.744256    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca27d6ba-772e-4abd-9c1e-dd842ba317c0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ca27d6ba-772e-4abd-9c1e-dd842ba317c0" (UID: "ca27d6ba-772e-4abd-9c1e-dd842ba317c0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:29:22.752535 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:22.752503    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca27d6ba-772e-4abd-9c1e-dd842ba317c0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ca27d6ba-772e-4abd-9c1e-dd842ba317c0" (UID: "ca27d6ba-772e-4abd-9c1e-dd842ba317c0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:29:22.843188 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:22.843158    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zplkg\" (UniqueName: \"kubernetes.io/projected/ca27d6ba-772e-4abd-9c1e-dd842ba317c0-kube-api-access-zplkg\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:29:22.843188 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:22.843185    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca27d6ba-772e-4abd-9c1e-dd842ba317c0-proxy-tls\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:29:22.843380 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:22.843209    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca27d6ba-772e-4abd-9c1e-dd842ba317c0-kserve-provision-location\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:29:22.843380 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:22.843219    2572 reconciler_common.go:299] "Volume detached for volume \"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca27d6ba-772e-4abd-9c1e-dd842ba317c0-isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:29:22.978672 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:22.978640    2572 generic.go:358] "Generic (PLEG): container finished" podID="ca27d6ba-772e-4abd-9c1e-dd842ba317c0" containerID="843a1b2f679e5fc9683ed245ecb725a337656ce42d1978ffc3a2708f175d870c" exitCode=0
May 26 16:29:22.978845 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:22.978723    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z"
May 26 16:29:22.978845 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:22.978728    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z" event={"ID":"ca27d6ba-772e-4abd-9c1e-dd842ba317c0","Type":"ContainerDied","Data":"843a1b2f679e5fc9683ed245ecb725a337656ce42d1978ffc3a2708f175d870c"}
May 26 16:29:22.978845 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:22.978770    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z" event={"ID":"ca27d6ba-772e-4abd-9c1e-dd842ba317c0","Type":"ContainerDied","Data":"6c84a17812bd84289eeefdc4a195d7d590551a550149e0fba412bdc265d20493"}
May 26 16:29:22.978845 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:22.978790    2572 scope.go:117] "RemoveContainer" containerID="37fe96f4055ece73e530811ebb2c3cf409b5677bbcc7fdbd283f49510d7db383"
May 26 16:29:22.987174 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:22.987157    2572 scope.go:117] "RemoveContainer" containerID="843a1b2f679e5fc9683ed245ecb725a337656ce42d1978ffc3a2708f175d870c"
May 26 16:29:22.994127 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:22.994108    2572 scope.go:117] "RemoveContainer" containerID="7329926b287bda1c6ef7e862ee676b733ed26861869acec9fe47d143b53209d3"
May 26 16:29:22.999010 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:22.998987    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z"]
May 26 16:29:23.001391 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:23.001328    2572 scope.go:117] "RemoveContainer" containerID="37fe96f4055ece73e530811ebb2c3cf409b5677bbcc7fdbd283f49510d7db383"
May 26 16:29:23.001647 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:29:23.001626    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37fe96f4055ece73e530811ebb2c3cf409b5677bbcc7fdbd283f49510d7db383\": container with ID starting with 37fe96f4055ece73e530811ebb2c3cf409b5677bbcc7fdbd283f49510d7db383 not found: ID does not exist" containerID="37fe96f4055ece73e530811ebb2c3cf409b5677bbcc7fdbd283f49510d7db383"
May 26 16:29:23.001713 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:23.001660    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37fe96f4055ece73e530811ebb2c3cf409b5677bbcc7fdbd283f49510d7db383"} err="failed to get container status \"37fe96f4055ece73e530811ebb2c3cf409b5677bbcc7fdbd283f49510d7db383\": rpc error: code = NotFound desc = could not find container \"37fe96f4055ece73e530811ebb2c3cf409b5677bbcc7fdbd283f49510d7db383\": container with ID starting with 37fe96f4055ece73e530811ebb2c3cf409b5677bbcc7fdbd283f49510d7db383 not found: ID does not exist"
May 26 16:29:23.001713 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:23.001704    2572 scope.go:117] "RemoveContainer" containerID="843a1b2f679e5fc9683ed245ecb725a337656ce42d1978ffc3a2708f175d870c"
May 26 16:29:23.001982 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:29:23.001948    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"843a1b2f679e5fc9683ed245ecb725a337656ce42d1978ffc3a2708f175d870c\": container with ID starting with 843a1b2f679e5fc9683ed245ecb725a337656ce42d1978ffc3a2708f175d870c not found: ID does not exist" containerID="843a1b2f679e5fc9683ed245ecb725a337656ce42d1978ffc3a2708f175d870c"
May 26 16:29:23.002096 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:23.001982    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"843a1b2f679e5fc9683ed245ecb725a337656ce42d1978ffc3a2708f175d870c"} err="failed to get container status \"843a1b2f679e5fc9683ed245ecb725a337656ce42d1978ffc3a2708f175d870c\": rpc error: code = NotFound desc = could not find container \"843a1b2f679e5fc9683ed245ecb725a337656ce42d1978ffc3a2708f175d870c\": container with ID starting with 843a1b2f679e5fc9683ed245ecb725a337656ce42d1978ffc3a2708f175d870c not found: ID does not exist"
May 26 16:29:23.002096 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:23.002004    2572 scope.go:117] "RemoveContainer" containerID="7329926b287bda1c6ef7e862ee676b733ed26861869acec9fe47d143b53209d3"
May 26 16:29:23.002535 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:29:23.002446    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7329926b287bda1c6ef7e862ee676b733ed26861869acec9fe47d143b53209d3\": container with ID starting with 7329926b287bda1c6ef7e862ee676b733ed26861869acec9fe47d143b53209d3 not found: ID does not exist" containerID="7329926b287bda1c6ef7e862ee676b733ed26861869acec9fe47d143b53209d3"
May 26 16:29:23.002535 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:23.002481    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7329926b287bda1c6ef7e862ee676b733ed26861869acec9fe47d143b53209d3"} err="failed to get container status \"7329926b287bda1c6ef7e862ee676b733ed26861869acec9fe47d143b53209d3\": rpc error: code = NotFound desc = could not find container \"7329926b287bda1c6ef7e862ee676b733ed26861869acec9fe47d143b53209d3\": container with ID starting with 7329926b287bda1c6ef7e862ee676b733ed26861869acec9fe47d143b53209d3 not found: ID does not exist"
May 26 16:29:23.003622 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:23.003581    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-7dbd59854-2nl8z"]
May 26 16:29:23.983541 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:23.983502    2572 generic.go:358] "Generic (PLEG): container finished" podID="9dbd5351-b06e-4957-a0c5-41a266c71fbc" containerID="a838c0f61bf03a0aee8967aa84d4cffb69656f890a7435c7c9fe35833a53925f" exitCode=0
May 26 16:29:23.983959 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:23.983572    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-4mm66" event={"ID":"9dbd5351-b06e-4957-a0c5-41a266c71fbc","Type":"ContainerDied","Data":"a838c0f61bf03a0aee8967aa84d4cffb69656f890a7435c7c9fe35833a53925f"}
May 26 16:29:24.079165 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:24.079128    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca27d6ba-772e-4abd-9c1e-dd842ba317c0" path="/var/lib/kubelet/pods/ca27d6ba-772e-4abd-9c1e-dd842ba317c0/volumes"
May 26 16:29:32.014467 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:32.014431    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-4mm66" event={"ID":"9dbd5351-b06e-4957-a0c5-41a266c71fbc","Type":"ContainerStarted","Data":"333e9f8f4670ad9e114ad05e065ed40edcac46897424ff1ab05a3ba4857ce23b"}
May 26 16:29:32.014981 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:32.014481    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-4mm66" event={"ID":"9dbd5351-b06e-4957-a0c5-41a266c71fbc","Type":"ContainerStarted","Data":"2c2caf7f127914b8e679af6317072808ad8689b74c56961d5c1aca660f26fd78"}
May 26 16:29:32.014981 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:32.014736    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-4mm66"
May 26 16:29:32.032889 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:32.032823    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-4mm66" podStartSLOduration=5.728872071 podStartE2EDuration="13.032804713s" podCreationTimestamp="2026-05-26 16:29:19 +0000 UTC" firstStartedPulling="2026-05-26 16:29:23.984890054 +0000 UTC m=+1680.460617383" lastFinishedPulling="2026-05-26 16:29:31.288822696 +0000 UTC m=+1687.764550025" observedRunningTime="2026-05-26 16:29:32.031498468 +0000 UTC m=+1688.507225830" watchObservedRunningTime="2026-05-26 16:29:32.032804713 +0000 UTC m=+1688.508532066"
May 26 16:29:33.017568 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:33.017536    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-4mm66"
May 26 16:29:33.018906 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:33.018876    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-4mm66" podUID="9dbd5351-b06e-4957-a0c5-41a266c71fbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
May 26 16:29:34.019773 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:34.019735    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-4mm66" podUID="9dbd5351-b06e-4957-a0c5-41a266c71fbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
May 26 16:29:39.023998 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:39.023968    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-4mm66"
May 26 16:29:39.024482 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:39.024458    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-4mm66" podUID="9dbd5351-b06e-4957-a0c5-41a266c71fbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
May 26 16:29:49.024970 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:49.024890    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-4mm66" podUID="9dbd5351-b06e-4957-a0c5-41a266c71fbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
May 26 16:29:59.025115 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:29:59.025071    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-4mm66" podUID="9dbd5351-b06e-4957-a0c5-41a266c71fbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
May 26 16:30:09.024731 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:30:09.024689    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-4mm66" podUID="9dbd5351-b06e-4957-a0c5-41a266c71fbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
May 26 16:30:19.025143 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:30:19.025108    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-4mm66" podUID="9dbd5351-b06e-4957-a0c5-41a266c71fbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
May 26 16:30:29.024875 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:30:29.024826    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-4mm66" podUID="9dbd5351-b06e-4957-a0c5-41a266c71fbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
May 26 16:30:39.025344 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:30:39.025303    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-4mm66" podUID="9dbd5351-b06e-4957-a0c5-41a266c71fbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
May 26 16:30:49.025220 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:30:49.025176    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-4mm66" podUID="9dbd5351-b06e-4957-a0c5-41a266c71fbc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
May 26 16:30:58.079294 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:30:58.079267    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-4mm66"
May 26 16:31:00.977436 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:00.977402    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-4mm66"]
May 26 16:31:00.977819 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:00.977729    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-4mm66" podUID="9dbd5351-b06e-4957-a0c5-41a266c71fbc" containerName="kserve-container" containerID="cri-o://2c2caf7f127914b8e679af6317072808ad8689b74c56961d5c1aca660f26fd78" gracePeriod=30
May 26 16:31:00.977886 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:00.977789    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-4mm66" podUID="9dbd5351-b06e-4957-a0c5-41a266c71fbc" containerName="kube-rbac-proxy" containerID="cri-o://333e9f8f4670ad9e114ad05e065ed40edcac46897424ff1ab05a3ba4857ce23b" gracePeriod=30
May 26 16:31:01.092944 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:01.092914    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-lhzhb"]
May 26 16:31:01.093193 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:01.093180    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca27d6ba-772e-4abd-9c1e-dd842ba317c0" containerName="storage-initializer"
May 26 16:31:01.093249 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:01.093194    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca27d6ba-772e-4abd-9c1e-dd842ba317c0" containerName="storage-initializer"
May 26 16:31:01.093249 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:01.093216    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca27d6ba-772e-4abd-9c1e-dd842ba317c0" containerName="kserve-container"
May 26 16:31:01.093249 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:01.093221    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca27d6ba-772e-4abd-9c1e-dd842ba317c0" containerName="kserve-container"
May 26 16:31:01.093249 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:01.093227    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca27d6ba-772e-4abd-9c1e-dd842ba317c0" containerName="kube-rbac-proxy"
May 26 16:31:01.093249 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:01.093232    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca27d6ba-772e-4abd-9c1e-dd842ba317c0" containerName="kube-rbac-proxy"
May 26 16:31:01.093451 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:01.093273    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="ca27d6ba-772e-4abd-9c1e-dd842ba317c0" containerName="kube-rbac-proxy"
May 26 16:31:01.093451 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:01.093282    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="ca27d6ba-772e-4abd-9c1e-dd842ba317c0" containerName="kserve-container"
May 26 16:31:01.095300 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:01.095267    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-lhzhb"
May 26 16:31:01.097259 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:01.097232    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-predictor-serving-cert\""
May 26 16:31:01.097377 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:01.097261    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
May 26 16:31:01.104783 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:01.104758    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-lhzhb"]
May 26 16:31:01.214818 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:01.214775    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-pmml-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83904f99-898e-4152-8952-8a73d2167656-isvc-pmml-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-67bc544947-lhzhb\" (UID: \"83904f99-898e-4152-8952-8a73d2167656\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-lhzhb"
May 26 16:31:01.214996 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:01.214846    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83904f99-898e-4152-8952-8a73d2167656-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-67bc544947-lhzhb\" (UID: \"83904f99-898e-4152-8952-8a73d2167656\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-lhzhb"
May 26 16:31:01.214996 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:01.214885    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83904f99-898e-4152-8952-8a73d2167656-proxy-tls\") pod \"isvc-pmml-runtime-predictor-67bc544947-lhzhb\" (UID: \"83904f99-898e-4152-8952-8a73d2167656\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-lhzhb"
May 26 16:31:01.214996 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:01.214933    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgsn7\" (UniqueName: \"kubernetes.io/projected/83904f99-898e-4152-8952-8a73d2167656-kube-api-access-qgsn7\") pod \"isvc-pmml-runtime-predictor-67bc544947-lhzhb\" (UID: \"83904f99-898e-4152-8952-8a73d2167656\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-lhzhb"
May 26 16:31:01.249027 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:01.248941    2572 generic.go:358] "Generic (PLEG): container finished" podID="9dbd5351-b06e-4957-a0c5-41a266c71fbc" containerID="333e9f8f4670ad9e114ad05e065ed40edcac46897424ff1ab05a3ba4857ce23b" exitCode=2
May 26 16:31:01.249027 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:01.248979    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-4mm66" event={"ID":"9dbd5351-b06e-4957-a0c5-41a266c71fbc","Type":"ContainerDied","Data":"333e9f8f4670ad9e114ad05e065ed40edcac46897424ff1ab05a3ba4857ce23b"}
May 26 16:31:01.315339 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:01.315300    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qgsn7\" (UniqueName: \"kubernetes.io/projected/83904f99-898e-4152-8952-8a73d2167656-kube-api-access-qgsn7\") pod \"isvc-pmml-runtime-predictor-67bc544947-lhzhb\" (UID: \"83904f99-898e-4152-8952-8a73d2167656\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-lhzhb"
May 26 16:31:01.315518 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:01.315354    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-pmml-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83904f99-898e-4152-8952-8a73d2167656-isvc-pmml-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-67bc544947-lhzhb\" (UID: \"83904f99-898e-4152-8952-8a73d2167656\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-lhzhb"
May 26 16:31:01.315518 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:01.315401    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83904f99-898e-4152-8952-8a73d2167656-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-67bc544947-lhzhb\" (UID: \"83904f99-898e-4152-8952-8a73d2167656\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-lhzhb"
May 26 16:31:01.315518 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:01.315427    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83904f99-898e-4152-8952-8a73d2167656-proxy-tls\") pod \"isvc-pmml-runtime-predictor-67bc544947-lhzhb\" (UID: \"83904f99-898e-4152-8952-8a73d2167656\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-lhzhb"
May 26 16:31:01.315666 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:31:01.315527    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-pmml-runtime-predictor-serving-cert: secret "isvc-pmml-runtime-predictor-serving-cert" not found
May 26 16:31:01.315666 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:31:01.315644    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83904f99-898e-4152-8952-8a73d2167656-proxy-tls podName:83904f99-898e-4152-8952-8a73d2167656 nodeName:}" failed. No retries permitted until 2026-05-26 16:31:01.815625745 +0000 UTC m=+1778.291353088 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/83904f99-898e-4152-8952-8a73d2167656-proxy-tls") pod "isvc-pmml-runtime-predictor-67bc544947-lhzhb" (UID: "83904f99-898e-4152-8952-8a73d2167656") : secret "isvc-pmml-runtime-predictor-serving-cert" not found
May 26 16:31:01.316006 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:01.315982    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83904f99-898e-4152-8952-8a73d2167656-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-67bc544947-lhzhb\" (UID: \"83904f99-898e-4152-8952-8a73d2167656\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-lhzhb"
May 26 16:31:01.316194 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:01.316177    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-pmml-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83904f99-898e-4152-8952-8a73d2167656-isvc-pmml-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-67bc544947-lhzhb\" (UID: \"83904f99-898e-4152-8952-8a73d2167656\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-lhzhb"
May 26 16:31:01.324482 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:01.324453    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgsn7\" (UniqueName: \"kubernetes.io/projected/83904f99-898e-4152-8952-8a73d2167656-kube-api-access-qgsn7\") pod \"isvc-pmml-runtime-predictor-67bc544947-lhzhb\" (UID: \"83904f99-898e-4152-8952-8a73d2167656\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-lhzhb"
May 26 16:31:01.818543 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:01.818504    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83904f99-898e-4152-8952-8a73d2167656-proxy-tls\") pod \"isvc-pmml-runtime-predictor-67bc544947-lhzhb\" (UID: \"83904f99-898e-4152-8952-8a73d2167656\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-lhzhb"
May 26 16:31:01.820936 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:01.820917    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83904f99-898e-4152-8952-8a73d2167656-proxy-tls\") pod \"isvc-pmml-runtime-predictor-67bc544947-lhzhb\" (UID: \"83904f99-898e-4152-8952-8a73d2167656\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-lhzhb"
May 26 16:31:02.006139 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:02.006093    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-lhzhb"
May 26 16:31:02.133986 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:02.133954    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-lhzhb"]
May 26 16:31:02.137389 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:31:02.137350    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83904f99_898e_4152_8952_8a73d2167656.slice/crio-a2f3641f734cf6558bffa659d6abfcf83a4f79c2f8f6f9b3dca3df512f45449d WatchSource:0}: Error finding container a2f3641f734cf6558bffa659d6abfcf83a4f79c2f8f6f9b3dca3df512f45449d: Status 404 returned error can't find the container with id a2f3641f734cf6558bffa659d6abfcf83a4f79c2f8f6f9b3dca3df512f45449d
May 26 16:31:02.252978 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:02.252936    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-lhzhb" event={"ID":"83904f99-898e-4152-8952-8a73d2167656","Type":"ContainerStarted","Data":"398b65cd87b27f4ddfa8c92cfbd3a98692806baf6fe0402bd22301a00ba0226a"}
May 26 16:31:02.253147 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:02.252987    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-lhzhb" event={"ID":"83904f99-898e-4152-8952-8a73d2167656","Type":"ContainerStarted","Data":"a2f3641f734cf6558bffa659d6abfcf83a4f79c2f8f6f9b3dca3df512f45449d"}
May 26 16:31:04.020414 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:04.020366    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-4mm66" podUID="9dbd5351-b06e-4957-a0c5-41a266c71fbc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
May 26 16:31:04.714846 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:04.714822    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-4mm66"
May 26 16:31:04.841021 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:04.840922    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-pmml-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9dbd5351-b06e-4957-a0c5-41a266c71fbc-isvc-pmml-kube-rbac-proxy-sar-config\") pod \"9dbd5351-b06e-4957-a0c5-41a266c71fbc\" (UID: \"9dbd5351-b06e-4957-a0c5-41a266c71fbc\") "
May 26 16:31:04.841021 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:04.840969    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrr7n\" (UniqueName: \"kubernetes.io/projected/9dbd5351-b06e-4957-a0c5-41a266c71fbc-kube-api-access-wrr7n\") pod \"9dbd5351-b06e-4957-a0c5-41a266c71fbc\" (UID: \"9dbd5351-b06e-4957-a0c5-41a266c71fbc\") "
May 26 16:31:04.841021 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:04.841001    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9dbd5351-b06e-4957-a0c5-41a266c71fbc-proxy-tls\") pod \"9dbd5351-b06e-4957-a0c5-41a266c71fbc\" (UID: \"9dbd5351-b06e-4957-a0c5-41a266c71fbc\") "
May 26 16:31:04.841021 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:04.841021    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9dbd5351-b06e-4957-a0c5-41a266c71fbc-kserve-provision-location\") pod \"9dbd5351-b06e-4957-a0c5-41a266c71fbc\" (UID: \"9dbd5351-b06e-4957-a0c5-41a266c71fbc\") "
May 26 16:31:04.841407 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:04.841380    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dbd5351-b06e-4957-a0c5-41a266c71fbc-isvc-pmml-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-pmml-kube-rbac-proxy-sar-config") pod "9dbd5351-b06e-4957-a0c5-41a266c71fbc" (UID: "9dbd5351-b06e-4957-a0c5-41a266c71fbc"). InnerVolumeSpecName "isvc-pmml-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:31:04.841464 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:04.841385    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dbd5351-b06e-4957-a0c5-41a266c71fbc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9dbd5351-b06e-4957-a0c5-41a266c71fbc" (UID: "9dbd5351-b06e-4957-a0c5-41a266c71fbc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:31:04.843092 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:04.843069    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dbd5351-b06e-4957-a0c5-41a266c71fbc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9dbd5351-b06e-4957-a0c5-41a266c71fbc" (UID: "9dbd5351-b06e-4957-a0c5-41a266c71fbc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:31:04.843177 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:04.843154    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dbd5351-b06e-4957-a0c5-41a266c71fbc-kube-api-access-wrr7n" (OuterVolumeSpecName: "kube-api-access-wrr7n") pod "9dbd5351-b06e-4957-a0c5-41a266c71fbc" (UID: "9dbd5351-b06e-4957-a0c5-41a266c71fbc"). InnerVolumeSpecName "kube-api-access-wrr7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:31:04.941915 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:04.941873    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9dbd5351-b06e-4957-a0c5-41a266c71fbc-proxy-tls\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:31:04.941915 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:04.941910    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9dbd5351-b06e-4957-a0c5-41a266c71fbc-kserve-provision-location\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:31:04.942127 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:04.941927    2572 reconciler_common.go:299] "Volume detached for volume \"isvc-pmml-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9dbd5351-b06e-4957-a0c5-41a266c71fbc-isvc-pmml-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:31:04.942127 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:04.941942    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrr7n\" (UniqueName: \"kubernetes.io/projected/9dbd5351-b06e-4957-a0c5-41a266c71fbc-kube-api-access-wrr7n\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:31:05.263024 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:05.262988    2572 generic.go:358] "Generic (PLEG): container finished" podID="9dbd5351-b06e-4957-a0c5-41a266c71fbc" containerID="2c2caf7f127914b8e679af6317072808ad8689b74c56961d5c1aca660f26fd78" exitCode=0
May 26 16:31:05.263486 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:05.263048    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-4mm66" event={"ID":"9dbd5351-b06e-4957-a0c5-41a266c71fbc","Type":"ContainerDied","Data":"2c2caf7f127914b8e679af6317072808ad8689b74c56961d5c1aca660f26fd78"}
May 26 16:31:05.263486 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:05.263065    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-4mm66"
May 26 16:31:05.263486 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:05.263083    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-4mm66" event={"ID":"9dbd5351-b06e-4957-a0c5-41a266c71fbc","Type":"ContainerDied","Data":"856cc776ad0236557b4db93b68a57879d2f997f4b28cdd21084950b648751b6a"}
May 26 16:31:05.263486 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:05.263103    2572 scope.go:117] "RemoveContainer" containerID="333e9f8f4670ad9e114ad05e065ed40edcac46897424ff1ab05a3ba4857ce23b"
May 26 16:31:05.272161 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:05.272045    2572 scope.go:117] "RemoveContainer" containerID="2c2caf7f127914b8e679af6317072808ad8689b74c56961d5c1aca660f26fd78"
May 26 16:31:05.279047 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:05.279028    2572 scope.go:117] "RemoveContainer" containerID="a838c0f61bf03a0aee8967aa84d4cffb69656f890a7435c7c9fe35833a53925f"
May 26 16:31:05.285004 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:05.284977    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-4mm66"]
May 26 16:31:05.286873 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:05.286851    2572 scope.go:117] "RemoveContainer" containerID="333e9f8f4670ad9e114ad05e065ed40edcac46897424ff1ab05a3ba4857ce23b"
May 26 16:31:05.287854 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:05.287833    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8bb578669-4mm66"]
May 26 16:31:05.288263 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:31:05.288234    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"333e9f8f4670ad9e114ad05e065ed40edcac46897424ff1ab05a3ba4857ce23b\": container with ID starting with 333e9f8f4670ad9e114ad05e065ed40edcac46897424ff1ab05a3ba4857ce23b not found: ID does not exist" containerID="333e9f8f4670ad9e114ad05e065ed40edcac46897424ff1ab05a3ba4857ce23b"
May 26 16:31:05.288382 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:05.288274    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"333e9f8f4670ad9e114ad05e065ed40edcac46897424ff1ab05a3ba4857ce23b"} err="failed to get container status \"333e9f8f4670ad9e114ad05e065ed40edcac46897424ff1ab05a3ba4857ce23b\": rpc error: code = NotFound desc = could not find container \"333e9f8f4670ad9e114ad05e065ed40edcac46897424ff1ab05a3ba4857ce23b\": container with ID starting with 333e9f8f4670ad9e114ad05e065ed40edcac46897424ff1ab05a3ba4857ce23b not found: ID does not exist"
May 26 16:31:05.288382 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:05.288299    2572 scope.go:117] "RemoveContainer" containerID="2c2caf7f127914b8e679af6317072808ad8689b74c56961d5c1aca660f26fd78"
May 26 16:31:05.288683 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:31:05.288661    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c2caf7f127914b8e679af6317072808ad8689b74c56961d5c1aca660f26fd78\": container with ID starting with 2c2caf7f127914b8e679af6317072808ad8689b74c56961d5c1aca660f26fd78 not found: ID does not exist" containerID="2c2caf7f127914b8e679af6317072808ad8689b74c56961d5c1aca660f26fd78"
May 26 16:31:05.288824 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:05.288799    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c2caf7f127914b8e679af6317072808ad8689b74c56961d5c1aca660f26fd78"} err="failed to get container status \"2c2caf7f127914b8e679af6317072808ad8689b74c56961d5c1aca660f26fd78\": rpc error: code = NotFound desc = could not find container \"2c2caf7f127914b8e679af6317072808ad8689b74c56961d5c1aca660f26fd78\": container with ID starting with 2c2caf7f127914b8e679af6317072808ad8689b74c56961d5c1aca660f26fd78 not found: ID does not exist"
May 26 16:31:05.288923 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:05.288912    2572 scope.go:117] "RemoveContainer" containerID="a838c0f61bf03a0aee8967aa84d4cffb69656f890a7435c7c9fe35833a53925f"
May 26 16:31:05.289349 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:31:05.289331    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a838c0f61bf03a0aee8967aa84d4cffb69656f890a7435c7c9fe35833a53925f\": container with ID starting with a838c0f61bf03a0aee8967aa84d4cffb69656f890a7435c7c9fe35833a53925f not found: ID does not exist" containerID="a838c0f61bf03a0aee8967aa84d4cffb69656f890a7435c7c9fe35833a53925f"
May 26 16:31:05.289489 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:05.289468    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a838c0f61bf03a0aee8967aa84d4cffb69656f890a7435c7c9fe35833a53925f"} err="failed to get container status \"a838c0f61bf03a0aee8967aa84d4cffb69656f890a7435c7c9fe35833a53925f\": rpc error: code = NotFound desc = could not find container \"a838c0f61bf03a0aee8967aa84d4cffb69656f890a7435c7c9fe35833a53925f\": container with ID starting with a838c0f61bf03a0aee8967aa84d4cffb69656f890a7435c7c9fe35833a53925f not found: ID does not exist"
May 26 16:31:06.079062 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:06.078983    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9dbd5351-b06e-4957-a0c5-41a266c71fbc" path="/var/lib/kubelet/pods/9dbd5351-b06e-4957-a0c5-41a266c71fbc/volumes"
May 26 16:31:06.267761 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:06.267730    2572 generic.go:358] "Generic (PLEG): container finished" podID="83904f99-898e-4152-8952-8a73d2167656" containerID="398b65cd87b27f4ddfa8c92cfbd3a98692806baf6fe0402bd22301a00ba0226a" exitCode=0
May 26 16:31:06.268123 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:06.267801    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-lhzhb" event={"ID":"83904f99-898e-4152-8952-8a73d2167656","Type":"ContainerDied","Data":"398b65cd87b27f4ddfa8c92cfbd3a98692806baf6fe0402bd22301a00ba0226a"}
May 26 16:31:07.272726 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:07.272688    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-lhzhb" event={"ID":"83904f99-898e-4152-8952-8a73d2167656","Type":"ContainerStarted","Data":"73c56176563197da76c5e7b1eb51397f5dfe0e26e2e63791e25d854bcaf61e35"}
May 26 16:31:07.272726 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:07.272732    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-lhzhb" event={"ID":"83904f99-898e-4152-8952-8a73d2167656","Type":"ContainerStarted","Data":"eca0716c4bc4c79f6d8e20e44e06f3d1d884629895f667a0a85eb89d4febcba7"}
May 26 16:31:07.273133 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:07.272932    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-lhzhb"
May 26 16:31:07.291441 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:07.291388    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-lhzhb" podStartSLOduration=6.291373175 podStartE2EDuration="6.291373175s" podCreationTimestamp="2026-05-26 16:31:01 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:31:07.289843936 +0000 UTC m=+1783.765571287" watchObservedRunningTime="2026-05-26 16:31:07.291373175 +0000 UTC m=+1783.767100525"
May 26 16:31:08.275807 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:08.275780    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-lhzhb"
May 26 16:31:08.276890 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:08.276865    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-lhzhb" podUID="83904f99-898e-4152-8952-8a73d2167656" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
May 26 16:31:09.282992 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:09.282949    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-lhzhb" podUID="83904f99-898e-4152-8952-8a73d2167656" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
May 26 16:31:14.284914 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:14.284839    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-lhzhb"
May 26 16:31:14.285405 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:14.285306    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-lhzhb" podUID="83904f99-898e-4152-8952-8a73d2167656" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
May 26 16:31:24.286011 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:24.285967    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-lhzhb" podUID="83904f99-898e-4152-8952-8a73d2167656" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
May 26 16:31:34.285435 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:34.285396    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-lhzhb" podUID="83904f99-898e-4152-8952-8a73d2167656" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
May 26 16:31:44.285306 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:44.285264    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-lhzhb" podUID="83904f99-898e-4152-8952-8a73d2167656" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
May 26 16:31:54.286000 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:31:54.285958    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-lhzhb" podUID="83904f99-898e-4152-8952-8a73d2167656" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
May 26 16:32:04.286194 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:04.286148    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-lhzhb" podUID="83904f99-898e-4152-8952-8a73d2167656" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
May 26 16:32:14.285446 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:14.285406    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-lhzhb" podUID="83904f99-898e-4152-8952-8a73d2167656" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
May 26 16:32:24.286107 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:24.286068    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-lhzhb" podUID="83904f99-898e-4152-8952-8a73d2167656" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
May 26 16:32:32.078981 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:32.078951    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-lhzhb"
May 26 16:32:42.096748 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:42.096716    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-lhzhb"]
May 26 16:32:42.099309 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:42.097116    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-lhzhb" podUID="83904f99-898e-4152-8952-8a73d2167656" containerName="kserve-container" containerID="cri-o://eca0716c4bc4c79f6d8e20e44e06f3d1d884629895f667a0a85eb89d4febcba7" gracePeriod=30
May 26 16:32:42.099309 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:42.097178    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-lhzhb" podUID="83904f99-898e-4152-8952-8a73d2167656" containerName="kube-rbac-proxy" containerID="cri-o://73c56176563197da76c5e7b1eb51397f5dfe0e26e2e63791e25d854bcaf61e35" gracePeriod=30
May 26 16:32:42.197481 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:42.197450    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz"]
May 26 16:32:42.197836 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:42.197818    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dbd5351-b06e-4957-a0c5-41a266c71fbc" containerName="kserve-container"
May 26 16:32:42.197934 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:42.197838    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dbd5351-b06e-4957-a0c5-41a266c71fbc" containerName="kserve-container"
May 26 16:32:42.197934 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:42.197852    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dbd5351-b06e-4957-a0c5-41a266c71fbc" containerName="storage-initializer"
May 26 16:32:42.197934 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:42.197860    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dbd5351-b06e-4957-a0c5-41a266c71fbc" containerName="storage-initializer"
May 26 16:32:42.197934 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:42.197871    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dbd5351-b06e-4957-a0c5-41a266c71fbc" containerName="kube-rbac-proxy"
May 26 16:32:42.197934 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:42.197887    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dbd5351-b06e-4957-a0c5-41a266c71fbc" containerName="kube-rbac-proxy"
May 26 16:32:42.198194 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:42.197954    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="9dbd5351-b06e-4957-a0c5-41a266c71fbc" containerName="kserve-container"
May 26 16:32:42.198194 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:42.197967    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="9dbd5351-b06e-4957-a0c5-41a266c71fbc" containerName="kube-rbac-proxy"
May 26 16:32:42.201066 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:42.201045    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz"
May 26 16:32:42.202909 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:42.202885    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\""
May 26 16:32:42.203020 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:42.202978    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\""
May 26 16:32:42.209651 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:42.209624    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz"]
May 26 16:32:42.388881 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:42.388850    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5725fa76-0a7d-4363-8337-c5518ef7331f-isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz\" (UID: \"5725fa76-0a7d-4363-8337-c5518ef7331f\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz"
May 26 16:32:42.389063 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:42.388898    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5725fa76-0a7d-4363-8337-c5518ef7331f-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz\" (UID: \"5725fa76-0a7d-4363-8337-c5518ef7331f\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz"
May 26 16:32:42.389063 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:42.388943    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5725fa76-0a7d-4363-8337-c5518ef7331f-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz\" (UID: \"5725fa76-0a7d-4363-8337-c5518ef7331f\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz"
May 26 16:32:42.389063 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:42.389022    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9hgf\" (UniqueName: \"kubernetes.io/projected/5725fa76-0a7d-4363-8337-c5518ef7331f-kube-api-access-v9hgf\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz\" (UID: \"5725fa76-0a7d-4363-8337-c5518ef7331f\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz"
May 26 16:32:42.489418 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:42.489380    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-v9hgf\" (UniqueName: \"kubernetes.io/projected/5725fa76-0a7d-4363-8337-c5518ef7331f-kube-api-access-v9hgf\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz\" (UID: \"5725fa76-0a7d-4363-8337-c5518ef7331f\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz"
May 26 16:32:42.489596 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:42.489437    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5725fa76-0a7d-4363-8337-c5518ef7331f-isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz\" (UID: \"5725fa76-0a7d-4363-8337-c5518ef7331f\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz"
May 26 16:32:42.489596 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:42.489470    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5725fa76-0a7d-4363-8337-c5518ef7331f-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz\" (UID: \"5725fa76-0a7d-4363-8337-c5518ef7331f\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz"
May 26 16:32:42.489596 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:42.489489    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5725fa76-0a7d-4363-8337-c5518ef7331f-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz\" (UID: \"5725fa76-0a7d-4363-8337-c5518ef7331f\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz"
May 26 16:32:42.490089 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:42.490056    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5725fa76-0a7d-4363-8337-c5518ef7331f-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz\" (UID: \"5725fa76-0a7d-4363-8337-c5518ef7331f\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz"
May 26 16:32:42.490192 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:42.490142    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5725fa76-0a7d-4363-8337-c5518ef7331f-isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz\" (UID: \"5725fa76-0a7d-4363-8337-c5518ef7331f\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz"
May 26 16:32:42.492090 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:42.492072    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5725fa76-0a7d-4363-8337-c5518ef7331f-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz\" (UID: \"5725fa76-0a7d-4363-8337-c5518ef7331f\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz"
May 26 16:32:42.497630 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:42.497579    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9hgf\" (UniqueName: \"kubernetes.io/projected/5725fa76-0a7d-4363-8337-c5518ef7331f-kube-api-access-v9hgf\") pod \"isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz\" (UID: \"5725fa76-0a7d-4363-8337-c5518ef7331f\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz"
May 26 16:32:42.512508 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:42.512489    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz"
May 26 16:32:42.538892 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:42.538852    2572 generic.go:358] "Generic (PLEG): container finished" podID="83904f99-898e-4152-8952-8a73d2167656" containerID="73c56176563197da76c5e7b1eb51397f5dfe0e26e2e63791e25d854bcaf61e35" exitCode=2
May 26 16:32:42.539038 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:42.538901    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-lhzhb" event={"ID":"83904f99-898e-4152-8952-8a73d2167656","Type":"ContainerDied","Data":"73c56176563197da76c5e7b1eb51397f5dfe0e26e2e63791e25d854bcaf61e35"}
May 26 16:32:42.640223 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:42.640166    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz"]
May 26 16:32:42.643043 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:32:42.643016    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5725fa76_0a7d_4363_8337_c5518ef7331f.slice/crio-f9a271a125fbe6135655678a7e3a5c0542877ef9c59b1dfffb82ea447602480c WatchSource:0}: Error finding container f9a271a125fbe6135655678a7e3a5c0542877ef9c59b1dfffb82ea447602480c: Status 404 returned error can't find the container with id f9a271a125fbe6135655678a7e3a5c0542877ef9c59b1dfffb82ea447602480c
May 26 16:32:43.543504 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:43.543428    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz" event={"ID":"5725fa76-0a7d-4363-8337-c5518ef7331f","Type":"ContainerStarted","Data":"7cc9720d94dbe69bcc6a508c56a18616636e11b565b11f2b7663e371ebbaaf1d"}
May 26 16:32:43.543504 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:43.543465    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz" event={"ID":"5725fa76-0a7d-4363-8337-c5518ef7331f","Type":"ContainerStarted","Data":"f9a271a125fbe6135655678a7e3a5c0542877ef9c59b1dfffb82ea447602480c"}
May 26 16:32:44.281489 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:44.281443    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-lhzhb" podUID="83904f99-898e-4152-8952-8a73d2167656" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
May 26 16:32:45.750135 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:45.750111    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-lhzhb"
May 26 16:32:45.917705 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:45.917669    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-pmml-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83904f99-898e-4152-8952-8a73d2167656-isvc-pmml-runtime-kube-rbac-proxy-sar-config\") pod \"83904f99-898e-4152-8952-8a73d2167656\" (UID: \"83904f99-898e-4152-8952-8a73d2167656\") "
May 26 16:32:45.917900 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:45.917712    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83904f99-898e-4152-8952-8a73d2167656-proxy-tls\") pod \"83904f99-898e-4152-8952-8a73d2167656\" (UID: \"83904f99-898e-4152-8952-8a73d2167656\") "
May 26 16:32:45.917900 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:45.917760    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qgsn7\" (UniqueName: \"kubernetes.io/projected/83904f99-898e-4152-8952-8a73d2167656-kube-api-access-qgsn7\") pod \"83904f99-898e-4152-8952-8a73d2167656\" (UID: \"83904f99-898e-4152-8952-8a73d2167656\") "
May 26 16:32:45.917900 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:45.917785    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83904f99-898e-4152-8952-8a73d2167656-kserve-provision-location\") pod \"83904f99-898e-4152-8952-8a73d2167656\" (UID: \"83904f99-898e-4152-8952-8a73d2167656\") "
May 26 16:32:45.918180 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:45.918155    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83904f99-898e-4152-8952-8a73d2167656-isvc-pmml-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-pmml-runtime-kube-rbac-proxy-sar-config") pod "83904f99-898e-4152-8952-8a73d2167656" (UID: "83904f99-898e-4152-8952-8a73d2167656"). InnerVolumeSpecName "isvc-pmml-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:32:45.918266 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:45.918190    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83904f99-898e-4152-8952-8a73d2167656-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "83904f99-898e-4152-8952-8a73d2167656" (UID: "83904f99-898e-4152-8952-8a73d2167656"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:32:45.919940 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:45.919908    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83904f99-898e-4152-8952-8a73d2167656-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "83904f99-898e-4152-8952-8a73d2167656" (UID: "83904f99-898e-4152-8952-8a73d2167656"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:32:45.920020 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:45.920001    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83904f99-898e-4152-8952-8a73d2167656-kube-api-access-qgsn7" (OuterVolumeSpecName: "kube-api-access-qgsn7") pod "83904f99-898e-4152-8952-8a73d2167656" (UID: "83904f99-898e-4152-8952-8a73d2167656"). InnerVolumeSpecName "kube-api-access-qgsn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:32:46.018810 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:46.018776    2572 reconciler_common.go:299] "Volume detached for volume \"isvc-pmml-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83904f99-898e-4152-8952-8a73d2167656-isvc-pmml-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:32:46.018810 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:46.018808    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83904f99-898e-4152-8952-8a73d2167656-proxy-tls\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:32:46.018985 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:46.018818    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgsn7\" (UniqueName: \"kubernetes.io/projected/83904f99-898e-4152-8952-8a73d2167656-kube-api-access-qgsn7\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:32:46.018985 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:46.018829    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83904f99-898e-4152-8952-8a73d2167656-kserve-provision-location\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:32:46.553299 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:46.553257    2572 generic.go:358] "Generic (PLEG): container finished" podID="83904f99-898e-4152-8952-8a73d2167656" containerID="eca0716c4bc4c79f6d8e20e44e06f3d1d884629895f667a0a85eb89d4febcba7" exitCode=0
May 26 16:32:46.553299 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:46.553303    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-lhzhb" event={"ID":"83904f99-898e-4152-8952-8a73d2167656","Type":"ContainerDied","Data":"eca0716c4bc4c79f6d8e20e44e06f3d1d884629895f667a0a85eb89d4febcba7"}
May 26 16:32:46.553527 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:46.553330    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-lhzhb" event={"ID":"83904f99-898e-4152-8952-8a73d2167656","Type":"ContainerDied","Data":"a2f3641f734cf6558bffa659d6abfcf83a4f79c2f8f6f9b3dca3df512f45449d"}
May 26 16:32:46.553527 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:46.553347    2572 scope.go:117] "RemoveContainer" containerID="73c56176563197da76c5e7b1eb51397f5dfe0e26e2e63791e25d854bcaf61e35"
May 26 16:32:46.553527 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:46.553359    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-lhzhb"
May 26 16:32:46.561773 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:46.561748    2572 scope.go:117] "RemoveContainer" containerID="eca0716c4bc4c79f6d8e20e44e06f3d1d884629895f667a0a85eb89d4febcba7"
May 26 16:32:46.569436 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:46.569418    2572 scope.go:117] "RemoveContainer" containerID="398b65cd87b27f4ddfa8c92cfbd3a98692806baf6fe0402bd22301a00ba0226a"
May 26 16:32:46.569538 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:46.569465    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-lhzhb"]
May 26 16:32:46.574586 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:46.574564    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-67bc544947-lhzhb"]
May 26 16:32:46.578364 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:46.578347    2572 scope.go:117] "RemoveContainer" containerID="73c56176563197da76c5e7b1eb51397f5dfe0e26e2e63791e25d854bcaf61e35"
May 26 16:32:46.578653 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:32:46.578633    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73c56176563197da76c5e7b1eb51397f5dfe0e26e2e63791e25d854bcaf61e35\": container with ID starting with 73c56176563197da76c5e7b1eb51397f5dfe0e26e2e63791e25d854bcaf61e35 not found: ID does not exist" containerID="73c56176563197da76c5e7b1eb51397f5dfe0e26e2e63791e25d854bcaf61e35"
May 26 16:32:46.578729 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:46.578660    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73c56176563197da76c5e7b1eb51397f5dfe0e26e2e63791e25d854bcaf61e35"} err="failed to get container status \"73c56176563197da76c5e7b1eb51397f5dfe0e26e2e63791e25d854bcaf61e35\": rpc error: code = NotFound desc = could not find container \"73c56176563197da76c5e7b1eb51397f5dfe0e26e2e63791e25d854bcaf61e35\": container with ID starting with 73c56176563197da76c5e7b1eb51397f5dfe0e26e2e63791e25d854bcaf61e35 not found: ID does not exist"
May 26 16:32:46.578729 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:46.578680    2572 scope.go:117] "RemoveContainer" containerID="eca0716c4bc4c79f6d8e20e44e06f3d1d884629895f667a0a85eb89d4febcba7"
May 26 16:32:46.578962 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:32:46.578945    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eca0716c4bc4c79f6d8e20e44e06f3d1d884629895f667a0a85eb89d4febcba7\": container with ID starting with eca0716c4bc4c79f6d8e20e44e06f3d1d884629895f667a0a85eb89d4febcba7 not found: ID does not exist" containerID="eca0716c4bc4c79f6d8e20e44e06f3d1d884629895f667a0a85eb89d4febcba7"
May 26 16:32:46.579016 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:46.578968    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eca0716c4bc4c79f6d8e20e44e06f3d1d884629895f667a0a85eb89d4febcba7"} err="failed to get container status \"eca0716c4bc4c79f6d8e20e44e06f3d1d884629895f667a0a85eb89d4febcba7\": rpc error: code = NotFound desc = could not find container \"eca0716c4bc4c79f6d8e20e44e06f3d1d884629895f667a0a85eb89d4febcba7\": container with ID starting with eca0716c4bc4c79f6d8e20e44e06f3d1d884629895f667a0a85eb89d4febcba7 not found: ID does not exist"
May 26 16:32:46.579016 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:46.578984    2572 scope.go:117] "RemoveContainer" containerID="398b65cd87b27f4ddfa8c92cfbd3a98692806baf6fe0402bd22301a00ba0226a"
May 26 16:32:46.579234 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:32:46.579217    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"398b65cd87b27f4ddfa8c92cfbd3a98692806baf6fe0402bd22301a00ba0226a\": container with ID starting with 398b65cd87b27f4ddfa8c92cfbd3a98692806baf6fe0402bd22301a00ba0226a not found: ID does not exist" containerID="398b65cd87b27f4ddfa8c92cfbd3a98692806baf6fe0402bd22301a00ba0226a"
May 26 16:32:46.579286 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:46.579242    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"398b65cd87b27f4ddfa8c92cfbd3a98692806baf6fe0402bd22301a00ba0226a"} err="failed to get container status \"398b65cd87b27f4ddfa8c92cfbd3a98692806baf6fe0402bd22301a00ba0226a\": rpc error: code = NotFound desc = could not find container \"398b65cd87b27f4ddfa8c92cfbd3a98692806baf6fe0402bd22301a00ba0226a\": container with ID starting with 398b65cd87b27f4ddfa8c92cfbd3a98692806baf6fe0402bd22301a00ba0226a not found: ID does not exist"
May 26 16:32:47.556987 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:47.556950    2572 generic.go:358] "Generic (PLEG): container finished" podID="5725fa76-0a7d-4363-8337-c5518ef7331f" containerID="7cc9720d94dbe69bcc6a508c56a18616636e11b565b11f2b7663e371ebbaaf1d" exitCode=0
May 26 16:32:47.557390 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:47.557021    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz" event={"ID":"5725fa76-0a7d-4363-8337-c5518ef7331f","Type":"ContainerDied","Data":"7cc9720d94dbe69bcc6a508c56a18616636e11b565b11f2b7663e371ebbaaf1d"}
May 26 16:32:48.079346 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:48.079313    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83904f99-898e-4152-8952-8a73d2167656" path="/var/lib/kubelet/pods/83904f99-898e-4152-8952-8a73d2167656/volumes"
May 26 16:32:48.562900 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:48.562869    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz" event={"ID":"5725fa76-0a7d-4363-8337-c5518ef7331f","Type":"ContainerStarted","Data":"0820af451486246140a8c0ec6b25cb6ab66beb90f858828654a593b9f3977d3d"}
May 26 16:32:48.563276 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:48.562907    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz" event={"ID":"5725fa76-0a7d-4363-8337-c5518ef7331f","Type":"ContainerStarted","Data":"96d736265bfb17f88a0a6f91bb8b8923cb5c6bc7ab49b144f083d1565552bcc0"}
May 26 16:32:48.563276 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:48.563178    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz"
May 26 16:32:48.563383 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:48.563279    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz"
May 26 16:32:48.564553 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:48.564533    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz" podUID="5725fa76-0a7d-4363-8337-c5518ef7331f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
May 26 16:32:48.581392 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:48.581329    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz" podStartSLOduration=6.581313913 podStartE2EDuration="6.581313913s" podCreationTimestamp="2026-05-26 16:32:42 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:32:48.579664583 +0000 UTC m=+1885.055391932" watchObservedRunningTime="2026-05-26 16:32:48.581313913 +0000 UTC m=+1885.057041262"
May 26 16:32:49.565531 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:49.565493    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz" podUID="5725fa76-0a7d-4363-8337-c5518ef7331f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
May 26 16:32:54.570574 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:54.570546    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz"
May 26 16:32:54.571047 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:32:54.571021    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz" podUID="5725fa76-0a7d-4363-8337-c5518ef7331f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
May 26 16:33:04.571991 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:33:04.571951    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz" podUID="5725fa76-0a7d-4363-8337-c5518ef7331f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
May 26 16:33:14.571583 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:33:14.571541    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz" podUID="5725fa76-0a7d-4363-8337-c5518ef7331f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
May 26 16:33:24.571505 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:33:24.571461    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz" podUID="5725fa76-0a7d-4363-8337-c5518ef7331f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
May 26 16:33:34.571191 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:33:34.571151    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz" podUID="5725fa76-0a7d-4363-8337-c5518ef7331f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
May 26 16:33:44.571803 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:33:44.571760    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz" podUID="5725fa76-0a7d-4363-8337-c5518ef7331f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
May 26 16:33:54.571817 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:33:54.571776    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz" podUID="5725fa76-0a7d-4363-8337-c5518ef7331f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
May 26 16:34:04.571928 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:04.571891    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz" podUID="5725fa76-0a7d-4363-8337-c5518ef7331f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
May 26 16:34:14.571834 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:14.571759    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz"
May 26 16:34:23.501596 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:23.501557    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz"]
May 26 16:34:23.502021 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:23.501943    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz" podUID="5725fa76-0a7d-4363-8337-c5518ef7331f" containerName="kserve-container" containerID="cri-o://96d736265bfb17f88a0a6f91bb8b8923cb5c6bc7ab49b144f083d1565552bcc0" gracePeriod=30
May 26 16:34:23.502021 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:23.501990    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz" podUID="5725fa76-0a7d-4363-8337-c5518ef7331f" containerName="kube-rbac-proxy" containerID="cri-o://0820af451486246140a8c0ec6b25cb6ab66beb90f858828654a593b9f3977d3d" gracePeriod=30
May 26 16:34:23.590804 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:23.590769    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-75c1b6-predictor-57dff89c5b-9gt85"]
May 26 16:34:23.591041 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:23.591029    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83904f99-898e-4152-8952-8a73d2167656" containerName="kube-rbac-proxy"
May 26 16:34:23.591086 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:23.591043    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="83904f99-898e-4152-8952-8a73d2167656" containerName="kube-rbac-proxy"
May 26 16:34:23.591086 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:23.591058    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83904f99-898e-4152-8952-8a73d2167656" containerName="storage-initializer"
May 26 16:34:23.591086 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:23.591064    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="83904f99-898e-4152-8952-8a73d2167656" containerName="storage-initializer"
May 26 16:34:23.591086 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:23.591077    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83904f99-898e-4152-8952-8a73d2167656" containerName="kserve-container"
May 26 16:34:23.591086 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:23.591082    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="83904f99-898e-4152-8952-8a73d2167656" containerName="kserve-container"
May 26 16:34:23.591241 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:23.591121    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="83904f99-898e-4152-8952-8a73d2167656" containerName="kube-rbac-proxy"
May 26 16:34:23.591241 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:23.591129    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="83904f99-898e-4152-8952-8a73d2167656" containerName="kserve-container"
May 26 16:34:23.594222 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:23.594205    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-75c1b6-predictor-57dff89c5b-9gt85"
May 26 16:34:23.596172 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:23.596134    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-75c1b6-kube-rbac-proxy-sar-config\""
May 26 16:34:23.596340 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:23.596324    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-75c1b6-predictor-serving-cert\""
May 26 16:34:23.601757 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:23.601734    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-75c1b6-predictor-57dff89c5b-9gt85"]
May 26 16:34:23.690263 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:23.690220    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-primary-75c1b6-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b-isvc-primary-75c1b6-kube-rbac-proxy-sar-config\") pod \"isvc-primary-75c1b6-predictor-57dff89c5b-9gt85\" (UID: \"2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b\") " pod="kserve-ci-e2e-test/isvc-primary-75c1b6-predictor-57dff89c5b-9gt85"
May 26 16:34:23.690263 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:23.690264    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm676\" (UniqueName: \"kubernetes.io/projected/2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b-kube-api-access-rm676\") pod \"isvc-primary-75c1b6-predictor-57dff89c5b-9gt85\" (UID: \"2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b\") " pod="kserve-ci-e2e-test/isvc-primary-75c1b6-predictor-57dff89c5b-9gt85"
May 26 16:34:23.690508 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:23.690288    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b-proxy-tls\") pod \"isvc-primary-75c1b6-predictor-57dff89c5b-9gt85\" (UID: \"2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b\") " pod="kserve-ci-e2e-test/isvc-primary-75c1b6-predictor-57dff89c5b-9gt85"
May 26 16:34:23.690508 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:23.690379    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b-kserve-provision-location\") pod \"isvc-primary-75c1b6-predictor-57dff89c5b-9gt85\" (UID: \"2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b\") " pod="kserve-ci-e2e-test/isvc-primary-75c1b6-predictor-57dff89c5b-9gt85"
May 26 16:34:23.791721 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:23.791632    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-primary-75c1b6-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b-isvc-primary-75c1b6-kube-rbac-proxy-sar-config\") pod \"isvc-primary-75c1b6-predictor-57dff89c5b-9gt85\" (UID: \"2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b\") " pod="kserve-ci-e2e-test/isvc-primary-75c1b6-predictor-57dff89c5b-9gt85"
May 26 16:34:23.791721 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:23.791671    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rm676\" (UniqueName: \"kubernetes.io/projected/2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b-kube-api-access-rm676\") pod \"isvc-primary-75c1b6-predictor-57dff89c5b-9gt85\" (UID: \"2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b\") " pod="kserve-ci-e2e-test/isvc-primary-75c1b6-predictor-57dff89c5b-9gt85"
May 26 16:34:23.791721 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:23.791694    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b-proxy-tls\") pod \"isvc-primary-75c1b6-predictor-57dff89c5b-9gt85\" (UID: \"2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b\") " pod="kserve-ci-e2e-test/isvc-primary-75c1b6-predictor-57dff89c5b-9gt85"
May 26 16:34:23.792069 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:23.791733    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b-kserve-provision-location\") pod \"isvc-primary-75c1b6-predictor-57dff89c5b-9gt85\" (UID: \"2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b\") " pod="kserve-ci-e2e-test/isvc-primary-75c1b6-predictor-57dff89c5b-9gt85"
May 26 16:34:23.792132 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:23.792076    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b-kserve-provision-location\") pod \"isvc-primary-75c1b6-predictor-57dff89c5b-9gt85\" (UID: \"2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b\") " pod="kserve-ci-e2e-test/isvc-primary-75c1b6-predictor-57dff89c5b-9gt85"
May 26 16:34:23.792433 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:23.792407    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-primary-75c1b6-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b-isvc-primary-75c1b6-kube-rbac-proxy-sar-config\") pod \"isvc-primary-75c1b6-predictor-57dff89c5b-9gt85\" (UID: \"2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b\") " pod="kserve-ci-e2e-test/isvc-primary-75c1b6-predictor-57dff89c5b-9gt85"
May 26 16:34:23.794151 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:23.794126    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b-proxy-tls\") pod \"isvc-primary-75c1b6-predictor-57dff89c5b-9gt85\" (UID: \"2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b\") " pod="kserve-ci-e2e-test/isvc-primary-75c1b6-predictor-57dff89c5b-9gt85"
May 26 16:34:23.799920 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:23.799900    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm676\" (UniqueName: \"kubernetes.io/projected/2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b-kube-api-access-rm676\") pod \"isvc-primary-75c1b6-predictor-57dff89c5b-9gt85\" (UID: \"2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b\") " pod="kserve-ci-e2e-test/isvc-primary-75c1b6-predictor-57dff89c5b-9gt85"
May 26 16:34:23.816342 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:23.816314    2572 generic.go:358] "Generic (PLEG): container finished" podID="5725fa76-0a7d-4363-8337-c5518ef7331f" containerID="0820af451486246140a8c0ec6b25cb6ab66beb90f858828654a593b9f3977d3d" exitCode=2
May 26 16:34:23.816453 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:23.816384    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz" event={"ID":"5725fa76-0a7d-4363-8337-c5518ef7331f","Type":"ContainerDied","Data":"0820af451486246140a8c0ec6b25cb6ab66beb90f858828654a593b9f3977d3d"}
May 26 16:34:23.905346 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:23.905315    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-75c1b6-predictor-57dff89c5b-9gt85"
May 26 16:34:24.036985 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:24.036787    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-75c1b6-predictor-57dff89c5b-9gt85"]
May 26 16:34:24.039577 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:34:24.039531    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b1b9f7e_7dda_4a0c_826e_bfec3f2d8c8b.slice/crio-a8bc2e097fa82257544403c37147e978c6985b4be37ff07fbaffcf5d77faca3d WatchSource:0}: Error finding container a8bc2e097fa82257544403c37147e978c6985b4be37ff07fbaffcf5d77faca3d: Status 404 returned error can't find the container with id a8bc2e097fa82257544403c37147e978c6985b4be37ff07fbaffcf5d77faca3d
May 26 16:34:24.041385 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:24.041368    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 16:34:24.566591 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:24.566546    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz" podUID="5725fa76-0a7d-4363-8337-c5518ef7331f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
May 26 16:34:24.571799 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:24.571769    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz" podUID="5725fa76-0a7d-4363-8337-c5518ef7331f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
May 26 16:34:24.822851 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:24.822761    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-75c1b6-predictor-57dff89c5b-9gt85" event={"ID":"2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b","Type":"ContainerStarted","Data":"16380073f01241bddcd0db2f46d82df4bcf80c105bc2cdad6ce75b5b86fc57d6"}
May 26 16:34:24.822851 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:24.822797    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-75c1b6-predictor-57dff89c5b-9gt85" event={"ID":"2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b","Type":"ContainerStarted","Data":"a8bc2e097fa82257544403c37147e978c6985b4be37ff07fbaffcf5d77faca3d"}
May 26 16:34:27.252450 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:27.252427    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz"
May 26 16:34:27.323145 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:27.323055    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5725fa76-0a7d-4363-8337-c5518ef7331f-proxy-tls\") pod \"5725fa76-0a7d-4363-8337-c5518ef7331f\" (UID: \"5725fa76-0a7d-4363-8337-c5518ef7331f\") "
May 26 16:34:27.323145 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:27.323096    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9hgf\" (UniqueName: \"kubernetes.io/projected/5725fa76-0a7d-4363-8337-c5518ef7331f-kube-api-access-v9hgf\") pod \"5725fa76-0a7d-4363-8337-c5518ef7331f\" (UID: \"5725fa76-0a7d-4363-8337-c5518ef7331f\") "
May 26 16:34:27.323145 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:27.323125    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5725fa76-0a7d-4363-8337-c5518ef7331f-isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\") pod \"5725fa76-0a7d-4363-8337-c5518ef7331f\" (UID: \"5725fa76-0a7d-4363-8337-c5518ef7331f\") "
May 26 16:34:27.323420 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:27.323174    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5725fa76-0a7d-4363-8337-c5518ef7331f-kserve-provision-location\") pod \"5725fa76-0a7d-4363-8337-c5518ef7331f\" (UID: \"5725fa76-0a7d-4363-8337-c5518ef7331f\") "
May 26 16:34:27.323491 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:27.323429    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5725fa76-0a7d-4363-8337-c5518ef7331f-isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config") pod "5725fa76-0a7d-4363-8337-c5518ef7331f" (UID: "5725fa76-0a7d-4363-8337-c5518ef7331f"). InnerVolumeSpecName "isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:34:27.323580 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:27.323554    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5725fa76-0a7d-4363-8337-c5518ef7331f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5725fa76-0a7d-4363-8337-c5518ef7331f" (UID: "5725fa76-0a7d-4363-8337-c5518ef7331f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:34:27.325155 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:27.325134    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5725fa76-0a7d-4363-8337-c5518ef7331f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5725fa76-0a7d-4363-8337-c5518ef7331f" (UID: "5725fa76-0a7d-4363-8337-c5518ef7331f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:34:27.325212 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:27.325176    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5725fa76-0a7d-4363-8337-c5518ef7331f-kube-api-access-v9hgf" (OuterVolumeSpecName: "kube-api-access-v9hgf") pod "5725fa76-0a7d-4363-8337-c5518ef7331f" (UID: "5725fa76-0a7d-4363-8337-c5518ef7331f"). InnerVolumeSpecName "kube-api-access-v9hgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:34:27.424576 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:27.424539    2572 reconciler_common.go:299] "Volume detached for volume \"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5725fa76-0a7d-4363-8337-c5518ef7331f-isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:34:27.424576 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:27.424572    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5725fa76-0a7d-4363-8337-c5518ef7331f-kserve-provision-location\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:34:27.424794 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:27.424583    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5725fa76-0a7d-4363-8337-c5518ef7331f-proxy-tls\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:34:27.424794 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:27.424594    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9hgf\" (UniqueName: \"kubernetes.io/projected/5725fa76-0a7d-4363-8337-c5518ef7331f-kube-api-access-v9hgf\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:34:27.831928 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:27.831894    2572 generic.go:358] "Generic (PLEG): container finished" podID="2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b" containerID="16380073f01241bddcd0db2f46d82df4bcf80c105bc2cdad6ce75b5b86fc57d6" exitCode=0
May 26 16:34:27.832103 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:27.831973    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-75c1b6-predictor-57dff89c5b-9gt85" event={"ID":"2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b","Type":"ContainerDied","Data":"16380073f01241bddcd0db2f46d82df4bcf80c105bc2cdad6ce75b5b86fc57d6"}
May 26 16:34:27.833705 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:27.833678    2572 generic.go:358] "Generic (PLEG): container finished" podID="5725fa76-0a7d-4363-8337-c5518ef7331f" containerID="96d736265bfb17f88a0a6f91bb8b8923cb5c6bc7ab49b144f083d1565552bcc0" exitCode=0
May 26 16:34:27.833815 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:27.833747    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz" event={"ID":"5725fa76-0a7d-4363-8337-c5518ef7331f","Type":"ContainerDied","Data":"96d736265bfb17f88a0a6f91bb8b8923cb5c6bc7ab49b144f083d1565552bcc0"}
May 26 16:34:27.833815 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:27.833778    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz" event={"ID":"5725fa76-0a7d-4363-8337-c5518ef7331f","Type":"ContainerDied","Data":"f9a271a125fbe6135655678a7e3a5c0542877ef9c59b1dfffb82ea447602480c"}
May 26 16:34:27.833815 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:27.833791    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz"
May 26 16:34:27.833815 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:27.833798    2572 scope.go:117] "RemoveContainer" containerID="0820af451486246140a8c0ec6b25cb6ab66beb90f858828654a593b9f3977d3d"
May 26 16:34:27.841508 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:27.841487    2572 scope.go:117] "RemoveContainer" containerID="96d736265bfb17f88a0a6f91bb8b8923cb5c6bc7ab49b144f083d1565552bcc0"
May 26 16:34:27.849197 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:27.849169    2572 scope.go:117] "RemoveContainer" containerID="7cc9720d94dbe69bcc6a508c56a18616636e11b565b11f2b7663e371ebbaaf1d"
May 26 16:34:27.857728 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:27.857705    2572 scope.go:117] "RemoveContainer" containerID="0820af451486246140a8c0ec6b25cb6ab66beb90f858828654a593b9f3977d3d"
May 26 16:34:27.858051 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:34:27.858024    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0820af451486246140a8c0ec6b25cb6ab66beb90f858828654a593b9f3977d3d\": container with ID starting with 0820af451486246140a8c0ec6b25cb6ab66beb90f858828654a593b9f3977d3d not found: ID does not exist" containerID="0820af451486246140a8c0ec6b25cb6ab66beb90f858828654a593b9f3977d3d"
May 26 16:34:27.858130 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:27.858065    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0820af451486246140a8c0ec6b25cb6ab66beb90f858828654a593b9f3977d3d"} err="failed to get container status \"0820af451486246140a8c0ec6b25cb6ab66beb90f858828654a593b9f3977d3d\": rpc error: code = NotFound desc = could not find container \"0820af451486246140a8c0ec6b25cb6ab66beb90f858828654a593b9f3977d3d\": container with ID starting with 0820af451486246140a8c0ec6b25cb6ab66beb90f858828654a593b9f3977d3d not found: ID does not exist"
May 26 16:34:27.858130 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:27.858103    2572 scope.go:117] "RemoveContainer" containerID="96d736265bfb17f88a0a6f91bb8b8923cb5c6bc7ab49b144f083d1565552bcc0"
May 26 16:34:27.858578 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:34:27.858547    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96d736265bfb17f88a0a6f91bb8b8923cb5c6bc7ab49b144f083d1565552bcc0\": container with ID starting with 96d736265bfb17f88a0a6f91bb8b8923cb5c6bc7ab49b144f083d1565552bcc0 not found: ID does not exist" containerID="96d736265bfb17f88a0a6f91bb8b8923cb5c6bc7ab49b144f083d1565552bcc0"
May 26 16:34:27.858711 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:27.858587    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96d736265bfb17f88a0a6f91bb8b8923cb5c6bc7ab49b144f083d1565552bcc0"} err="failed to get container status \"96d736265bfb17f88a0a6f91bb8b8923cb5c6bc7ab49b144f083d1565552bcc0\": rpc error: code = NotFound desc = could not find container \"96d736265bfb17f88a0a6f91bb8b8923cb5c6bc7ab49b144f083d1565552bcc0\": container with ID starting with 96d736265bfb17f88a0a6f91bb8b8923cb5c6bc7ab49b144f083d1565552bcc0 not found: ID does not exist"
May 26 16:34:27.858711 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:27.858633    2572 scope.go:117] "RemoveContainer" containerID="7cc9720d94dbe69bcc6a508c56a18616636e11b565b11f2b7663e371ebbaaf1d"
May 26 16:34:27.859311 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:34:27.859102    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cc9720d94dbe69bcc6a508c56a18616636e11b565b11f2b7663e371ebbaaf1d\": container with ID starting with 7cc9720d94dbe69bcc6a508c56a18616636e11b565b11f2b7663e371ebbaaf1d not found: ID does not exist" containerID="7cc9720d94dbe69bcc6a508c56a18616636e11b565b11f2b7663e371ebbaaf1d"
May 26 16:34:27.859311 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:27.859137    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cc9720d94dbe69bcc6a508c56a18616636e11b565b11f2b7663e371ebbaaf1d"} err="failed to get container status \"7cc9720d94dbe69bcc6a508c56a18616636e11b565b11f2b7663e371ebbaaf1d\": rpc error: code = NotFound desc = could not find container \"7cc9720d94dbe69bcc6a508c56a18616636e11b565b11f2b7663e371ebbaaf1d\": container with ID starting with 7cc9720d94dbe69bcc6a508c56a18616636e11b565b11f2b7663e371ebbaaf1d not found: ID does not exist"
May 26 16:34:27.861325 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:27.861182    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz"]
May 26 16:34:27.862377 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:27.862357    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-6578f8ffc7-s9qrz"]
May 26 16:34:28.079352 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:28.079321    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5725fa76-0a7d-4363-8337-c5518ef7331f" path="/var/lib/kubelet/pods/5725fa76-0a7d-4363-8337-c5518ef7331f/volumes"
May 26 16:34:28.838637 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:28.838585    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-75c1b6-predictor-57dff89c5b-9gt85" event={"ID":"2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b","Type":"ContainerStarted","Data":"23f66704f836fc4bd2b8ab6d01e17de13c9bfb33bd1e75713ffd1e317e256c20"}
May 26 16:34:28.838637 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:28.838633    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-75c1b6-predictor-57dff89c5b-9gt85" event={"ID":"2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b","Type":"ContainerStarted","Data":"2e197b075c75b0da9aba783db9ee673da5d2a5d2f6d2f5196eb5889b0f5a87f2"}
May 26 16:34:28.839188 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:28.838855    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-75c1b6-predictor-57dff89c5b-9gt85"
May 26 16:34:28.856627 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:28.856551    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-75c1b6-predictor-57dff89c5b-9gt85" podStartSLOduration=5.856534005 podStartE2EDuration="5.856534005s" podCreationTimestamp="2026-05-26 16:34:23 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:34:28.854760374 +0000 UTC m=+1985.330487727" watchObservedRunningTime="2026-05-26 16:34:28.856534005 +0000 UTC m=+1985.332261355"
May 26 16:34:29.842778 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:29.842748    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-75c1b6-predictor-57dff89c5b-9gt85"
May 26 16:34:29.843843 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:29.843818    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-75c1b6-predictor-57dff89c5b-9gt85" podUID="2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
May 26 16:34:30.846070 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:30.846030    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-75c1b6-predictor-57dff89c5b-9gt85" podUID="2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
May 26 16:34:35.851039 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:35.851012    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-75c1b6-predictor-57dff89c5b-9gt85"
May 26 16:34:35.851597 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:35.851570    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-75c1b6-predictor-57dff89c5b-9gt85" podUID="2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
May 26 16:34:45.852272 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:45.852231    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-75c1b6-predictor-57dff89c5b-9gt85" podUID="2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
May 26 16:34:55.852109 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:34:55.852059    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-75c1b6-predictor-57dff89c5b-9gt85" podUID="2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
May 26 16:35:05.852346 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:05.852305    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-75c1b6-predictor-57dff89c5b-9gt85" podUID="2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
May 26 16:35:15.852372 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:15.852330    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-75c1b6-predictor-57dff89c5b-9gt85" podUID="2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
May 26 16:35:25.852148 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:25.852105    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-75c1b6-predictor-57dff89c5b-9gt85" podUID="2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
May 26 16:35:35.852760 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:35.852727    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-75c1b6-predictor-57dff89c5b-9gt85"
May 26 16:35:43.712484 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:43.712402    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-75c1b6-predictor-cdb677c59-skj8w"]
May 26 16:35:43.712958 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:43.712706    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5725fa76-0a7d-4363-8337-c5518ef7331f" containerName="kserve-container"
May 26 16:35:43.712958 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:43.712719    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5725fa76-0a7d-4363-8337-c5518ef7331f" containerName="kserve-container"
May 26 16:35:43.712958 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:43.712732    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5725fa76-0a7d-4363-8337-c5518ef7331f" containerName="storage-initializer"
May 26 16:35:43.712958 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:43.712737    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5725fa76-0a7d-4363-8337-c5518ef7331f" containerName="storage-initializer"
May 26 16:35:43.712958 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:43.712752    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5725fa76-0a7d-4363-8337-c5518ef7331f" containerName="kube-rbac-proxy"
May 26 16:35:43.712958 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:43.712760    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5725fa76-0a7d-4363-8337-c5518ef7331f" containerName="kube-rbac-proxy"
May 26 16:35:43.712958 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:43.712830    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="5725fa76-0a7d-4363-8337-c5518ef7331f" containerName="kserve-container"
May 26 16:35:43.712958 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:43.712844    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="5725fa76-0a7d-4363-8337-c5518ef7331f" containerName="kube-rbac-proxy"
May 26 16:35:43.717882 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:43.717852    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-75c1b6-predictor-cdb677c59-skj8w"
May 26 16:35:43.720091 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:43.720061    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-75c1b6-predictor-serving-cert\""
May 26 16:35:43.720305 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:43.720065    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-75c1b6\""
May 26 16:35:43.720555 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:43.720527    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-75c1b6-kube-rbac-proxy-sar-config\""
May 26 16:35:43.720695 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:43.720682    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-75c1b6-dockercfg-sn4t7\""
May 26 16:35:43.720767 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:43.720688    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
May 26 16:35:43.730259 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:43.730233    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-75c1b6-predictor-cdb677c59-skj8w"]
May 26 16:35:43.814881 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:43.814846    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08854e68-d614-485b-8243-b2766da71fbe-kserve-provision-location\") pod \"isvc-secondary-75c1b6-predictor-cdb677c59-skj8w\" (UID: \"08854e68-d614-485b-8243-b2766da71fbe\") " pod="kserve-ci-e2e-test/isvc-secondary-75c1b6-predictor-cdb677c59-skj8w"
May 26 16:35:43.814881 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:43.814887    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08854e68-d614-485b-8243-b2766da71fbe-proxy-tls\") pod \"isvc-secondary-75c1b6-predictor-cdb677c59-skj8w\" (UID: \"08854e68-d614-485b-8243-b2766da71fbe\") " pod="kserve-ci-e2e-test/isvc-secondary-75c1b6-predictor-cdb677c59-skj8w"
May 26 16:35:43.815165 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:43.814907    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/08854e68-d614-485b-8243-b2766da71fbe-cabundle-cert\") pod \"isvc-secondary-75c1b6-predictor-cdb677c59-skj8w\" (UID: \"08854e68-d614-485b-8243-b2766da71fbe\") " pod="kserve-ci-e2e-test/isvc-secondary-75c1b6-predictor-cdb677c59-skj8w"
May 26 16:35:43.815165 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:43.814981    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-secondary-75c1b6-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/08854e68-d614-485b-8243-b2766da71fbe-isvc-secondary-75c1b6-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-75c1b6-predictor-cdb677c59-skj8w\" (UID: \"08854e68-d614-485b-8243-b2766da71fbe\") " pod="kserve-ci-e2e-test/isvc-secondary-75c1b6-predictor-cdb677c59-skj8w"
May 26 16:35:43.815165 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:43.815038    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9lrh\" (UniqueName: \"kubernetes.io/projected/08854e68-d614-485b-8243-b2766da71fbe-kube-api-access-m9lrh\") pod \"isvc-secondary-75c1b6-predictor-cdb677c59-skj8w\" (UID: \"08854e68-d614-485b-8243-b2766da71fbe\") " pod="kserve-ci-e2e-test/isvc-secondary-75c1b6-predictor-cdb677c59-skj8w"
May 26 16:35:43.915845 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:43.915811    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08854e68-d614-485b-8243-b2766da71fbe-kserve-provision-location\") pod \"isvc-secondary-75c1b6-predictor-cdb677c59-skj8w\" (UID: \"08854e68-d614-485b-8243-b2766da71fbe\") " pod="kserve-ci-e2e-test/isvc-secondary-75c1b6-predictor-cdb677c59-skj8w"
May 26 16:35:43.915845 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:43.915850    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08854e68-d614-485b-8243-b2766da71fbe-proxy-tls\") pod \"isvc-secondary-75c1b6-predictor-cdb677c59-skj8w\" (UID: \"08854e68-d614-485b-8243-b2766da71fbe\") " pod="kserve-ci-e2e-test/isvc-secondary-75c1b6-predictor-cdb677c59-skj8w"
May 26 16:35:43.916088 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:43.915869    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/08854e68-d614-485b-8243-b2766da71fbe-cabundle-cert\") pod \"isvc-secondary-75c1b6-predictor-cdb677c59-skj8w\" (UID: \"08854e68-d614-485b-8243-b2766da71fbe\") " pod="kserve-ci-e2e-test/isvc-secondary-75c1b6-predictor-cdb677c59-skj8w"
May 26 16:35:43.916088 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:43.915915    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-secondary-75c1b6-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/08854e68-d614-485b-8243-b2766da71fbe-isvc-secondary-75c1b6-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-75c1b6-predictor-cdb677c59-skj8w\" (UID: \"08854e68-d614-485b-8243-b2766da71fbe\") " pod="kserve-ci-e2e-test/isvc-secondary-75c1b6-predictor-cdb677c59-skj8w"
May 26 16:35:43.916088 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:43.915950    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-m9lrh\" (UniqueName: \"kubernetes.io/projected/08854e68-d614-485b-8243-b2766da71fbe-kube-api-access-m9lrh\") pod \"isvc-secondary-75c1b6-predictor-cdb677c59-skj8w\" (UID: \"08854e68-d614-485b-8243-b2766da71fbe\") " pod="kserve-ci-e2e-test/isvc-secondary-75c1b6-predictor-cdb677c59-skj8w"
May 26 16:35:43.916088 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:35:43.915972    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-75c1b6-predictor-serving-cert: secret "isvc-secondary-75c1b6-predictor-serving-cert" not found
May 26 16:35:43.916088 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:35:43.916060    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08854e68-d614-485b-8243-b2766da71fbe-proxy-tls podName:08854e68-d614-485b-8243-b2766da71fbe nodeName:}" failed. No retries permitted until 2026-05-26 16:35:44.416037798 +0000 UTC m=+2060.891765128 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/08854e68-d614-485b-8243-b2766da71fbe-proxy-tls") pod "isvc-secondary-75c1b6-predictor-cdb677c59-skj8w" (UID: "08854e68-d614-485b-8243-b2766da71fbe") : secret "isvc-secondary-75c1b6-predictor-serving-cert" not found
May 26 16:35:43.916300 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:43.916282    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08854e68-d614-485b-8243-b2766da71fbe-kserve-provision-location\") pod \"isvc-secondary-75c1b6-predictor-cdb677c59-skj8w\" (UID: \"08854e68-d614-485b-8243-b2766da71fbe\") " pod="kserve-ci-e2e-test/isvc-secondary-75c1b6-predictor-cdb677c59-skj8w"
May 26 16:35:43.916561 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:43.916543    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-secondary-75c1b6-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/08854e68-d614-485b-8243-b2766da71fbe-isvc-secondary-75c1b6-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-75c1b6-predictor-cdb677c59-skj8w\" (UID: \"08854e68-d614-485b-8243-b2766da71fbe\") " pod="kserve-ci-e2e-test/isvc-secondary-75c1b6-predictor-cdb677c59-skj8w"
May 26 16:35:43.916598 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:43.916544    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/08854e68-d614-485b-8243-b2766da71fbe-cabundle-cert\") pod \"isvc-secondary-75c1b6-predictor-cdb677c59-skj8w\" (UID: \"08854e68-d614-485b-8243-b2766da71fbe\") " pod="kserve-ci-e2e-test/isvc-secondary-75c1b6-predictor-cdb677c59-skj8w"
May 26 16:35:43.924107 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:43.924088    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9lrh\" (UniqueName: \"kubernetes.io/projected/08854e68-d614-485b-8243-b2766da71fbe-kube-api-access-m9lrh\") pod \"isvc-secondary-75c1b6-predictor-cdb677c59-skj8w\" (UID: \"08854e68-d614-485b-8243-b2766da71fbe\") " pod="kserve-ci-e2e-test/isvc-secondary-75c1b6-predictor-cdb677c59-skj8w"
May 26 16:35:44.419969 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:44.419938    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08854e68-d614-485b-8243-b2766da71fbe-proxy-tls\") pod \"isvc-secondary-75c1b6-predictor-cdb677c59-skj8w\" (UID: \"08854e68-d614-485b-8243-b2766da71fbe\") " pod="kserve-ci-e2e-test/isvc-secondary-75c1b6-predictor-cdb677c59-skj8w"
May 26 16:35:44.422680 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:44.422655    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08854e68-d614-485b-8243-b2766da71fbe-proxy-tls\") pod \"isvc-secondary-75c1b6-predictor-cdb677c59-skj8w\" (UID: \"08854e68-d614-485b-8243-b2766da71fbe\") " pod="kserve-ci-e2e-test/isvc-secondary-75c1b6-predictor-cdb677c59-skj8w"
May 26 16:35:44.630166 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:44.630116    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-75c1b6-predictor-cdb677c59-skj8w"
May 26 16:35:44.760303 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:44.760131    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-75c1b6-predictor-cdb677c59-skj8w"]
May 26 16:35:44.763059 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:35:44.763030    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08854e68_d614_485b_8243_b2766da71fbe.slice/crio-b850e57082af750c3403a0c425c87280933cf486e3464c6c84cd22f7e3f6bb1c WatchSource:0}: Error finding container b850e57082af750c3403a0c425c87280933cf486e3464c6c84cd22f7e3f6bb1c: Status 404 returned error can't find the container with id b850e57082af750c3403a0c425c87280933cf486e3464c6c84cd22f7e3f6bb1c
May 26 16:35:45.046415 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:45.046330    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-75c1b6-predictor-cdb677c59-skj8w" event={"ID":"08854e68-d614-485b-8243-b2766da71fbe","Type":"ContainerStarted","Data":"51bc80c23536ec4f118d378da9263216ee80aa00544dd589ae3db60771f993ee"}
May 26 16:35:45.046415 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:45.046367    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-75c1b6-predictor-cdb677c59-skj8w" event={"ID":"08854e68-d614-485b-8243-b2766da71fbe","Type":"ContainerStarted","Data":"b850e57082af750c3403a0c425c87280933cf486e3464c6c84cd22f7e3f6bb1c"}
May 26 16:35:49.058848 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:49.058820    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-75c1b6-predictor-cdb677c59-skj8w_08854e68-d614-485b-8243-b2766da71fbe/storage-initializer/0.log"
May 26 16:35:49.059213 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:49.058857    2572 generic.go:358] "Generic (PLEG): container finished" podID="08854e68-d614-485b-8243-b2766da71fbe" containerID="51bc80c23536ec4f118d378da9263216ee80aa00544dd589ae3db60771f993ee" exitCode=1
May 26 16:35:49.059213 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:49.058897    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-75c1b6-predictor-cdb677c59-skj8w" event={"ID":"08854e68-d614-485b-8243-b2766da71fbe","Type":"ContainerDied","Data":"51bc80c23536ec4f118d378da9263216ee80aa00544dd589ae3db60771f993ee"}
May 26 16:35:50.063559 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:50.063533    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-75c1b6-predictor-cdb677c59-skj8w_08854e68-d614-485b-8243-b2766da71fbe/storage-initializer/0.log"
May 26 16:35:50.063960 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:50.063594    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-75c1b6-predictor-cdb677c59-skj8w" event={"ID":"08854e68-d614-485b-8243-b2766da71fbe","Type":"ContainerStarted","Data":"614a1c770063b88d438528c534a4cd3dc403f76f58a84912f905ee0d20653da1"}
May 26 16:35:54.078689 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:54.078345    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-75c1b6-predictor-cdb677c59-skj8w_08854e68-d614-485b-8243-b2766da71fbe/storage-initializer/1.log"
May 26 16:35:54.079161 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:54.078980    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-75c1b6-predictor-cdb677c59-skj8w_08854e68-d614-485b-8243-b2766da71fbe/storage-initializer/0.log"
May 26 16:35:54.079161 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:54.079016    2572 generic.go:358] "Generic (PLEG): container finished" podID="08854e68-d614-485b-8243-b2766da71fbe" containerID="614a1c770063b88d438528c534a4cd3dc403f76f58a84912f905ee0d20653da1" exitCode=1
May 26 16:35:54.082900 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:54.082865    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-75c1b6-predictor-cdb677c59-skj8w" event={"ID":"08854e68-d614-485b-8243-b2766da71fbe","Type":"ContainerDied","Data":"614a1c770063b88d438528c534a4cd3dc403f76f58a84912f905ee0d20653da1"}
May 26 16:35:54.083062 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:54.082907    2572 scope.go:117] "RemoveContainer" containerID="51bc80c23536ec4f118d378da9263216ee80aa00544dd589ae3db60771f993ee"
May 26 16:35:54.083266 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:54.083248    2572 scope.go:117] "RemoveContainer" containerID="51bc80c23536ec4f118d378da9263216ee80aa00544dd589ae3db60771f993ee"
May 26 16:35:54.093146 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:35:54.093114    2572 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-75c1b6-predictor-cdb677c59-skj8w_kserve-ci-e2e-test_08854e68-d614-485b-8243-b2766da71fbe_0 in pod sandbox b850e57082af750c3403a0c425c87280933cf486e3464c6c84cd22f7e3f6bb1c from index: no such id: '51bc80c23536ec4f118d378da9263216ee80aa00544dd589ae3db60771f993ee'" containerID="51bc80c23536ec4f118d378da9263216ee80aa00544dd589ae3db60771f993ee"
May 26 16:35:54.093219 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:35:54.093167    2572 kuberuntime_container.go:951] "Unhandled Error" err="failed to remove pod init container \"storage-initializer\": rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-75c1b6-predictor-cdb677c59-skj8w_kserve-ci-e2e-test_08854e68-d614-485b-8243-b2766da71fbe_0 in pod sandbox b850e57082af750c3403a0c425c87280933cf486e3464c6c84cd22f7e3f6bb1c from index: no such id: '51bc80c23536ec4f118d378da9263216ee80aa00544dd589ae3db60771f993ee'; Skipping pod \"isvc-secondary-75c1b6-predictor-cdb677c59-skj8w_kserve-ci-e2e-test(08854e68-d614-485b-8243-b2766da71fbe)\"" logger="UnhandledError"
May 26 16:35:54.094472 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:35:54.094450    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-75c1b6-predictor-cdb677c59-skj8w_kserve-ci-e2e-test(08854e68-d614-485b-8243-b2766da71fbe)\"" pod="kserve-ci-e2e-test/isvc-secondary-75c1b6-predictor-cdb677c59-skj8w" podUID="08854e68-d614-485b-8243-b2766da71fbe"
May 26 16:35:55.083378 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:55.083346    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-75c1b6-predictor-cdb677c59-skj8w_08854e68-d614-485b-8243-b2766da71fbe/storage-initializer/1.log"
May 26 16:35:59.759967 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:59.759935    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-75c1b6-predictor-cdb677c59-skj8w"]
May 26 16:35:59.805354 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:59.805281    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-75c1b6-predictor-57dff89c5b-9gt85"]
May 26 16:35:59.805671 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:59.805638    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-75c1b6-predictor-57dff89c5b-9gt85" podUID="2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b" containerName="kserve-container" containerID="cri-o://2e197b075c75b0da9aba783db9ee673da5d2a5d2f6d2f5196eb5889b0f5a87f2" gracePeriod=30
May 26 16:35:59.805803 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:59.805683    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-75c1b6-predictor-57dff89c5b-9gt85" podUID="2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b" containerName="kube-rbac-proxy" containerID="cri-o://23f66704f836fc4bd2b8ab6d01e17de13c9bfb33bd1e75713ffd1e317e256c20" gracePeriod=30
May 26 16:35:59.873118 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:59.873079    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4"]
May 26 16:35:59.876445 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:59.876420    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4"
May 26 16:35:59.878730 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:59.878702    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-919f7e-kube-rbac-proxy-sar-config\""
May 26 16:35:59.878898 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:59.878729    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-919f7e-dockercfg-zzn6l\""
May 26 16:35:59.878898 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:59.878731    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-919f7e\""
May 26 16:35:59.878898 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:59.878731    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-919f7e-predictor-serving-cert\""
May 26 16:35:59.887438 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:59.887400    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4"]
May 26 16:35:59.926049 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:59.926023    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-75c1b6-predictor-cdb677c59-skj8w_08854e68-d614-485b-8243-b2766da71fbe/storage-initializer/1.log"
May 26 16:35:59.926176 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:59.926105    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-75c1b6-predictor-cdb677c59-skj8w"
May 26 16:35:59.942429 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:59.942375    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-init-fail-919f7e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/409f8af5-196a-4d5f-92da-7bd629992251-isvc-init-fail-919f7e-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4\" (UID: \"409f8af5-196a-4d5f-92da-7bd629992251\") " pod="kserve-ci-e2e-test/isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4"
May 26 16:35:59.942568 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:59.942478    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/409f8af5-196a-4d5f-92da-7bd629992251-kserve-provision-location\") pod \"isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4\" (UID: \"409f8af5-196a-4d5f-92da-7bd629992251\") " pod="kserve-ci-e2e-test/isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4"
May 26 16:35:59.942568 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:59.942507    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/409f8af5-196a-4d5f-92da-7bd629992251-cabundle-cert\") pod \"isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4\" (UID: \"409f8af5-196a-4d5f-92da-7bd629992251\") " pod="kserve-ci-e2e-test/isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4"
May 26 16:35:59.942568 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:59.942539    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7d27\" (UniqueName: \"kubernetes.io/projected/409f8af5-196a-4d5f-92da-7bd629992251-kube-api-access-b7d27\") pod \"isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4\" (UID: \"409f8af5-196a-4d5f-92da-7bd629992251\") " pod="kserve-ci-e2e-test/isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4"
May 26 16:35:59.942568 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:35:59.942563    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/409f8af5-196a-4d5f-92da-7bd629992251-proxy-tls\") pod \"isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4\" (UID: \"409f8af5-196a-4d5f-92da-7bd629992251\") " pod="kserve-ci-e2e-test/isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4"
May 26 16:36:00.043540 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:00.043441    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08854e68-d614-485b-8243-b2766da71fbe-kserve-provision-location\") pod \"08854e68-d614-485b-8243-b2766da71fbe\" (UID: \"08854e68-d614-485b-8243-b2766da71fbe\") "
May 26 16:36:00.043540 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:00.043497    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m9lrh\" (UniqueName: \"kubernetes.io/projected/08854e68-d614-485b-8243-b2766da71fbe-kube-api-access-m9lrh\") pod \"08854e68-d614-485b-8243-b2766da71fbe\" (UID: \"08854e68-d614-485b-8243-b2766da71fbe\") "
May 26 16:36:00.043540 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:00.043520    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/08854e68-d614-485b-8243-b2766da71fbe-cabundle-cert\") pod \"08854e68-d614-485b-8243-b2766da71fbe\" (UID: \"08854e68-d614-485b-8243-b2766da71fbe\") "
May 26 16:36:00.043858 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:00.043652    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08854e68-d614-485b-8243-b2766da71fbe-proxy-tls\") pod \"08854e68-d614-485b-8243-b2766da71fbe\" (UID: \"08854e68-d614-485b-8243-b2766da71fbe\") "
May 26 16:36:00.043858 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:00.043697    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-secondary-75c1b6-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/08854e68-d614-485b-8243-b2766da71fbe-isvc-secondary-75c1b6-kube-rbac-proxy-sar-config\") pod \"08854e68-d614-485b-8243-b2766da71fbe\" (UID: \"08854e68-d614-485b-8243-b2766da71fbe\") "
May 26 16:36:00.043858 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:00.043842    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/409f8af5-196a-4d5f-92da-7bd629992251-kserve-provision-location\") pod \"isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4\" (UID: \"409f8af5-196a-4d5f-92da-7bd629992251\") " pod="kserve-ci-e2e-test/isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4"
May 26 16:36:00.044014 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:00.043874    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/409f8af5-196a-4d5f-92da-7bd629992251-cabundle-cert\") pod \"isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4\" (UID: \"409f8af5-196a-4d5f-92da-7bd629992251\") " pod="kserve-ci-e2e-test/isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4"
May 26 16:36:00.044014 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:00.043868    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08854e68-d614-485b-8243-b2766da71fbe-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "08854e68-d614-485b-8243-b2766da71fbe" (UID: "08854e68-d614-485b-8243-b2766da71fbe"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:36:00.044014 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:00.043966    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-b7d27\" (UniqueName: \"kubernetes.io/projected/409f8af5-196a-4d5f-92da-7bd629992251-kube-api-access-b7d27\") pod \"isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4\" (UID: \"409f8af5-196a-4d5f-92da-7bd629992251\") " pod="kserve-ci-e2e-test/isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4"
May 26 16:36:00.044161 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:00.044003    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08854e68-d614-485b-8243-b2766da71fbe-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "08854e68-d614-485b-8243-b2766da71fbe" (UID: "08854e68-d614-485b-8243-b2766da71fbe"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:36:00.044161 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:00.044013    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/409f8af5-196a-4d5f-92da-7bd629992251-proxy-tls\") pod \"isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4\" (UID: \"409f8af5-196a-4d5f-92da-7bd629992251\") " pod="kserve-ci-e2e-test/isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4"
May 26 16:36:00.044161 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:00.044104    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08854e68-d614-485b-8243-b2766da71fbe-isvc-secondary-75c1b6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-secondary-75c1b6-kube-rbac-proxy-sar-config") pod "08854e68-d614-485b-8243-b2766da71fbe" (UID: "08854e68-d614-485b-8243-b2766da71fbe"). InnerVolumeSpecName "isvc-secondary-75c1b6-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:36:00.044161 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:00.044124    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-init-fail-919f7e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/409f8af5-196a-4d5f-92da-7bd629992251-isvc-init-fail-919f7e-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4\" (UID: \"409f8af5-196a-4d5f-92da-7bd629992251\") " pod="kserve-ci-e2e-test/isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4"
May 26 16:36:00.044409 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:00.044197    2572 reconciler_common.go:299] "Volume detached for volume \"isvc-secondary-75c1b6-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/08854e68-d614-485b-8243-b2766da71fbe-isvc-secondary-75c1b6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:36:00.044409 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:00.044216    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08854e68-d614-485b-8243-b2766da71fbe-kserve-provision-location\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:36:00.044409 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:00.044232    2572 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/08854e68-d614-485b-8243-b2766da71fbe-cabundle-cert\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:36:00.044409 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:00.044287    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/409f8af5-196a-4d5f-92da-7bd629992251-kserve-provision-location\") pod \"isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4\" (UID: \"409f8af5-196a-4d5f-92da-7bd629992251\") " pod="kserve-ci-e2e-test/isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4"
May 26 16:36:00.044742 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:00.044722    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/409f8af5-196a-4d5f-92da-7bd629992251-cabundle-cert\") pod \"isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4\" (UID: \"409f8af5-196a-4d5f-92da-7bd629992251\") " pod="kserve-ci-e2e-test/isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4"
May 26 16:36:00.044949 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:00.044924    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-init-fail-919f7e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/409f8af5-196a-4d5f-92da-7bd629992251-isvc-init-fail-919f7e-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4\" (UID: \"409f8af5-196a-4d5f-92da-7bd629992251\") " pod="kserve-ci-e2e-test/isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4"
May 26 16:36:00.045948 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:00.045924    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08854e68-d614-485b-8243-b2766da71fbe-kube-api-access-m9lrh" (OuterVolumeSpecName: "kube-api-access-m9lrh") pod "08854e68-d614-485b-8243-b2766da71fbe" (UID: "08854e68-d614-485b-8243-b2766da71fbe"). InnerVolumeSpecName "kube-api-access-m9lrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:36:00.046039 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:00.046022    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08854e68-d614-485b-8243-b2766da71fbe-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "08854e68-d614-485b-8243-b2766da71fbe" (UID: "08854e68-d614-485b-8243-b2766da71fbe"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:36:00.046483 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:00.046467    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/409f8af5-196a-4d5f-92da-7bd629992251-proxy-tls\") pod \"isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4\" (UID: \"409f8af5-196a-4d5f-92da-7bd629992251\") " pod="kserve-ci-e2e-test/isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4"
May 26 16:36:00.052431 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:00.052407    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7d27\" (UniqueName: \"kubernetes.io/projected/409f8af5-196a-4d5f-92da-7bd629992251-kube-api-access-b7d27\") pod \"isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4\" (UID: \"409f8af5-196a-4d5f-92da-7bd629992251\") " pod="kserve-ci-e2e-test/isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4"
May 26 16:36:00.100881 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:00.100850    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-75c1b6-predictor-cdb677c59-skj8w_08854e68-d614-485b-8243-b2766da71fbe/storage-initializer/1.log"
May 26 16:36:00.101044 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:00.100964    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-75c1b6-predictor-cdb677c59-skj8w" event={"ID":"08854e68-d614-485b-8243-b2766da71fbe","Type":"ContainerDied","Data":"b850e57082af750c3403a0c425c87280933cf486e3464c6c84cd22f7e3f6bb1c"}
May 26 16:36:00.101044 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:00.100995    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-75c1b6-predictor-cdb677c59-skj8w"
May 26 16:36:00.101044 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:00.101011    2572 scope.go:117] "RemoveContainer" containerID="614a1c770063b88d438528c534a4cd3dc403f76f58a84912f905ee0d20653da1"
May 26 16:36:00.103288 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:00.103261    2572 generic.go:358] "Generic (PLEG): container finished" podID="2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b" containerID="23f66704f836fc4bd2b8ab6d01e17de13c9bfb33bd1e75713ffd1e317e256c20" exitCode=2
May 26 16:36:00.103385 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:00.103298    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-75c1b6-predictor-57dff89c5b-9gt85" event={"ID":"2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b","Type":"ContainerDied","Data":"23f66704f836fc4bd2b8ab6d01e17de13c9bfb33bd1e75713ffd1e317e256c20"}
May 26 16:36:00.128898 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:00.128868    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-75c1b6-predictor-cdb677c59-skj8w"]
May 26 16:36:00.132541 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:00.132512    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-75c1b6-predictor-cdb677c59-skj8w"]
May 26 16:36:00.145560 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:00.145525    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9lrh\" (UniqueName: \"kubernetes.io/projected/08854e68-d614-485b-8243-b2766da71fbe-kube-api-access-m9lrh\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:36:00.145560 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:00.145555    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/08854e68-d614-485b-8243-b2766da71fbe-proxy-tls\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:36:00.188937 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:00.188905    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4"
May 26 16:36:00.319174 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:00.319151    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4"]
May 26 16:36:00.321797 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:36:00.321768    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod409f8af5_196a_4d5f_92da_7bd629992251.slice/crio-6b850349f8f2488363433777f9e3d800823608711dd4d8347a28c06973060645 WatchSource:0}: Error finding container 6b850349f8f2488363433777f9e3d800823608711dd4d8347a28c06973060645: Status 404 returned error can't find the container with id 6b850349f8f2488363433777f9e3d800823608711dd4d8347a28c06973060645
May 26 16:36:00.846989 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:00.846942    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-75c1b6-predictor-57dff89c5b-9gt85" podUID="2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
May 26 16:36:01.107662 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:01.107545    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4" event={"ID":"409f8af5-196a-4d5f-92da-7bd629992251","Type":"ContainerStarted","Data":"6a7774064bde0cc41bf4504c952229a1d2954f8226352aaf461d77fc7c5fbfe0"}
May 26 16:36:01.107662 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:01.107584    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4" event={"ID":"409f8af5-196a-4d5f-92da-7bd629992251","Type":"ContainerStarted","Data":"6b850349f8f2488363433777f9e3d800823608711dd4d8347a28c06973060645"}
May 26 16:36:02.078598 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:02.078562    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08854e68-d614-485b-8243-b2766da71fbe" path="/var/lib/kubelet/pods/08854e68-d614-485b-8243-b2766da71fbe/volumes"
May 26 16:36:04.449974 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:04.449944    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-75c1b6-predictor-57dff89c5b-9gt85"
May 26 16:36:04.584363 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:04.584276    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b-kserve-provision-location\") pod \"2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b\" (UID: \"2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b\") "
May 26 16:36:04.584363 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:04.584325    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rm676\" (UniqueName: \"kubernetes.io/projected/2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b-kube-api-access-rm676\") pod \"2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b\" (UID: \"2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b\") "
May 26 16:36:04.584363 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:04.584358    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b-proxy-tls\") pod \"2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b\" (UID: \"2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b\") "
May 26 16:36:04.584624 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:04.584406    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-primary-75c1b6-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b-isvc-primary-75c1b6-kube-rbac-proxy-sar-config\") pod \"2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b\" (UID: \"2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b\") "
May 26 16:36:04.584729 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:04.584702    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b" (UID: "2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:36:04.584808 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:04.584780    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b-isvc-primary-75c1b6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-primary-75c1b6-kube-rbac-proxy-sar-config") pod "2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b" (UID: "2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b"). InnerVolumeSpecName "isvc-primary-75c1b6-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:36:04.586505 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:04.586476    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b-kube-api-access-rm676" (OuterVolumeSpecName: "kube-api-access-rm676") pod "2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b" (UID: "2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b"). InnerVolumeSpecName "kube-api-access-rm676". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:36:04.586505 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:04.586486    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b" (UID: "2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:36:04.685034 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:04.685003    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b-kserve-provision-location\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:36:04.685034 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:04.685031    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm676\" (UniqueName: \"kubernetes.io/projected/2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b-kube-api-access-rm676\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:36:04.685227 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:04.685041    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b-proxy-tls\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:36:04.685227 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:04.685053    2572 reconciler_common.go:299] "Volume detached for volume \"isvc-primary-75c1b6-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b-isvc-primary-75c1b6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:36:05.120484 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:05.120453    2572 generic.go:358] "Generic (PLEG): container finished" podID="2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b" containerID="2e197b075c75b0da9aba783db9ee673da5d2a5d2f6d2f5196eb5889b0f5a87f2" exitCode=0
May 26 16:36:05.120658 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:05.120546    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-75c1b6-predictor-57dff89c5b-9gt85"
May 26 16:36:05.120658 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:05.120545    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-75c1b6-predictor-57dff89c5b-9gt85" event={"ID":"2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b","Type":"ContainerDied","Data":"2e197b075c75b0da9aba783db9ee673da5d2a5d2f6d2f5196eb5889b0f5a87f2"}
May 26 16:36:05.120658 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:05.120591    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-75c1b6-predictor-57dff89c5b-9gt85" event={"ID":"2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b","Type":"ContainerDied","Data":"a8bc2e097fa82257544403c37147e978c6985b4be37ff07fbaffcf5d77faca3d"}
May 26 16:36:05.120658 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:05.120627    2572 scope.go:117] "RemoveContainer" containerID="23f66704f836fc4bd2b8ab6d01e17de13c9bfb33bd1e75713ffd1e317e256c20"
May 26 16:36:05.128628 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:05.128591    2572 scope.go:117] "RemoveContainer" containerID="2e197b075c75b0da9aba783db9ee673da5d2a5d2f6d2f5196eb5889b0f5a87f2"
May 26 16:36:05.135325 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:05.135309    2572 scope.go:117] "RemoveContainer" containerID="16380073f01241bddcd0db2f46d82df4bcf80c105bc2cdad6ce75b5b86fc57d6"
May 26 16:36:05.140681 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:05.140659    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-75c1b6-predictor-57dff89c5b-9gt85"]
May 26 16:36:05.142595 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:05.142570    2572 scope.go:117] "RemoveContainer" containerID="23f66704f836fc4bd2b8ab6d01e17de13c9bfb33bd1e75713ffd1e317e256c20"
May 26 16:36:05.142932 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:36:05.142904    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23f66704f836fc4bd2b8ab6d01e17de13c9bfb33bd1e75713ffd1e317e256c20\": container with ID starting with 23f66704f836fc4bd2b8ab6d01e17de13c9bfb33bd1e75713ffd1e317e256c20 not found: ID does not exist" containerID="23f66704f836fc4bd2b8ab6d01e17de13c9bfb33bd1e75713ffd1e317e256c20"
May 26 16:36:05.143016 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:05.142945    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23f66704f836fc4bd2b8ab6d01e17de13c9bfb33bd1e75713ffd1e317e256c20"} err="failed to get container status \"23f66704f836fc4bd2b8ab6d01e17de13c9bfb33bd1e75713ffd1e317e256c20\": rpc error: code = NotFound desc = could not find container \"23f66704f836fc4bd2b8ab6d01e17de13c9bfb33bd1e75713ffd1e317e256c20\": container with ID starting with 23f66704f836fc4bd2b8ab6d01e17de13c9bfb33bd1e75713ffd1e317e256c20 not found: ID does not exist"
May 26 16:36:05.143016 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:05.142972    2572 scope.go:117] "RemoveContainer" containerID="2e197b075c75b0da9aba783db9ee673da5d2a5d2f6d2f5196eb5889b0f5a87f2"
May 26 16:36:05.143249 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:36:05.143233    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e197b075c75b0da9aba783db9ee673da5d2a5d2f6d2f5196eb5889b0f5a87f2\": container with ID starting with 2e197b075c75b0da9aba783db9ee673da5d2a5d2f6d2f5196eb5889b0f5a87f2 not found: ID does not exist" containerID="2e197b075c75b0da9aba783db9ee673da5d2a5d2f6d2f5196eb5889b0f5a87f2"
May 26 16:36:05.143291 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:05.143256    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e197b075c75b0da9aba783db9ee673da5d2a5d2f6d2f5196eb5889b0f5a87f2"} err="failed to get container status \"2e197b075c75b0da9aba783db9ee673da5d2a5d2f6d2f5196eb5889b0f5a87f2\": rpc error: code = NotFound desc = could not find container \"2e197b075c75b0da9aba783db9ee673da5d2a5d2f6d2f5196eb5889b0f5a87f2\": container with ID starting with 2e197b075c75b0da9aba783db9ee673da5d2a5d2f6d2f5196eb5889b0f5a87f2 not found: ID does not exist"
May 26 16:36:05.143291 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:05.143286    2572 scope.go:117] "RemoveContainer" containerID="16380073f01241bddcd0db2f46d82df4bcf80c105bc2cdad6ce75b5b86fc57d6"
May 26 16:36:05.143502 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:36:05.143487    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16380073f01241bddcd0db2f46d82df4bcf80c105bc2cdad6ce75b5b86fc57d6\": container with ID starting with 16380073f01241bddcd0db2f46d82df4bcf80c105bc2cdad6ce75b5b86fc57d6 not found: ID does not exist" containerID="16380073f01241bddcd0db2f46d82df4bcf80c105bc2cdad6ce75b5b86fc57d6"
May 26 16:36:05.143564 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:05.143504    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16380073f01241bddcd0db2f46d82df4bcf80c105bc2cdad6ce75b5b86fc57d6"} err="failed to get container status \"16380073f01241bddcd0db2f46d82df4bcf80c105bc2cdad6ce75b5b86fc57d6\": rpc error: code = NotFound desc = could not find container \"16380073f01241bddcd0db2f46d82df4bcf80c105bc2cdad6ce75b5b86fc57d6\": container with ID starting with 16380073f01241bddcd0db2f46d82df4bcf80c105bc2cdad6ce75b5b86fc57d6 not found: ID does not exist"
May 26 16:36:05.144336 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:05.144317    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-75c1b6-predictor-57dff89c5b-9gt85"]
May 26 16:36:06.079923 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:06.079876    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b" path="/var/lib/kubelet/pods/2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b/volumes"
May 26 16:36:06.125917 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:06.125887    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4_409f8af5-196a-4d5f-92da-7bd629992251/storage-initializer/0.log"
May 26 16:36:06.126102 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:06.125926    2572 generic.go:358] "Generic (PLEG): container finished" podID="409f8af5-196a-4d5f-92da-7bd629992251" containerID="6a7774064bde0cc41bf4504c952229a1d2954f8226352aaf461d77fc7c5fbfe0" exitCode=1
May 26 16:36:06.126102 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:06.125991    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4" event={"ID":"409f8af5-196a-4d5f-92da-7bd629992251","Type":"ContainerDied","Data":"6a7774064bde0cc41bf4504c952229a1d2954f8226352aaf461d77fc7c5fbfe0"}
May 26 16:36:07.130457 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:07.130428    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4_409f8af5-196a-4d5f-92da-7bd629992251/storage-initializer/0.log"
May 26 16:36:07.130865 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:07.130506    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4" event={"ID":"409f8af5-196a-4d5f-92da-7bd629992251","Type":"ContainerStarted","Data":"4b45c9fadebbf221f2b8f10995e496a3ac82847fe7e64734212674e179dc76c3"}
May 26 16:36:09.871170 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:09.871133    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4"]
May 26 16:36:09.871598 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:09.871473    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4" podUID="409f8af5-196a-4d5f-92da-7bd629992251" containerName="storage-initializer" containerID="cri-o://4b45c9fadebbf221f2b8f10995e496a3ac82847fe7e64734212674e179dc76c3" gracePeriod=30
May 26 16:36:09.993281 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:09.993244    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-kc74j"]
May 26 16:36:09.993523 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:09.993512    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08854e68-d614-485b-8243-b2766da71fbe" containerName="storage-initializer"
May 26 16:36:09.993582 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:09.993526    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="08854e68-d614-485b-8243-b2766da71fbe" containerName="storage-initializer"
May 26 16:36:09.993582 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:09.993537    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b" containerName="storage-initializer"
May 26 16:36:09.993582 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:09.993544    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b" containerName="storage-initializer"
May 26 16:36:09.993582 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:09.993555    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b" containerName="kserve-container"
May 26 16:36:09.993582 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:09.993561    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b" containerName="kserve-container"
May 26 16:36:09.993582 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:09.993571    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b" containerName="kube-rbac-proxy"
May 26 16:36:09.993582 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:09.993576    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b" containerName="kube-rbac-proxy"
May 26 16:36:09.993817 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:09.993643    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b" containerName="kserve-container"
May 26 16:36:09.993817 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:09.993651    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="2b1b9f7e-7dda-4a0c-826e-bfec3f2d8c8b" containerName="kube-rbac-proxy"
May 26 16:36:09.993817 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:09.993657    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="08854e68-d614-485b-8243-b2766da71fbe" containerName="storage-initializer"
May 26 16:36:09.993817 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:09.993664    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="08854e68-d614-485b-8243-b2766da71fbe" containerName="storage-initializer"
May 26 16:36:09.993817 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:09.993714    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08854e68-d614-485b-8243-b2766da71fbe" containerName="storage-initializer"
May 26 16:36:09.993817 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:09.993720    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="08854e68-d614-485b-8243-b2766da71fbe" containerName="storage-initializer"
May 26 16:36:09.996908 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:09.996886    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-kc74j"
May 26 16:36:09.998913 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:09.998891    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
May 26 16:36:09.999049 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:09.998914    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-c9tjh\""
May 26 16:36:09.999049 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:09.998914    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
May 26 16:36:10.007621 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:10.007433    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-kc74j"]
May 26 16:36:10.134501 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:10.134406    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6294l\" (UniqueName: \"kubernetes.io/projected/54b401eb-76b3-4ca5-b4c3-de9c9d6a2081-kube-api-access-6294l\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-kc74j\" (UID: \"54b401eb-76b3-4ca5-b4c3-de9c9d6a2081\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-kc74j"
May 26 16:36:10.134501 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:10.134458    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54b401eb-76b3-4ca5-b4c3-de9c9d6a2081-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-kc74j\" (UID: \"54b401eb-76b3-4ca5-b4c3-de9c9d6a2081\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-kc74j"
May 26 16:36:10.134746 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:10.134540    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/54b401eb-76b3-4ca5-b4c3-de9c9d6a2081-isvc-predictive-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-kc74j\" (UID: \"54b401eb-76b3-4ca5-b4c3-de9c9d6a2081\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-kc74j"
May 26 16:36:10.134746 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:10.134583    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54b401eb-76b3-4ca5-b4c3-de9c9d6a2081-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-kc74j\" (UID: \"54b401eb-76b3-4ca5-b4c3-de9c9d6a2081\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-kc74j"
May 26 16:36:10.234999 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:10.234953    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/54b401eb-76b3-4ca5-b4c3-de9c9d6a2081-isvc-predictive-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-kc74j\" (UID: \"54b401eb-76b3-4ca5-b4c3-de9c9d6a2081\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-kc74j"
May 26 16:36:10.235162 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:10.235018    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54b401eb-76b3-4ca5-b4c3-de9c9d6a2081-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-kc74j\" (UID: \"54b401eb-76b3-4ca5-b4c3-de9c9d6a2081\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-kc74j"
May 26 16:36:10.235162 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:10.235055    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6294l\" (UniqueName: \"kubernetes.io/projected/54b401eb-76b3-4ca5-b4c3-de9c9d6a2081-kube-api-access-6294l\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-kc74j\" (UID: \"54b401eb-76b3-4ca5-b4c3-de9c9d6a2081\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-kc74j"
May 26 16:36:10.235162 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:10.235091    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54b401eb-76b3-4ca5-b4c3-de9c9d6a2081-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-kc74j\" (UID: \"54b401eb-76b3-4ca5-b4c3-de9c9d6a2081\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-kc74j"
May 26 16:36:10.235480 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:10.235456    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54b401eb-76b3-4ca5-b4c3-de9c9d6a2081-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-kc74j\" (UID: \"54b401eb-76b3-4ca5-b4c3-de9c9d6a2081\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-kc74j"
May 26 16:36:10.235712 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:10.235693    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/54b401eb-76b3-4ca5-b4c3-de9c9d6a2081-isvc-predictive-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-kc74j\" (UID: \"54b401eb-76b3-4ca5-b4c3-de9c9d6a2081\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-kc74j"
May 26 16:36:10.237456 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:10.237438    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54b401eb-76b3-4ca5-b4c3-de9c9d6a2081-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-kc74j\" (UID: \"54b401eb-76b3-4ca5-b4c3-de9c9d6a2081\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-kc74j"
May 26 16:36:10.244069 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:10.244042    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6294l\" (UniqueName: \"kubernetes.io/projected/54b401eb-76b3-4ca5-b4c3-de9c9d6a2081-kube-api-access-6294l\") pod \"isvc-predictive-sklearn-predictor-cd7c759c9-kc74j\" (UID: \"54b401eb-76b3-4ca5-b4c3-de9c9d6a2081\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-kc74j"
May 26 16:36:10.311550 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:10.311514    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-kc74j"
May 26 16:36:10.444300 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:10.444149    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-kc74j"]
May 26 16:36:10.446756 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:36:10.446729    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54b401eb_76b3_4ca5_b4c3_de9c9d6a2081.slice/crio-292f7b14c8b201d66733f916b5b4dd3d3a4506f938bee61fbae0d044d3883ed0 WatchSource:0}: Error finding container 292f7b14c8b201d66733f916b5b4dd3d3a4506f938bee61fbae0d044d3883ed0: Status 404 returned error can't find the container with id 292f7b14c8b201d66733f916b5b4dd3d3a4506f938bee61fbae0d044d3883ed0
May 26 16:36:11.114890 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:11.114867    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4_409f8af5-196a-4d5f-92da-7bd629992251/storage-initializer/1.log"
May 26 16:36:11.115219 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:11.115209    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4_409f8af5-196a-4d5f-92da-7bd629992251/storage-initializer/0.log"
May 26 16:36:11.115279 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:11.115268    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4"
May 26 16:36:11.142538 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:11.142511    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4_409f8af5-196a-4d5f-92da-7bd629992251/storage-initializer/1.log"
May 26 16:36:11.142874 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:11.142859    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4_409f8af5-196a-4d5f-92da-7bd629992251/storage-initializer/0.log"
May 26 16:36:11.142939 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:11.142893    2572 generic.go:358] "Generic (PLEG): container finished" podID="409f8af5-196a-4d5f-92da-7bd629992251" containerID="4b45c9fadebbf221f2b8f10995e496a3ac82847fe7e64734212674e179dc76c3" exitCode=1
May 26 16:36:11.142991 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:11.142977    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4"
May 26 16:36:11.143032 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:11.142977    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4" event={"ID":"409f8af5-196a-4d5f-92da-7bd629992251","Type":"ContainerDied","Data":"4b45c9fadebbf221f2b8f10995e496a3ac82847fe7e64734212674e179dc76c3"}
May 26 16:36:11.143032 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:11.143022    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4" event={"ID":"409f8af5-196a-4d5f-92da-7bd629992251","Type":"ContainerDied","Data":"6b850349f8f2488363433777f9e3d800823608711dd4d8347a28c06973060645"}
May 26 16:36:11.143120 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:11.143045    2572 scope.go:117] "RemoveContainer" containerID="4b45c9fadebbf221f2b8f10995e496a3ac82847fe7e64734212674e179dc76c3"
May 26 16:36:11.144290 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:11.144259    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-kc74j" event={"ID":"54b401eb-76b3-4ca5-b4c3-de9c9d6a2081","Type":"ContainerStarted","Data":"b9d41b9eafbf3ae6e338e05b209698a894bd5941bf0a70ebe25da5d437fd4fb4"}
May 26 16:36:11.144290 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:11.144285    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-kc74j" event={"ID":"54b401eb-76b3-4ca5-b4c3-de9c9d6a2081","Type":"ContainerStarted","Data":"292f7b14c8b201d66733f916b5b4dd3d3a4506f938bee61fbae0d044d3883ed0"}
May 26 16:36:11.150572 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:11.150557    2572 scope.go:117] "RemoveContainer" containerID="6a7774064bde0cc41bf4504c952229a1d2954f8226352aaf461d77fc7c5fbfe0"
May 26 16:36:11.157735 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:11.157718    2572 scope.go:117] "RemoveContainer" containerID="4b45c9fadebbf221f2b8f10995e496a3ac82847fe7e64734212674e179dc76c3"
May 26 16:36:11.157993 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:36:11.157973    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b45c9fadebbf221f2b8f10995e496a3ac82847fe7e64734212674e179dc76c3\": container with ID starting with 4b45c9fadebbf221f2b8f10995e496a3ac82847fe7e64734212674e179dc76c3 not found: ID does not exist" containerID="4b45c9fadebbf221f2b8f10995e496a3ac82847fe7e64734212674e179dc76c3"
May 26 16:36:11.158057 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:11.158002    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b45c9fadebbf221f2b8f10995e496a3ac82847fe7e64734212674e179dc76c3"} err="failed to get container status \"4b45c9fadebbf221f2b8f10995e496a3ac82847fe7e64734212674e179dc76c3\": rpc error: code = NotFound desc = could not find container \"4b45c9fadebbf221f2b8f10995e496a3ac82847fe7e64734212674e179dc76c3\": container with ID starting with 4b45c9fadebbf221f2b8f10995e496a3ac82847fe7e64734212674e179dc76c3 not found: ID does not exist"
May 26 16:36:11.158057 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:11.158018    2572 scope.go:117] "RemoveContainer" containerID="6a7774064bde0cc41bf4504c952229a1d2954f8226352aaf461d77fc7c5fbfe0"
May 26 16:36:11.158244 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:36:11.158228    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a7774064bde0cc41bf4504c952229a1d2954f8226352aaf461d77fc7c5fbfe0\": container with ID starting with 6a7774064bde0cc41bf4504c952229a1d2954f8226352aaf461d77fc7c5fbfe0 not found: ID does not exist" containerID="6a7774064bde0cc41bf4504c952229a1d2954f8226352aaf461d77fc7c5fbfe0"
May 26 16:36:11.158287 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:11.158250    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a7774064bde0cc41bf4504c952229a1d2954f8226352aaf461d77fc7c5fbfe0"} err="failed to get container status \"6a7774064bde0cc41bf4504c952229a1d2954f8226352aaf461d77fc7c5fbfe0\": rpc error: code = NotFound desc = could not find container \"6a7774064bde0cc41bf4504c952229a1d2954f8226352aaf461d77fc7c5fbfe0\": container with ID starting with 6a7774064bde0cc41bf4504c952229a1d2954f8226352aaf461d77fc7c5fbfe0 not found: ID does not exist"
May 26 16:36:11.240762 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:11.240679    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7d27\" (UniqueName: \"kubernetes.io/projected/409f8af5-196a-4d5f-92da-7bd629992251-kube-api-access-b7d27\") pod \"409f8af5-196a-4d5f-92da-7bd629992251\" (UID: \"409f8af5-196a-4d5f-92da-7bd629992251\") "
May 26 16:36:11.240762 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:11.240719    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/409f8af5-196a-4d5f-92da-7bd629992251-proxy-tls\") pod \"409f8af5-196a-4d5f-92da-7bd629992251\" (UID: \"409f8af5-196a-4d5f-92da-7bd629992251\") "
May 26 16:36:11.240762 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:11.240757    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/409f8af5-196a-4d5f-92da-7bd629992251-kserve-provision-location\") pod \"409f8af5-196a-4d5f-92da-7bd629992251\" (UID: \"409f8af5-196a-4d5f-92da-7bd629992251\") "
May 26 16:36:11.241031 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:11.240785    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-init-fail-919f7e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/409f8af5-196a-4d5f-92da-7bd629992251-isvc-init-fail-919f7e-kube-rbac-proxy-sar-config\") pod \"409f8af5-196a-4d5f-92da-7bd629992251\" (UID: \"409f8af5-196a-4d5f-92da-7bd629992251\") "
May 26 16:36:11.241031 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:11.240845    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/409f8af5-196a-4d5f-92da-7bd629992251-cabundle-cert\") pod \"409f8af5-196a-4d5f-92da-7bd629992251\" (UID: \"409f8af5-196a-4d5f-92da-7bd629992251\") "
May 26 16:36:11.241384 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:11.241319    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/409f8af5-196a-4d5f-92da-7bd629992251-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "409f8af5-196a-4d5f-92da-7bd629992251" (UID: "409f8af5-196a-4d5f-92da-7bd629992251"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:36:11.241523 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:11.241451    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/409f8af5-196a-4d5f-92da-7bd629992251-isvc-init-fail-919f7e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-init-fail-919f7e-kube-rbac-proxy-sar-config") pod "409f8af5-196a-4d5f-92da-7bd629992251" (UID: "409f8af5-196a-4d5f-92da-7bd629992251"). InnerVolumeSpecName "isvc-init-fail-919f7e-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:36:11.241710 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:11.241680    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/409f8af5-196a-4d5f-92da-7bd629992251-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "409f8af5-196a-4d5f-92da-7bd629992251" (UID: "409f8af5-196a-4d5f-92da-7bd629992251"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:36:11.243194 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:11.243032    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/409f8af5-196a-4d5f-92da-7bd629992251-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "409f8af5-196a-4d5f-92da-7bd629992251" (UID: "409f8af5-196a-4d5f-92da-7bd629992251"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:36:11.243368 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:11.243309    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/409f8af5-196a-4d5f-92da-7bd629992251-kube-api-access-b7d27" (OuterVolumeSpecName: "kube-api-access-b7d27") pod "409f8af5-196a-4d5f-92da-7bd629992251" (UID: "409f8af5-196a-4d5f-92da-7bd629992251"). InnerVolumeSpecName "kube-api-access-b7d27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:36:11.342066 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:11.342025    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7d27\" (UniqueName: \"kubernetes.io/projected/409f8af5-196a-4d5f-92da-7bd629992251-kube-api-access-b7d27\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:36:11.342066 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:11.342056    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/409f8af5-196a-4d5f-92da-7bd629992251-proxy-tls\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:36:11.342066 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:11.342067    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/409f8af5-196a-4d5f-92da-7bd629992251-kserve-provision-location\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:36:11.342312 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:11.342081    2572 reconciler_common.go:299] "Volume detached for volume \"isvc-init-fail-919f7e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/409f8af5-196a-4d5f-92da-7bd629992251-isvc-init-fail-919f7e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:36:11.342312 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:11.342095    2572 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/409f8af5-196a-4d5f-92da-7bd629992251-cabundle-cert\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:36:11.483302 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:11.483267    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4"]
May 26 16:36:11.485861 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:11.485822    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-919f7e-predictor-69bf6757bc-7pqp4"]
May 26 16:36:12.079200 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:12.079166    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="409f8af5-196a-4d5f-92da-7bd629992251" path="/var/lib/kubelet/pods/409f8af5-196a-4d5f-92da-7bd629992251/volumes"
May 26 16:36:15.157197 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:15.157153    2572 generic.go:358] "Generic (PLEG): container finished" podID="54b401eb-76b3-4ca5-b4c3-de9c9d6a2081" containerID="b9d41b9eafbf3ae6e338e05b209698a894bd5941bf0a70ebe25da5d437fd4fb4" exitCode=0
May 26 16:36:15.157822 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:15.157233    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-kc74j" event={"ID":"54b401eb-76b3-4ca5-b4c3-de9c9d6a2081","Type":"ContainerDied","Data":"b9d41b9eafbf3ae6e338e05b209698a894bd5941bf0a70ebe25da5d437fd4fb4"}
May 26 16:36:39.238643 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:39.238592    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-kc74j" event={"ID":"54b401eb-76b3-4ca5-b4c3-de9c9d6a2081","Type":"ContainerStarted","Data":"418df0f7b5044ded5505fe8ce7f209b42d629eab38f8422ef2d9d4c6d1fc2bed"}
May 26 16:36:39.239143 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:39.238652    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-kc74j" event={"ID":"54b401eb-76b3-4ca5-b4c3-de9c9d6a2081","Type":"ContainerStarted","Data":"87fed2636b0bf204d961947f22ace934e3e602748d5ec4679a5d6036374658e6"}
May 26 16:36:39.239143 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:39.238890    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-kc74j"
May 26 16:36:39.257114 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:39.257061    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-kc74j" podStartSLOduration=7.016353923 podStartE2EDuration="30.257045281s" podCreationTimestamp="2026-05-26 16:36:09 +0000 UTC" firstStartedPulling="2026-05-26 16:36:15.158529234 +0000 UTC m=+2091.634256562" lastFinishedPulling="2026-05-26 16:36:38.399220592 +0000 UTC m=+2114.874947920" observedRunningTime="2026-05-26 16:36:39.255595694 +0000 UTC m=+2115.731323044" watchObservedRunningTime="2026-05-26 16:36:39.257045281 +0000 UTC m=+2115.732772630"
May 26 16:36:40.241438 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:40.241403    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-kc74j"
May 26 16:36:40.242762 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:40.242731    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-kc74j" podUID="54b401eb-76b3-4ca5-b4c3-de9c9d6a2081" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
May 26 16:36:41.243776 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:41.243736    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-kc74j" podUID="54b401eb-76b3-4ca5-b4c3-de9c9d6a2081" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
May 26 16:36:46.248442 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:46.248361    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-kc74j"
May 26 16:36:46.249000 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:46.248965    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-kc74j" podUID="54b401eb-76b3-4ca5-b4c3-de9c9d6a2081" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
May 26 16:36:56.249495 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:36:56.249452    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-kc74j" podUID="54b401eb-76b3-4ca5-b4c3-de9c9d6a2081" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
May 26 16:37:06.248951 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:37:06.248910    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-kc74j" podUID="54b401eb-76b3-4ca5-b4c3-de9c9d6a2081" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
May 26 16:37:16.248886 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:37:16.248797    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-kc74j" podUID="54b401eb-76b3-4ca5-b4c3-de9c9d6a2081" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
May 26 16:37:26.249836 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:37:26.249796    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-kc74j" podUID="54b401eb-76b3-4ca5-b4c3-de9c9d6a2081" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
May 26 16:37:36.249124 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:37:36.249082    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-kc74j" podUID="54b401eb-76b3-4ca5-b4c3-de9c9d6a2081" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
May 26 16:37:46.249692 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:37:46.249655    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-kc74j" podUID="54b401eb-76b3-4ca5-b4c3-de9c9d6a2081" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
May 26 16:37:48.075751 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:37:48.075704    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-kc74j" podUID="54b401eb-76b3-4ca5-b4c3-de9c9d6a2081" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
May 26 16:37:58.079136 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:37:58.079107    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-kc74j"
May 26 16:38:00.130612 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:00.130557    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-kc74j"]
May 26 16:38:00.131068 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:00.130992    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-kc74j" podUID="54b401eb-76b3-4ca5-b4c3-de9c9d6a2081" containerName="kserve-container" containerID="cri-o://87fed2636b0bf204d961947f22ace934e3e602748d5ec4679a5d6036374658e6" gracePeriod=30
May 26 16:38:00.131137 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:00.131105    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-kc74j" podUID="54b401eb-76b3-4ca5-b4c3-de9c9d6a2081" containerName="kube-rbac-proxy" containerID="cri-o://418df0f7b5044ded5505fe8ce7f209b42d629eab38f8422ef2d9d4c6d1fc2bed" gracePeriod=30
May 26 16:38:00.245191 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:00.245157    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-868qv"]
May 26 16:38:00.245452 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:00.245439    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="409f8af5-196a-4d5f-92da-7bd629992251" containerName="storage-initializer"
May 26 16:38:00.245497 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:00.245453    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="409f8af5-196a-4d5f-92da-7bd629992251" containerName="storage-initializer"
May 26 16:38:00.245533 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:00.245504    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="409f8af5-196a-4d5f-92da-7bd629992251" containerName="storage-initializer"
May 26 16:38:00.245533 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:00.245512    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="409f8af5-196a-4d5f-92da-7bd629992251" containerName="storage-initializer"
May 26 16:38:00.245617 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:00.245560    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="409f8af5-196a-4d5f-92da-7bd629992251" containerName="storage-initializer"
May 26 16:38:00.245617 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:00.245567    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="409f8af5-196a-4d5f-92da-7bd629992251" containerName="storage-initializer"
May 26 16:38:00.247690 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:00.247672    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-868qv"
May 26 16:38:00.249637 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:00.249613    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
May 26 16:38:00.249755 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:00.249618    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
May 26 16:38:00.258235 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:00.258208    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-868qv"]
May 26 16:38:00.363567 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:00.363530    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69c4cf3e-68fc-459c-b994-af1ddfb54dae-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-868qv\" (UID: \"69c4cf3e-68fc-459c-b994-af1ddfb54dae\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-868qv"
May 26 16:38:00.363778 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:00.363588    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mww2j\" (UniqueName: \"kubernetes.io/projected/69c4cf3e-68fc-459c-b994-af1ddfb54dae-kube-api-access-mww2j\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-868qv\" (UID: \"69c4cf3e-68fc-459c-b994-af1ddfb54dae\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-868qv"
May 26 16:38:00.363778 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:00.363635    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69c4cf3e-68fc-459c-b994-af1ddfb54dae-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-868qv\" (UID: \"69c4cf3e-68fc-459c-b994-af1ddfb54dae\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-868qv"
May 26 16:38:00.363778 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:00.363679    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69c4cf3e-68fc-459c-b994-af1ddfb54dae-isvc-predictive-xgboost-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-868qv\" (UID: \"69c4cf3e-68fc-459c-b994-af1ddfb54dae\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-868qv"
May 26 16:38:00.464161 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:00.464057    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mww2j\" (UniqueName: \"kubernetes.io/projected/69c4cf3e-68fc-459c-b994-af1ddfb54dae-kube-api-access-mww2j\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-868qv\" (UID: \"69c4cf3e-68fc-459c-b994-af1ddfb54dae\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-868qv"
May 26 16:38:00.464161 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:00.464108    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69c4cf3e-68fc-459c-b994-af1ddfb54dae-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-868qv\" (UID: \"69c4cf3e-68fc-459c-b994-af1ddfb54dae\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-868qv"
May 26 16:38:00.464161 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:00.464141    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69c4cf3e-68fc-459c-b994-af1ddfb54dae-isvc-predictive-xgboost-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-868qv\" (UID: \"69c4cf3e-68fc-459c-b994-af1ddfb54dae\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-868qv"
May 26 16:38:00.464448 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:00.464221    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69c4cf3e-68fc-459c-b994-af1ddfb54dae-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-868qv\" (UID: \"69c4cf3e-68fc-459c-b994-af1ddfb54dae\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-868qv"
May 26 16:38:00.464448 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:38:00.464279    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-serving-cert: secret "isvc-predictive-xgboost-predictor-serving-cert" not found
May 26 16:38:00.464448 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:38:00.464356    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69c4cf3e-68fc-459c-b994-af1ddfb54dae-proxy-tls podName:69c4cf3e-68fc-459c-b994-af1ddfb54dae nodeName:}" failed. No retries permitted until 2026-05-26 16:38:00.964336626 +0000 UTC m=+2197.440063953 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/69c4cf3e-68fc-459c-b994-af1ddfb54dae-proxy-tls") pod "isvc-predictive-xgboost-predictor-7ff98fd74d-868qv" (UID: "69c4cf3e-68fc-459c-b994-af1ddfb54dae") : secret "isvc-predictive-xgboost-predictor-serving-cert" not found
May 26 16:38:00.464717 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:00.464696    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69c4cf3e-68fc-459c-b994-af1ddfb54dae-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-868qv\" (UID: \"69c4cf3e-68fc-459c-b994-af1ddfb54dae\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-868qv"
May 26 16:38:00.464950 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:00.464933    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69c4cf3e-68fc-459c-b994-af1ddfb54dae-isvc-predictive-xgboost-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-868qv\" (UID: \"69c4cf3e-68fc-459c-b994-af1ddfb54dae\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-868qv"
May 26 16:38:00.467636 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:00.467596    2572 generic.go:358] "Generic (PLEG): container finished" podID="54b401eb-76b3-4ca5-b4c3-de9c9d6a2081" containerID="418df0f7b5044ded5505fe8ce7f209b42d629eab38f8422ef2d9d4c6d1fc2bed" exitCode=2
May 26 16:38:00.467636 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:00.467628    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-kc74j" event={"ID":"54b401eb-76b3-4ca5-b4c3-de9c9d6a2081","Type":"ContainerDied","Data":"418df0f7b5044ded5505fe8ce7f209b42d629eab38f8422ef2d9d4c6d1fc2bed"}
May 26 16:38:00.472831 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:00.472811    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mww2j\" (UniqueName: \"kubernetes.io/projected/69c4cf3e-68fc-459c-b994-af1ddfb54dae-kube-api-access-mww2j\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-868qv\" (UID: \"69c4cf3e-68fc-459c-b994-af1ddfb54dae\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-868qv"
May 26 16:38:00.967180 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:00.967147    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69c4cf3e-68fc-459c-b994-af1ddfb54dae-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-868qv\" (UID: \"69c4cf3e-68fc-459c-b994-af1ddfb54dae\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-868qv"
May 26 16:38:00.967411 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:38:00.967321    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-serving-cert: secret "isvc-predictive-xgboost-predictor-serving-cert" not found
May 26 16:38:00.967411 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:38:00.967399    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69c4cf3e-68fc-459c-b994-af1ddfb54dae-proxy-tls podName:69c4cf3e-68fc-459c-b994-af1ddfb54dae nodeName:}" failed. No retries permitted until 2026-05-26 16:38:01.967378743 +0000 UTC m=+2198.443106082 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/69c4cf3e-68fc-459c-b994-af1ddfb54dae-proxy-tls") pod "isvc-predictive-xgboost-predictor-7ff98fd74d-868qv" (UID: "69c4cf3e-68fc-459c-b994-af1ddfb54dae") : secret "isvc-predictive-xgboost-predictor-serving-cert" not found
May 26 16:38:01.244450 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:01.244349    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-kc74j" podUID="54b401eb-76b3-4ca5-b4c3-de9c9d6a2081" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused"
May 26 16:38:01.974429 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:01.974384    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69c4cf3e-68fc-459c-b994-af1ddfb54dae-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-868qv\" (UID: \"69c4cf3e-68fc-459c-b994-af1ddfb54dae\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-868qv"
May 26 16:38:01.976855 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:01.976832    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69c4cf3e-68fc-459c-b994-af1ddfb54dae-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-7ff98fd74d-868qv\" (UID: \"69c4cf3e-68fc-459c-b994-af1ddfb54dae\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-868qv"
May 26 16:38:02.058799 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:02.058752    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-868qv"
May 26 16:38:02.190703 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:02.190639    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-868qv"]
May 26 16:38:02.193516 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:38:02.193487    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69c4cf3e_68fc_459c_b994_af1ddfb54dae.slice/crio-e4f255b48ca76bdca26f6c814fd91de5f90d2f1fbce90fdcff3e1ad9d1b5f40d WatchSource:0}: Error finding container e4f255b48ca76bdca26f6c814fd91de5f90d2f1fbce90fdcff3e1ad9d1b5f40d: Status 404 returned error can't find the container with id e4f255b48ca76bdca26f6c814fd91de5f90d2f1fbce90fdcff3e1ad9d1b5f40d
May 26 16:38:02.475141 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:02.475102    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-868qv" event={"ID":"69c4cf3e-68fc-459c-b994-af1ddfb54dae","Type":"ContainerStarted","Data":"06f2c7d5668c9c7823f06a46253914ae37de8219287df5a346b6a6c347897413"}
May 26 16:38:02.475141 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:02.475143    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-868qv" event={"ID":"69c4cf3e-68fc-459c-b994-af1ddfb54dae","Type":"ContainerStarted","Data":"e4f255b48ca76bdca26f6c814fd91de5f90d2f1fbce90fdcff3e1ad9d1b5f40d"}
May 26 16:38:05.178543 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:05.178518    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-kc74j"
May 26 16:38:05.297092 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:05.296998    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/54b401eb-76b3-4ca5-b4c3-de9c9d6a2081-isvc-predictive-sklearn-kube-rbac-proxy-sar-config\") pod \"54b401eb-76b3-4ca5-b4c3-de9c9d6a2081\" (UID: \"54b401eb-76b3-4ca5-b4c3-de9c9d6a2081\") "
May 26 16:38:05.297092 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:05.297048    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54b401eb-76b3-4ca5-b4c3-de9c9d6a2081-proxy-tls\") pod \"54b401eb-76b3-4ca5-b4c3-de9c9d6a2081\" (UID: \"54b401eb-76b3-4ca5-b4c3-de9c9d6a2081\") "
May 26 16:38:05.297092 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:05.297071    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6294l\" (UniqueName: \"kubernetes.io/projected/54b401eb-76b3-4ca5-b4c3-de9c9d6a2081-kube-api-access-6294l\") pod \"54b401eb-76b3-4ca5-b4c3-de9c9d6a2081\" (UID: \"54b401eb-76b3-4ca5-b4c3-de9c9d6a2081\") "
May 26 16:38:05.297368 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:05.297095    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54b401eb-76b3-4ca5-b4c3-de9c9d6a2081-kserve-provision-location\") pod \"54b401eb-76b3-4ca5-b4c3-de9c9d6a2081\" (UID: \"54b401eb-76b3-4ca5-b4c3-de9c9d6a2081\") "
May 26 16:38:05.297429 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:05.297389    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54b401eb-76b3-4ca5-b4c3-de9c9d6a2081-isvc-predictive-sklearn-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-predictive-sklearn-kube-rbac-proxy-sar-config") pod "54b401eb-76b3-4ca5-b4c3-de9c9d6a2081" (UID: "54b401eb-76b3-4ca5-b4c3-de9c9d6a2081"). InnerVolumeSpecName "isvc-predictive-sklearn-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:38:05.297571 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:05.297546    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54b401eb-76b3-4ca5-b4c3-de9c9d6a2081-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "54b401eb-76b3-4ca5-b4c3-de9c9d6a2081" (UID: "54b401eb-76b3-4ca5-b4c3-de9c9d6a2081"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:38:05.299208 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:05.299185    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54b401eb-76b3-4ca5-b4c3-de9c9d6a2081-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "54b401eb-76b3-4ca5-b4c3-de9c9d6a2081" (UID: "54b401eb-76b3-4ca5-b4c3-de9c9d6a2081"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:38:05.299271 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:05.299221    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54b401eb-76b3-4ca5-b4c3-de9c9d6a2081-kube-api-access-6294l" (OuterVolumeSpecName: "kube-api-access-6294l") pod "54b401eb-76b3-4ca5-b4c3-de9c9d6a2081" (UID: "54b401eb-76b3-4ca5-b4c3-de9c9d6a2081"). InnerVolumeSpecName "kube-api-access-6294l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:38:05.398033 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:05.397993    2572 reconciler_common.go:299] "Volume detached for volume \"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/54b401eb-76b3-4ca5-b4c3-de9c9d6a2081-isvc-predictive-sklearn-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:38:05.398033 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:05.398026    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54b401eb-76b3-4ca5-b4c3-de9c9d6a2081-proxy-tls\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:38:05.398033 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:05.398041    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6294l\" (UniqueName: \"kubernetes.io/projected/54b401eb-76b3-4ca5-b4c3-de9c9d6a2081-kube-api-access-6294l\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:38:05.398270 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:05.398051    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54b401eb-76b3-4ca5-b4c3-de9c9d6a2081-kserve-provision-location\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:38:05.485174 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:05.485135    2572 generic.go:358] "Generic (PLEG): container finished" podID="54b401eb-76b3-4ca5-b4c3-de9c9d6a2081" containerID="87fed2636b0bf204d961947f22ace934e3e602748d5ec4679a5d6036374658e6" exitCode=0
May 26 16:38:05.485361 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:05.485217    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-kc74j"
May 26 16:38:05.485361 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:05.485212    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-kc74j" event={"ID":"54b401eb-76b3-4ca5-b4c3-de9c9d6a2081","Type":"ContainerDied","Data":"87fed2636b0bf204d961947f22ace934e3e602748d5ec4679a5d6036374658e6"}
May 26 16:38:05.485361 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:05.485323    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-kc74j" event={"ID":"54b401eb-76b3-4ca5-b4c3-de9c9d6a2081","Type":"ContainerDied","Data":"292f7b14c8b201d66733f916b5b4dd3d3a4506f938bee61fbae0d044d3883ed0"}
May 26 16:38:05.485361 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:05.485338    2572 scope.go:117] "RemoveContainer" containerID="418df0f7b5044ded5505fe8ce7f209b42d629eab38f8422ef2d9d4c6d1fc2bed"
May 26 16:38:05.493776 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:05.493755    2572 scope.go:117] "RemoveContainer" containerID="87fed2636b0bf204d961947f22ace934e3e602748d5ec4679a5d6036374658e6"
May 26 16:38:05.500794 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:05.500773    2572 scope.go:117] "RemoveContainer" containerID="b9d41b9eafbf3ae6e338e05b209698a894bd5941bf0a70ebe25da5d437fd4fb4"
May 26 16:38:05.507078 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:05.507033    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-kc74j"]
May 26 16:38:05.508307 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:05.508279    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-cd7c759c9-kc74j"]
May 26 16:38:05.509162 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:05.509148    2572 scope.go:117] "RemoveContainer" containerID="418df0f7b5044ded5505fe8ce7f209b42d629eab38f8422ef2d9d4c6d1fc2bed"
May 26 16:38:05.509457 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:38:05.509436    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"418df0f7b5044ded5505fe8ce7f209b42d629eab38f8422ef2d9d4c6d1fc2bed\": container with ID starting with 418df0f7b5044ded5505fe8ce7f209b42d629eab38f8422ef2d9d4c6d1fc2bed not found: ID does not exist" containerID="418df0f7b5044ded5505fe8ce7f209b42d629eab38f8422ef2d9d4c6d1fc2bed"
May 26 16:38:05.509547 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:05.509464    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"418df0f7b5044ded5505fe8ce7f209b42d629eab38f8422ef2d9d4c6d1fc2bed"} err="failed to get container status \"418df0f7b5044ded5505fe8ce7f209b42d629eab38f8422ef2d9d4c6d1fc2bed\": rpc error: code = NotFound desc = could not find container \"418df0f7b5044ded5505fe8ce7f209b42d629eab38f8422ef2d9d4c6d1fc2bed\": container with ID starting with 418df0f7b5044ded5505fe8ce7f209b42d629eab38f8422ef2d9d4c6d1fc2bed not found: ID does not exist"
May 26 16:38:05.509547 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:05.509484    2572 scope.go:117] "RemoveContainer" containerID="87fed2636b0bf204d961947f22ace934e3e602748d5ec4679a5d6036374658e6"
May 26 16:38:05.509757 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:38:05.509739    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87fed2636b0bf204d961947f22ace934e3e602748d5ec4679a5d6036374658e6\": container with ID starting with 87fed2636b0bf204d961947f22ace934e3e602748d5ec4679a5d6036374658e6 not found: ID does not exist" containerID="87fed2636b0bf204d961947f22ace934e3e602748d5ec4679a5d6036374658e6"
May 26 16:38:05.509804 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:05.509763    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87fed2636b0bf204d961947f22ace934e3e602748d5ec4679a5d6036374658e6"} err="failed to get container status \"87fed2636b0bf204d961947f22ace934e3e602748d5ec4679a5d6036374658e6\": rpc error: code = NotFound desc = could not find container \"87fed2636b0bf204d961947f22ace934e3e602748d5ec4679a5d6036374658e6\": container with ID starting with 87fed2636b0bf204d961947f22ace934e3e602748d5ec4679a5d6036374658e6 not found: ID does not exist"
May 26 16:38:05.509804 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:05.509782    2572 scope.go:117] "RemoveContainer" containerID="b9d41b9eafbf3ae6e338e05b209698a894bd5941bf0a70ebe25da5d437fd4fb4"
May 26 16:38:05.510004 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:38:05.509989    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9d41b9eafbf3ae6e338e05b209698a894bd5941bf0a70ebe25da5d437fd4fb4\": container with ID starting with b9d41b9eafbf3ae6e338e05b209698a894bd5941bf0a70ebe25da5d437fd4fb4 not found: ID does not exist" containerID="b9d41b9eafbf3ae6e338e05b209698a894bd5941bf0a70ebe25da5d437fd4fb4"
May 26 16:38:05.510050 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:05.510007    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9d41b9eafbf3ae6e338e05b209698a894bd5941bf0a70ebe25da5d437fd4fb4"} err="failed to get container status \"b9d41b9eafbf3ae6e338e05b209698a894bd5941bf0a70ebe25da5d437fd4fb4\": rpc error: code = NotFound desc = could not find container \"b9d41b9eafbf3ae6e338e05b209698a894bd5941bf0a70ebe25da5d437fd4fb4\": container with ID starting with b9d41b9eafbf3ae6e338e05b209698a894bd5941bf0a70ebe25da5d437fd4fb4 not found: ID does not exist"
May 26 16:38:06.078648 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:06.078588    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54b401eb-76b3-4ca5-b4c3-de9c9d6a2081" path="/var/lib/kubelet/pods/54b401eb-76b3-4ca5-b4c3-de9c9d6a2081/volumes"
May 26 16:38:06.489869 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:06.489833    2572 generic.go:358] "Generic (PLEG): container finished" podID="69c4cf3e-68fc-459c-b994-af1ddfb54dae" containerID="06f2c7d5668c9c7823f06a46253914ae37de8219287df5a346b6a6c347897413" exitCode=0
May 26 16:38:06.490232 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:06.489911    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-868qv" event={"ID":"69c4cf3e-68fc-459c-b994-af1ddfb54dae","Type":"ContainerDied","Data":"06f2c7d5668c9c7823f06a46253914ae37de8219287df5a346b6a6c347897413"}
May 26 16:38:07.494709 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:07.494675    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-868qv" event={"ID":"69c4cf3e-68fc-459c-b994-af1ddfb54dae","Type":"ContainerStarted","Data":"a22628369f65bdc7cc4b5c12910d84db33b6a67de3ec433d5bc2cf53fd0749d8"}
May 26 16:38:07.494709 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:07.494716    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-868qv" event={"ID":"69c4cf3e-68fc-459c-b994-af1ddfb54dae","Type":"ContainerStarted","Data":"2ea348b0210b90a52e9ea8352ac76bea3050b53989e7eb10af0ade77faf04191"}
May 26 16:38:07.495121 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:07.494917    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-868qv"
May 26 16:38:07.513580 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:07.513537    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-868qv" podStartSLOduration=7.513521147 podStartE2EDuration="7.513521147s" podCreationTimestamp="2026-05-26 16:38:00 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:38:07.512043707 +0000 UTC m=+2203.987771056" watchObservedRunningTime="2026-05-26 16:38:07.513521147 +0000 UTC m=+2203.989248523"
May 26 16:38:08.498299 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:08.498266    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-868qv"
May 26 16:38:08.499642 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:08.499588    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-868qv" podUID="69c4cf3e-68fc-459c-b994-af1ddfb54dae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
May 26 16:38:09.500849 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:09.500813    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-868qv" podUID="69c4cf3e-68fc-459c-b994-af1ddfb54dae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
May 26 16:38:14.504934 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:14.504904    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-868qv"
May 26 16:38:14.505534 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:14.505505    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-868qv" podUID="69c4cf3e-68fc-459c-b994-af1ddfb54dae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
May 26 16:38:24.505788 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:24.505749    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-868qv" podUID="69c4cf3e-68fc-459c-b994-af1ddfb54dae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
May 26 16:38:34.506172 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:34.506135    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-868qv" podUID="69c4cf3e-68fc-459c-b994-af1ddfb54dae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
May 26 16:38:44.506398 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:44.506310    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-868qv" podUID="69c4cf3e-68fc-459c-b994-af1ddfb54dae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
May 26 16:38:54.505772 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:38:54.505733    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-868qv" podUID="69c4cf3e-68fc-459c-b994-af1ddfb54dae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
May 26 16:39:04.506461 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:04.506419    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-868qv" podUID="69c4cf3e-68fc-459c-b994-af1ddfb54dae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
May 26 16:39:14.506183 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:14.506142    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-868qv" podUID="69c4cf3e-68fc-459c-b994-af1ddfb54dae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
May 26 16:39:22.079503 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:22.079474    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-868qv"
May 26 16:39:30.336805 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:30.336767    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-868qv"]
May 26 16:39:30.337281 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:30.337149    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-868qv" podUID="69c4cf3e-68fc-459c-b994-af1ddfb54dae" containerName="kserve-container" containerID="cri-o://2ea348b0210b90a52e9ea8352ac76bea3050b53989e7eb10af0ade77faf04191" gracePeriod=30
May 26 16:39:30.337357 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:30.337289    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-868qv" podUID="69c4cf3e-68fc-459c-b994-af1ddfb54dae" containerName="kube-rbac-proxy" containerID="cri-o://a22628369f65bdc7cc4b5c12910d84db33b6a67de3ec433d5bc2cf53fd0749d8" gracePeriod=30
May 26 16:39:30.430515 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:30.430481    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz"]
May 26 16:39:30.430768 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:30.430756    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54b401eb-76b3-4ca5-b4c3-de9c9d6a2081" containerName="kserve-container"
May 26 16:39:30.430821 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:30.430769    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="54b401eb-76b3-4ca5-b4c3-de9c9d6a2081" containerName="kserve-container"
May 26 16:39:30.430821 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:30.430791    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54b401eb-76b3-4ca5-b4c3-de9c9d6a2081" containerName="kube-rbac-proxy"
May 26 16:39:30.430821 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:30.430796    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="54b401eb-76b3-4ca5-b4c3-de9c9d6a2081" containerName="kube-rbac-proxy"
May 26 16:39:30.430821 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:30.430803    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54b401eb-76b3-4ca5-b4c3-de9c9d6a2081" containerName="storage-initializer"
May 26 16:39:30.430821 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:30.430810    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="54b401eb-76b3-4ca5-b4c3-de9c9d6a2081" containerName="storage-initializer"
May 26 16:39:30.430974 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:30.430854    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="54b401eb-76b3-4ca5-b4c3-de9c9d6a2081" containerName="kube-rbac-proxy"
May 26 16:39:30.430974 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:30.430864    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="54b401eb-76b3-4ca5-b4c3-de9c9d6a2081" containerName="kserve-container"
May 26 16:39:30.433941 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:30.433922    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz"
May 26 16:39:30.435937 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:30.435915    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
May 26 16:39:30.435937 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:30.435928    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
May 26 16:39:30.443872 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:30.443848    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz"]
May 26 16:39:30.513571 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:30.513529    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f84bcaa2-9e1d-4578-97e9-f50a4b651c46-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz\" (UID: \"f84bcaa2-9e1d-4578-97e9-f50a4b651c46\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz"
May 26 16:39:30.513571 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:30.513573    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f84bcaa2-9e1d-4578-97e9-f50a4b651c46-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz\" (UID: \"f84bcaa2-9e1d-4578-97e9-f50a4b651c46\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz"
May 26 16:39:30.513821 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:30.513670    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f84bcaa2-9e1d-4578-97e9-f50a4b651c46-isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz\" (UID: \"f84bcaa2-9e1d-4578-97e9-f50a4b651c46\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz"
May 26 16:39:30.513821 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:30.513732    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvjdf\" (UniqueName: \"kubernetes.io/projected/f84bcaa2-9e1d-4578-97e9-f50a4b651c46-kube-api-access-vvjdf\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz\" (UID: \"f84bcaa2-9e1d-4578-97e9-f50a4b651c46\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz"
May 26 16:39:30.614594 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:30.614501    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f84bcaa2-9e1d-4578-97e9-f50a4b651c46-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz\" (UID: \"f84bcaa2-9e1d-4578-97e9-f50a4b651c46\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz"
May 26 16:39:30.614594 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:30.614538    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f84bcaa2-9e1d-4578-97e9-f50a4b651c46-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz\" (UID: \"f84bcaa2-9e1d-4578-97e9-f50a4b651c46\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz"
May 26 16:39:30.614594 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:30.614585    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f84bcaa2-9e1d-4578-97e9-f50a4b651c46-isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz\" (UID: \"f84bcaa2-9e1d-4578-97e9-f50a4b651c46\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz"
May 26 16:39:30.614923 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:30.614645    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vvjdf\" (UniqueName: \"kubernetes.io/projected/f84bcaa2-9e1d-4578-97e9-f50a4b651c46-kube-api-access-vvjdf\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz\" (UID: \"f84bcaa2-9e1d-4578-97e9-f50a4b651c46\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz"
May 26 16:39:30.614990 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:30.614919    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f84bcaa2-9e1d-4578-97e9-f50a4b651c46-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz\" (UID: \"f84bcaa2-9e1d-4578-97e9-f50a4b651c46\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz"
May 26 16:39:30.615226 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:30.615205    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f84bcaa2-9e1d-4578-97e9-f50a4b651c46-isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz\" (UID: \"f84bcaa2-9e1d-4578-97e9-f50a4b651c46\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz"
May 26 16:39:30.617042 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:30.617020    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f84bcaa2-9e1d-4578-97e9-f50a4b651c46-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz\" (UID: \"f84bcaa2-9e1d-4578-97e9-f50a4b651c46\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz"
May 26 16:39:30.621983 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:30.621965    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvjdf\" (UniqueName: \"kubernetes.io/projected/f84bcaa2-9e1d-4578-97e9-f50a4b651c46-kube-api-access-vvjdf\") pod \"isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz\" (UID: \"f84bcaa2-9e1d-4578-97e9-f50a4b651c46\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz"
May 26 16:39:30.729001 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:30.728964    2572 generic.go:358] "Generic (PLEG): container finished" podID="69c4cf3e-68fc-459c-b994-af1ddfb54dae" containerID="a22628369f65bdc7cc4b5c12910d84db33b6a67de3ec433d5bc2cf53fd0749d8" exitCode=2
May 26 16:39:30.729167 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:30.729034    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-868qv" event={"ID":"69c4cf3e-68fc-459c-b994-af1ddfb54dae","Type":"ContainerDied","Data":"a22628369f65bdc7cc4b5c12910d84db33b6a67de3ec433d5bc2cf53fd0749d8"}
May 26 16:39:30.745235 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:30.745199    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz"
May 26 16:39:30.868246 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:30.868222    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz"]
May 26 16:39:30.870794 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:39:30.870762    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf84bcaa2_9e1d_4578_97e9_f50a4b651c46.slice/crio-12ea0015709b4736432e047515094f9d6dc027a9d20c5369e47e4e3edb799463 WatchSource:0}: Error finding container 12ea0015709b4736432e047515094f9d6dc027a9d20c5369e47e4e3edb799463: Status 404 returned error can't find the container with id 12ea0015709b4736432e047515094f9d6dc027a9d20c5369e47e4e3edb799463
May 26 16:39:30.872563 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:30.872546    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 16:39:31.732440 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:31.732403    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz" event={"ID":"f84bcaa2-9e1d-4578-97e9-f50a4b651c46","Type":"ContainerStarted","Data":"7e2a76fc8c1ff09628ba8e90ddf23795c92ae435261ef20312c287e287ef323c"}
May 26 16:39:31.732440 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:31.732440    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz" event={"ID":"f84bcaa2-9e1d-4578-97e9-f50a4b651c46","Type":"ContainerStarted","Data":"12ea0015709b4736432e047515094f9d6dc027a9d20c5369e47e4e3edb799463"}
May 26 16:39:32.079405 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:32.079324    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-868qv" podUID="69c4cf3e-68fc-459c-b994-af1ddfb54dae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
May 26 16:39:34.501974 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:34.501924    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-868qv" podUID="69c4cf3e-68fc-459c-b994-af1ddfb54dae" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.41:8643/healthz\": dial tcp 10.132.0.41:8643: connect: connection refused"
May 26 16:39:34.743116 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:34.743084    2572 generic.go:358] "Generic (PLEG): container finished" podID="f84bcaa2-9e1d-4578-97e9-f50a4b651c46" containerID="7e2a76fc8c1ff09628ba8e90ddf23795c92ae435261ef20312c287e287ef323c" exitCode=0
May 26 16:39:34.743281 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:34.743160    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz" event={"ID":"f84bcaa2-9e1d-4578-97e9-f50a4b651c46","Type":"ContainerDied","Data":"7e2a76fc8c1ff09628ba8e90ddf23795c92ae435261ef20312c287e287ef323c"}
May 26 16:39:35.270068 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:35.270043    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-868qv"
May 26 16:39:35.345250 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:35.345150    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69c4cf3e-68fc-459c-b994-af1ddfb54dae-proxy-tls\") pod \"69c4cf3e-68fc-459c-b994-af1ddfb54dae\" (UID: \"69c4cf3e-68fc-459c-b994-af1ddfb54dae\") "
May 26 16:39:35.345250 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:35.345208    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69c4cf3e-68fc-459c-b994-af1ddfb54dae-kserve-provision-location\") pod \"69c4cf3e-68fc-459c-b994-af1ddfb54dae\" (UID: \"69c4cf3e-68fc-459c-b994-af1ddfb54dae\") "
May 26 16:39:35.345250 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:35.345228    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mww2j\" (UniqueName: \"kubernetes.io/projected/69c4cf3e-68fc-459c-b994-af1ddfb54dae-kube-api-access-mww2j\") pod \"69c4cf3e-68fc-459c-b994-af1ddfb54dae\" (UID: \"69c4cf3e-68fc-459c-b994-af1ddfb54dae\") "
May 26 16:39:35.345598 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:35.345255    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69c4cf3e-68fc-459c-b994-af1ddfb54dae-isvc-predictive-xgboost-kube-rbac-proxy-sar-config\") pod \"69c4cf3e-68fc-459c-b994-af1ddfb54dae\" (UID: \"69c4cf3e-68fc-459c-b994-af1ddfb54dae\") "
May 26 16:39:35.345704 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:35.345629    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69c4cf3e-68fc-459c-b994-af1ddfb54dae-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "69c4cf3e-68fc-459c-b994-af1ddfb54dae" (UID: "69c4cf3e-68fc-459c-b994-af1ddfb54dae"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:39:35.345704 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:35.345643    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69c4cf3e-68fc-459c-b994-af1ddfb54dae-isvc-predictive-xgboost-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-predictive-xgboost-kube-rbac-proxy-sar-config") pod "69c4cf3e-68fc-459c-b994-af1ddfb54dae" (UID: "69c4cf3e-68fc-459c-b994-af1ddfb54dae"). InnerVolumeSpecName "isvc-predictive-xgboost-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:39:35.347295 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:35.347271    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69c4cf3e-68fc-459c-b994-af1ddfb54dae-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "69c4cf3e-68fc-459c-b994-af1ddfb54dae" (UID: "69c4cf3e-68fc-459c-b994-af1ddfb54dae"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:39:35.347406 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:35.347318    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69c4cf3e-68fc-459c-b994-af1ddfb54dae-kube-api-access-mww2j" (OuterVolumeSpecName: "kube-api-access-mww2j") pod "69c4cf3e-68fc-459c-b994-af1ddfb54dae" (UID: "69c4cf3e-68fc-459c-b994-af1ddfb54dae"). InnerVolumeSpecName "kube-api-access-mww2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:39:35.445710 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:35.445674    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69c4cf3e-68fc-459c-b994-af1ddfb54dae-proxy-tls\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:39:35.445710 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:35.445702    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69c4cf3e-68fc-459c-b994-af1ddfb54dae-kserve-provision-location\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:39:35.445710 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:35.445714    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mww2j\" (UniqueName: \"kubernetes.io/projected/69c4cf3e-68fc-459c-b994-af1ddfb54dae-kube-api-access-mww2j\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:39:35.445941 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:35.445724    2572 reconciler_common.go:299] "Volume detached for volume \"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69c4cf3e-68fc-459c-b994-af1ddfb54dae-isvc-predictive-xgboost-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:39:35.747324 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:35.747285    2572 generic.go:358] "Generic (PLEG): container finished" podID="69c4cf3e-68fc-459c-b994-af1ddfb54dae" containerID="2ea348b0210b90a52e9ea8352ac76bea3050b53989e7eb10af0ade77faf04191" exitCode=0
May 26 16:39:35.747751 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:35.747368    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-868qv" event={"ID":"69c4cf3e-68fc-459c-b994-af1ddfb54dae","Type":"ContainerDied","Data":"2ea348b0210b90a52e9ea8352ac76bea3050b53989e7eb10af0ade77faf04191"}
May 26 16:39:35.747751 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:35.747412    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-868qv" event={"ID":"69c4cf3e-68fc-459c-b994-af1ddfb54dae","Type":"ContainerDied","Data":"e4f255b48ca76bdca26f6c814fd91de5f90d2f1fbce90fdcff3e1ad9d1b5f40d"}
May 26 16:39:35.747751 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:35.747433    2572 scope.go:117] "RemoveContainer" containerID="a22628369f65bdc7cc4b5c12910d84db33b6a67de3ec433d5bc2cf53fd0749d8"
May 26 16:39:35.747751 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:35.747375    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-868qv"
May 26 16:39:35.749500 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:35.749479    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz" event={"ID":"f84bcaa2-9e1d-4578-97e9-f50a4b651c46","Type":"ContainerStarted","Data":"b4a7c0e7b7210f2a5138e5db486546b4b9e43d925d45cb27b3ac74dd2a237582"}
May 26 16:39:35.749648 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:35.749509    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz" event={"ID":"f84bcaa2-9e1d-4578-97e9-f50a4b651c46","Type":"ContainerStarted","Data":"3835071899aadeaeff235c5e6627c18c72b833b8623c32c0017da24afb149323"}
May 26 16:39:35.749872 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:35.749854    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz"
May 26 16:39:35.749948 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:35.749894    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz"
May 26 16:39:35.751593 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:35.751566    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz" podUID="f84bcaa2-9e1d-4578-97e9-f50a4b651c46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
May 26 16:39:35.755961 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:35.755937    2572 scope.go:117] "RemoveContainer" containerID="2ea348b0210b90a52e9ea8352ac76bea3050b53989e7eb10af0ade77faf04191"
May 26 16:39:35.763011 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:35.762991    2572 scope.go:117] "RemoveContainer" containerID="06f2c7d5668c9c7823f06a46253914ae37de8219287df5a346b6a6c347897413"
May 26 16:39:35.768538 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:35.768500    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz" podStartSLOduration=5.768486431 podStartE2EDuration="5.768486431s" podCreationTimestamp="2026-05-26 16:39:30 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:39:35.767342245 +0000 UTC m=+2292.243069594" watchObservedRunningTime="2026-05-26 16:39:35.768486431 +0000 UTC m=+2292.244213781"
May 26 16:39:35.770172 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:35.770155    2572 scope.go:117] "RemoveContainer" containerID="a22628369f65bdc7cc4b5c12910d84db33b6a67de3ec433d5bc2cf53fd0749d8"
May 26 16:39:35.770449 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:39:35.770431    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a22628369f65bdc7cc4b5c12910d84db33b6a67de3ec433d5bc2cf53fd0749d8\": container with ID starting with a22628369f65bdc7cc4b5c12910d84db33b6a67de3ec433d5bc2cf53fd0749d8 not found: ID does not exist" containerID="a22628369f65bdc7cc4b5c12910d84db33b6a67de3ec433d5bc2cf53fd0749d8"
May 26 16:39:35.770570 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:35.770457    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a22628369f65bdc7cc4b5c12910d84db33b6a67de3ec433d5bc2cf53fd0749d8"} err="failed to get container status \"a22628369f65bdc7cc4b5c12910d84db33b6a67de3ec433d5bc2cf53fd0749d8\": rpc error: code = NotFound desc = could not find container \"a22628369f65bdc7cc4b5c12910d84db33b6a67de3ec433d5bc2cf53fd0749d8\": container with ID starting with a22628369f65bdc7cc4b5c12910d84db33b6a67de3ec433d5bc2cf53fd0749d8 not found: ID does not exist"
May 26 16:39:35.770570 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:35.770474    2572 scope.go:117] "RemoveContainer" containerID="2ea348b0210b90a52e9ea8352ac76bea3050b53989e7eb10af0ade77faf04191"
May 26 16:39:35.770751 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:39:35.770733    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ea348b0210b90a52e9ea8352ac76bea3050b53989e7eb10af0ade77faf04191\": container with ID starting with 2ea348b0210b90a52e9ea8352ac76bea3050b53989e7eb10af0ade77faf04191 not found: ID does not exist" containerID="2ea348b0210b90a52e9ea8352ac76bea3050b53989e7eb10af0ade77faf04191"
May 26 16:39:35.770791 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:35.770756    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ea348b0210b90a52e9ea8352ac76bea3050b53989e7eb10af0ade77faf04191"} err="failed to get container status \"2ea348b0210b90a52e9ea8352ac76bea3050b53989e7eb10af0ade77faf04191\": rpc error: code = NotFound desc = could not find container \"2ea348b0210b90a52e9ea8352ac76bea3050b53989e7eb10af0ade77faf04191\": container with ID starting with 2ea348b0210b90a52e9ea8352ac76bea3050b53989e7eb10af0ade77faf04191 not found: ID does not exist"
May 26 16:39:35.770791 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:35.770771    2572 scope.go:117] "RemoveContainer" containerID="06f2c7d5668c9c7823f06a46253914ae37de8219287df5a346b6a6c347897413"
May 26 16:39:35.770989 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:39:35.770972    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06f2c7d5668c9c7823f06a46253914ae37de8219287df5a346b6a6c347897413\": container with ID starting with 06f2c7d5668c9c7823f06a46253914ae37de8219287df5a346b6a6c347897413 not found: ID does not exist" containerID="06f2c7d5668c9c7823f06a46253914ae37de8219287df5a346b6a6c347897413"
May 26 16:39:35.771034 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:35.770992    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06f2c7d5668c9c7823f06a46253914ae37de8219287df5a346b6a6c347897413"} err="failed to get container status \"06f2c7d5668c9c7823f06a46253914ae37de8219287df5a346b6a6c347897413\": rpc error: code = NotFound desc = could not find container \"06f2c7d5668c9c7823f06a46253914ae37de8219287df5a346b6a6c347897413\": container with ID starting with 06f2c7d5668c9c7823f06a46253914ae37de8219287df5a346b6a6c347897413 not found: ID does not exist"
May 26 16:39:35.781041 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:35.781018    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-868qv"]
May 26 16:39:35.785148 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:35.785127    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-7ff98fd74d-868qv"]
May 26 16:39:36.082791 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:36.082756    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69c4cf3e-68fc-459c-b994-af1ddfb54dae" path="/var/lib/kubelet/pods/69c4cf3e-68fc-459c-b994-af1ddfb54dae/volumes"
May 26 16:39:36.753809 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:36.753770    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz" podUID="f84bcaa2-9e1d-4578-97e9-f50a4b651c46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
May 26 16:39:41.758349 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:41.758319    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz"
May 26 16:39:41.758870 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:41.758842    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz" podUID="f84bcaa2-9e1d-4578-97e9-f50a4b651c46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
May 26 16:39:51.758871 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:39:51.758829    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz" podUID="f84bcaa2-9e1d-4578-97e9-f50a4b651c46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
May 26 16:40:01.759124 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:40:01.759085    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz" podUID="f84bcaa2-9e1d-4578-97e9-f50a4b651c46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
May 26 16:40:11.759790 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:40:11.759748    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz" podUID="f84bcaa2-9e1d-4578-97e9-f50a4b651c46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
May 26 16:40:21.759411 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:40:21.759329    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz" podUID="f84bcaa2-9e1d-4578-97e9-f50a4b651c46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
May 26 16:40:31.759334 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:40:31.759289    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz" podUID="f84bcaa2-9e1d-4578-97e9-f50a4b651c46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
May 26 16:40:41.759071 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:40:41.759032    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz" podUID="f84bcaa2-9e1d-4578-97e9-f50a4b651c46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
May 26 16:40:51.759669 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:40:51.759639    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz"
May 26 16:41:00.684912 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:00.684876    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz"]
May 26 16:41:00.685369 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:00.685345    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz" podUID="f84bcaa2-9e1d-4578-97e9-f50a4b651c46" containerName="kserve-container" containerID="cri-o://3835071899aadeaeff235c5e6627c18c72b833b8623c32c0017da24afb149323" gracePeriod=30
May 26 16:41:00.685438 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:00.685347    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz" podUID="f84bcaa2-9e1d-4578-97e9-f50a4b651c46" containerName="kube-rbac-proxy" containerID="cri-o://b4a7c0e7b7210f2a5138e5db486546b4b9e43d925d45cb27b3ac74dd2a237582" gracePeriod=30
May 26 16:41:00.797162 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:00.797124    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm"]
May 26 16:41:00.797545 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:00.797527    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69c4cf3e-68fc-459c-b994-af1ddfb54dae" containerName="storage-initializer"
May 26 16:41:00.797594 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:00.797550    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="69c4cf3e-68fc-459c-b994-af1ddfb54dae" containerName="storage-initializer"
May 26 16:41:00.797594 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:00.797582    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69c4cf3e-68fc-459c-b994-af1ddfb54dae" containerName="kserve-container"
May 26 16:41:00.797594 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:00.797591    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="69c4cf3e-68fc-459c-b994-af1ddfb54dae" containerName="kserve-container"
May 26 16:41:00.797719 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:00.797621    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69c4cf3e-68fc-459c-b994-af1ddfb54dae" containerName="kube-rbac-proxy"
May 26 16:41:00.797719 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:00.797631    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="69c4cf3e-68fc-459c-b994-af1ddfb54dae" containerName="kube-rbac-proxy"
May 26 16:41:00.797719 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:00.797706    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="69c4cf3e-68fc-459c-b994-af1ddfb54dae" containerName="kube-rbac-proxy"
May 26 16:41:00.797873 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:00.797719    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="69c4cf3e-68fc-459c-b994-af1ddfb54dae" containerName="kserve-container"
May 26 16:41:00.801150 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:00.801110    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm"
May 26 16:41:00.803108 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:00.803089    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\""
May 26 16:41:00.803222 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:00.803109    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\""
May 26 16:41:00.810453 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:00.810429    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm"]
May 26 16:41:00.895926 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:00.895893    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t64hl\" (UniqueName: \"kubernetes.io/projected/7314872d-3df0-4439-828d-8c26d132733a-kube-api-access-t64hl\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm\" (UID: \"7314872d-3df0-4439-828d-8c26d132733a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm"
May 26 16:41:00.896095 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:00.895967    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7314872d-3df0-4439-828d-8c26d132733a-isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm\" (UID: \"7314872d-3df0-4439-828d-8c26d132733a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm"
May 26 16:41:00.896095 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:00.896000    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7314872d-3df0-4439-828d-8c26d132733a-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm\" (UID: \"7314872d-3df0-4439-828d-8c26d132733a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm"
May 26 16:41:00.896095 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:00.896033    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7314872d-3df0-4439-828d-8c26d132733a-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm\" (UID: \"7314872d-3df0-4439-828d-8c26d132733a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm"
May 26 16:41:00.982885 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:00.982788    2572 generic.go:358] "Generic (PLEG): container finished" podID="f84bcaa2-9e1d-4578-97e9-f50a4b651c46" containerID="b4a7c0e7b7210f2a5138e5db486546b4b9e43d925d45cb27b3ac74dd2a237582" exitCode=2
May 26 16:41:00.983035 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:00.982871    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz" event={"ID":"f84bcaa2-9e1d-4578-97e9-f50a4b651c46","Type":"ContainerDied","Data":"b4a7c0e7b7210f2a5138e5db486546b4b9e43d925d45cb27b3ac74dd2a237582"}
May 26 16:41:00.997298 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:00.997262    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7314872d-3df0-4439-828d-8c26d132733a-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm\" (UID: \"7314872d-3df0-4439-828d-8c26d132733a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm"
May 26 16:41:00.997453 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:00.997324    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-t64hl\" (UniqueName: \"kubernetes.io/projected/7314872d-3df0-4439-828d-8c26d132733a-kube-api-access-t64hl\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm\" (UID: \"7314872d-3df0-4439-828d-8c26d132733a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm"
May 26 16:41:00.997453 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:00.997404    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7314872d-3df0-4439-828d-8c26d132733a-isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm\" (UID: \"7314872d-3df0-4439-828d-8c26d132733a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm"
May 26 16:41:00.997453 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:00.997447    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7314872d-3df0-4439-828d-8c26d132733a-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm\" (UID: \"7314872d-3df0-4439-828d-8c26d132733a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm"
May 26 16:41:00.997884 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:00.997849    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7314872d-3df0-4439-828d-8c26d132733a-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm\" (UID: \"7314872d-3df0-4439-828d-8c26d132733a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm"
May 26 16:41:00.998105 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:00.998088    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7314872d-3df0-4439-828d-8c26d132733a-isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm\" (UID: \"7314872d-3df0-4439-828d-8c26d132733a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm"
May 26 16:41:00.999770 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:00.999754    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7314872d-3df0-4439-828d-8c26d132733a-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm\" (UID: \"7314872d-3df0-4439-828d-8c26d132733a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm"
May 26 16:41:01.005525 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:01.005506    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t64hl\" (UniqueName: \"kubernetes.io/projected/7314872d-3df0-4439-828d-8c26d132733a-kube-api-access-t64hl\") pod \"isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm\" (UID: \"7314872d-3df0-4439-828d-8c26d132733a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm"
May 26 16:41:01.111767 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:01.111729    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm"
May 26 16:41:01.241848 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:01.241763    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm"]
May 26 16:41:01.245056 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:41:01.245016    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7314872d_3df0_4439_828d_8c26d132733a.slice/crio-fdb6cabf154ff0f0e3035bbe1791283dd2dd5e39f8df4aec19d9d3ef384902b2 WatchSource:0}: Error finding container fdb6cabf154ff0f0e3035bbe1791283dd2dd5e39f8df4aec19d9d3ef384902b2: Status 404 returned error can't find the container with id fdb6cabf154ff0f0e3035bbe1791283dd2dd5e39f8df4aec19d9d3ef384902b2
May 26 16:41:01.754699 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:01.754650    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz" podUID="f84bcaa2-9e1d-4578-97e9-f50a4b651c46" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
May 26 16:41:01.759079 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:01.759029    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz" podUID="f84bcaa2-9e1d-4578-97e9-f50a4b651c46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
May 26 16:41:01.987380 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:01.987346    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm" event={"ID":"7314872d-3df0-4439-828d-8c26d132733a","Type":"ContainerStarted","Data":"9a5251ef4b1507af6b492546cb4e29c26bb4f454aec2f1aa04a451e088eb6211"}
May 26 16:41:01.987380 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:01.987385    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm" event={"ID":"7314872d-3df0-4439-828d-8c26d132733a","Type":"ContainerStarted","Data":"fdb6cabf154ff0f0e3035bbe1791283dd2dd5e39f8df4aec19d9d3ef384902b2"}
May 26 16:41:05.915674 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:05.915647    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz"
May 26 16:41:05.934840 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:05.934810    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f84bcaa2-9e1d-4578-97e9-f50a4b651c46-proxy-tls\") pod \"f84bcaa2-9e1d-4578-97e9-f50a4b651c46\" (UID: \"f84bcaa2-9e1d-4578-97e9-f50a4b651c46\") "
May 26 16:41:05.935024 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:05.934921    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f84bcaa2-9e1d-4578-97e9-f50a4b651c46-kserve-provision-location\") pod \"f84bcaa2-9e1d-4578-97e9-f50a4b651c46\" (UID: \"f84bcaa2-9e1d-4578-97e9-f50a4b651c46\") "
May 26 16:41:05.935024 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:05.934944    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vvjdf\" (UniqueName: \"kubernetes.io/projected/f84bcaa2-9e1d-4578-97e9-f50a4b651c46-kube-api-access-vvjdf\") pod \"f84bcaa2-9e1d-4578-97e9-f50a4b651c46\" (UID: \"f84bcaa2-9e1d-4578-97e9-f50a4b651c46\") "
May 26 16:41:05.935024 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:05.934980    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f84bcaa2-9e1d-4578-97e9-f50a4b651c46-isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\") pod \"f84bcaa2-9e1d-4578-97e9-f50a4b651c46\" (UID: \"f84bcaa2-9e1d-4578-97e9-f50a4b651c46\") "
May 26 16:41:05.935971 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:05.935931    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f84bcaa2-9e1d-4578-97e9-f50a4b651c46-isvc-predictive-lightgbm-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-predictive-lightgbm-kube-rbac-proxy-sar-config") pod "f84bcaa2-9e1d-4578-97e9-f50a4b651c46" (UID: "f84bcaa2-9e1d-4578-97e9-f50a4b651c46"). InnerVolumeSpecName "isvc-predictive-lightgbm-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:41:05.935971 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:05.935931    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f84bcaa2-9e1d-4578-97e9-f50a4b651c46-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f84bcaa2-9e1d-4578-97e9-f50a4b651c46" (UID: "f84bcaa2-9e1d-4578-97e9-f50a4b651c46"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:41:05.938285 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:05.938257    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f84bcaa2-9e1d-4578-97e9-f50a4b651c46-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f84bcaa2-9e1d-4578-97e9-f50a4b651c46" (UID: "f84bcaa2-9e1d-4578-97e9-f50a4b651c46"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:41:05.938373 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:05.938261    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f84bcaa2-9e1d-4578-97e9-f50a4b651c46-kube-api-access-vvjdf" (OuterVolumeSpecName: "kube-api-access-vvjdf") pod "f84bcaa2-9e1d-4578-97e9-f50a4b651c46" (UID: "f84bcaa2-9e1d-4578-97e9-f50a4b651c46"). InnerVolumeSpecName "kube-api-access-vvjdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:41:06.000719 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:06.000688    2572 generic.go:358] "Generic (PLEG): container finished" podID="7314872d-3df0-4439-828d-8c26d132733a" containerID="9a5251ef4b1507af6b492546cb4e29c26bb4f454aec2f1aa04a451e088eb6211" exitCode=0
May 26 16:41:06.000902 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:06.000773    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm" event={"ID":"7314872d-3df0-4439-828d-8c26d132733a","Type":"ContainerDied","Data":"9a5251ef4b1507af6b492546cb4e29c26bb4f454aec2f1aa04a451e088eb6211"}
May 26 16:41:06.002479 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:06.002453    2572 generic.go:358] "Generic (PLEG): container finished" podID="f84bcaa2-9e1d-4578-97e9-f50a4b651c46" containerID="3835071899aadeaeff235c5e6627c18c72b833b8623c32c0017da24afb149323" exitCode=0
May 26 16:41:06.002584 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:06.002483    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz" event={"ID":"f84bcaa2-9e1d-4578-97e9-f50a4b651c46","Type":"ContainerDied","Data":"3835071899aadeaeff235c5e6627c18c72b833b8623c32c0017da24afb149323"}
May 26 16:41:06.002584 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:06.002511    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz" event={"ID":"f84bcaa2-9e1d-4578-97e9-f50a4b651c46","Type":"ContainerDied","Data":"12ea0015709b4736432e047515094f9d6dc027a9d20c5369e47e4e3edb799463"}
May 26 16:41:06.002584 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:06.002536    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz"
May 26 16:41:06.002584 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:06.002541    2572 scope.go:117] "RemoveContainer" containerID="b4a7c0e7b7210f2a5138e5db486546b4b9e43d925d45cb27b3ac74dd2a237582"
May 26 16:41:06.011340 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:06.011321    2572 scope.go:117] "RemoveContainer" containerID="3835071899aadeaeff235c5e6627c18c72b833b8623c32c0017da24afb149323"
May 26 16:41:06.019291 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:06.019268    2572 scope.go:117] "RemoveContainer" containerID="7e2a76fc8c1ff09628ba8e90ddf23795c92ae435261ef20312c287e287ef323c"
May 26 16:41:06.029379 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:06.029348    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz"]
May 26 16:41:06.032488 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:06.032459    2572 scope.go:117] "RemoveContainer" containerID="b4a7c0e7b7210f2a5138e5db486546b4b9e43d925d45cb27b3ac74dd2a237582"
May 26 16:41:06.032838 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:41:06.032816    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4a7c0e7b7210f2a5138e5db486546b4b9e43d925d45cb27b3ac74dd2a237582\": container with ID starting with b4a7c0e7b7210f2a5138e5db486546b4b9e43d925d45cb27b3ac74dd2a237582 not found: ID does not exist" containerID="b4a7c0e7b7210f2a5138e5db486546b4b9e43d925d45cb27b3ac74dd2a237582"
May 26 16:41:06.032940 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:06.032847    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4a7c0e7b7210f2a5138e5db486546b4b9e43d925d45cb27b3ac74dd2a237582"} err="failed to get container status \"b4a7c0e7b7210f2a5138e5db486546b4b9e43d925d45cb27b3ac74dd2a237582\": rpc error: code = NotFound desc = could not find container \"b4a7c0e7b7210f2a5138e5db486546b4b9e43d925d45cb27b3ac74dd2a237582\": container with ID starting with b4a7c0e7b7210f2a5138e5db486546b4b9e43d925d45cb27b3ac74dd2a237582 not found: ID does not exist"
May 26 16:41:06.032940 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:06.032869    2572 scope.go:117] "RemoveContainer" containerID="3835071899aadeaeff235c5e6627c18c72b833b8623c32c0017da24afb149323"
May 26 16:41:06.033155 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:41:06.033137    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3835071899aadeaeff235c5e6627c18c72b833b8623c32c0017da24afb149323\": container with ID starting with 3835071899aadeaeff235c5e6627c18c72b833b8623c32c0017da24afb149323 not found: ID does not exist" containerID="3835071899aadeaeff235c5e6627c18c72b833b8623c32c0017da24afb149323"
May 26 16:41:06.033202 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:06.033163    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3835071899aadeaeff235c5e6627c18c72b833b8623c32c0017da24afb149323"} err="failed to get container status \"3835071899aadeaeff235c5e6627c18c72b833b8623c32c0017da24afb149323\": rpc error: code = NotFound desc = could not find container \"3835071899aadeaeff235c5e6627c18c72b833b8623c32c0017da24afb149323\": container with ID starting with 3835071899aadeaeff235c5e6627c18c72b833b8623c32c0017da24afb149323 not found: ID does not exist"
May 26 16:41:06.033202 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:06.033185    2572 scope.go:117] "RemoveContainer" containerID="7e2a76fc8c1ff09628ba8e90ddf23795c92ae435261ef20312c287e287ef323c"
May 26 16:41:06.033291 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:06.033237    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-75cb94f9f-dtccz"]
May 26 16:41:06.033459 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:41:06.033442    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e2a76fc8c1ff09628ba8e90ddf23795c92ae435261ef20312c287e287ef323c\": container with ID starting with 7e2a76fc8c1ff09628ba8e90ddf23795c92ae435261ef20312c287e287ef323c not found: ID does not exist" containerID="7e2a76fc8c1ff09628ba8e90ddf23795c92ae435261ef20312c287e287ef323c"
May 26 16:41:06.033535 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:06.033463    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e2a76fc8c1ff09628ba8e90ddf23795c92ae435261ef20312c287e287ef323c"} err="failed to get container status \"7e2a76fc8c1ff09628ba8e90ddf23795c92ae435261ef20312c287e287ef323c\": rpc error: code = NotFound desc = could not find container \"7e2a76fc8c1ff09628ba8e90ddf23795c92ae435261ef20312c287e287ef323c\": container with ID starting with 7e2a76fc8c1ff09628ba8e90ddf23795c92ae435261ef20312c287e287ef323c not found: ID does not exist"
May 26 16:41:06.035906 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:06.035882    2572 reconciler_common.go:299] "Volume detached for volume \"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f84bcaa2-9e1d-4578-97e9-f50a4b651c46-isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:41:06.035987 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:06.035909    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f84bcaa2-9e1d-4578-97e9-f50a4b651c46-proxy-tls\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:41:06.035987 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:06.035925    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f84bcaa2-9e1d-4578-97e9-f50a4b651c46-kserve-provision-location\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:41:06.035987 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:06.035937    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvjdf\" (UniqueName: \"kubernetes.io/projected/f84bcaa2-9e1d-4578-97e9-f50a4b651c46-kube-api-access-vvjdf\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:41:06.079107 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:06.079079    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f84bcaa2-9e1d-4578-97e9-f50a4b651c46" path="/var/lib/kubelet/pods/f84bcaa2-9e1d-4578-97e9-f50a4b651c46/volumes"
May 26 16:41:07.008997 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:07.008955    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm" event={"ID":"7314872d-3df0-4439-828d-8c26d132733a","Type":"ContainerStarted","Data":"3bdd1ecbb734963dd2c57b6342dfdc3218b25d2c51c5cd3faac4ce50979b3979"}
May 26 16:41:07.009481 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:07.009007    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm" event={"ID":"7314872d-3df0-4439-828d-8c26d132733a","Type":"ContainerStarted","Data":"f1b253e1f3c1aae76ac8628cc79b363973174de4f8399c4b8a29d90bee9d8a1f"}
May 26 16:41:07.009481 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:07.009252    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm"
May 26 16:41:07.027037 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:07.026991    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm" podStartSLOduration=7.02697713 podStartE2EDuration="7.02697713s" podCreationTimestamp="2026-05-26 16:41:00 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:41:07.025723954 +0000 UTC m=+2383.501451306" watchObservedRunningTime="2026-05-26 16:41:07.02697713 +0000 UTC m=+2383.502704512"
May 26 16:41:08.012756 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:08.012728    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm"
May 26 16:41:14.020492 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:14.020463    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm"
May 26 16:41:44.021980 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:44.021902    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm" podUID="7314872d-3df0-4439-828d-8c26d132733a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.43:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.43:8080: connect: connection refused"
May 26 16:41:54.021368 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:41:54.021325    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm" podUID="7314872d-3df0-4439-828d-8c26d132733a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.43:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.43:8080: connect: connection refused"
May 26 16:42:04.021959 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:04.021917    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm" podUID="7314872d-3df0-4439-828d-8c26d132733a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.43:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.43:8080: connect: connection refused"
May 26 16:42:14.021443 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:14.021390    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm" podUID="7314872d-3df0-4439-828d-8c26d132733a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.43:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.43:8080: connect: connection refused"
May 26 16:42:24.024623 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:24.024570    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm"
May 26 16:42:30.901393 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:30.901358    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm"]
May 26 16:42:30.901812 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:30.901672    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm" podUID="7314872d-3df0-4439-828d-8c26d132733a" containerName="kserve-container" containerID="cri-o://f1b253e1f3c1aae76ac8628cc79b363973174de4f8399c4b8a29d90bee9d8a1f" gracePeriod=30
May 26 16:42:30.901812 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:30.901743    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm" podUID="7314872d-3df0-4439-828d-8c26d132733a" containerName="kube-rbac-proxy" containerID="cri-o://3bdd1ecbb734963dd2c57b6342dfdc3218b25d2c51c5cd3faac4ce50979b3979" gracePeriod=30
May 26 16:42:31.045681 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:31.045631    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n"]
May 26 16:42:31.045924 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:31.045910    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f84bcaa2-9e1d-4578-97e9-f50a4b651c46" containerName="kserve-container"
May 26 16:42:31.045924 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:31.045925    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f84bcaa2-9e1d-4578-97e9-f50a4b651c46" containerName="kserve-container"
May 26 16:42:31.046010 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:31.045938    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f84bcaa2-9e1d-4578-97e9-f50a4b651c46" containerName="kube-rbac-proxy"
May 26 16:42:31.046010 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:31.045944    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f84bcaa2-9e1d-4578-97e9-f50a4b651c46" containerName="kube-rbac-proxy"
May 26 16:42:31.046010 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:31.045955    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f84bcaa2-9e1d-4578-97e9-f50a4b651c46" containerName="storage-initializer"
May 26 16:42:31.046010 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:31.045960    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f84bcaa2-9e1d-4578-97e9-f50a4b651c46" containerName="storage-initializer"
May 26 16:42:31.046010 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:31.046008    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="f84bcaa2-9e1d-4578-97e9-f50a4b651c46" containerName="kserve-container"
May 26 16:42:31.046162 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:31.046016    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="f84bcaa2-9e1d-4578-97e9-f50a4b651c46" containerName="kube-rbac-proxy"
May 26 16:42:31.048989 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:31.048969    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n"
May 26 16:42:31.051244 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:31.051221    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
May 26 16:42:31.051350 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:31.051263    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
May 26 16:42:31.059160 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:31.059131    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n"]
May 26 16:42:31.075019 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:31.074960    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e86d2b84-af8e-43d9-97fa-0bc866345225-isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n\" (UID: \"e86d2b84-af8e-43d9-97fa-0bc866345225\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n"
May 26 16:42:31.075019 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:31.075014    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e86d2b84-af8e-43d9-97fa-0bc866345225-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n\" (UID: \"e86d2b84-af8e-43d9-97fa-0bc866345225\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n"
May 26 16:42:31.075303 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:31.075053    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e86d2b84-af8e-43d9-97fa-0bc866345225-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n\" (UID: \"e86d2b84-af8e-43d9-97fa-0bc866345225\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n"
May 26 16:42:31.075303 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:31.075079    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbvp4\" (UniqueName: \"kubernetes.io/projected/e86d2b84-af8e-43d9-97fa-0bc866345225-kube-api-access-zbvp4\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n\" (UID: \"e86d2b84-af8e-43d9-97fa-0bc866345225\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n"
May 26 16:42:31.176120 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:31.176020    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e86d2b84-af8e-43d9-97fa-0bc866345225-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n\" (UID: \"e86d2b84-af8e-43d9-97fa-0bc866345225\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n"
May 26 16:42:31.176120 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:31.176070    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e86d2b84-af8e-43d9-97fa-0bc866345225-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n\" (UID: \"e86d2b84-af8e-43d9-97fa-0bc866345225\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n"
May 26 16:42:31.176120 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:31.176090    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zbvp4\" (UniqueName: \"kubernetes.io/projected/e86d2b84-af8e-43d9-97fa-0bc866345225-kube-api-access-zbvp4\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n\" (UID: \"e86d2b84-af8e-43d9-97fa-0bc866345225\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n"
May 26 16:42:31.176424 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:31.176310    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e86d2b84-af8e-43d9-97fa-0bc866345225-isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n\" (UID: \"e86d2b84-af8e-43d9-97fa-0bc866345225\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n"
May 26 16:42:31.176510 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:31.176473    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e86d2b84-af8e-43d9-97fa-0bc866345225-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n\" (UID: \"e86d2b84-af8e-43d9-97fa-0bc866345225\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n"
May 26 16:42:31.176974 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:31.176945    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e86d2b84-af8e-43d9-97fa-0bc866345225-isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n\" (UID: \"e86d2b84-af8e-43d9-97fa-0bc866345225\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n"
May 26 16:42:31.178560 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:31.178539    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e86d2b84-af8e-43d9-97fa-0bc866345225-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n\" (UID: \"e86d2b84-af8e-43d9-97fa-0bc866345225\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n"
May 26 16:42:31.184225 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:31.184204    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbvp4\" (UniqueName: \"kubernetes.io/projected/e86d2b84-af8e-43d9-97fa-0bc866345225-kube-api-access-zbvp4\") pod \"isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n\" (UID: \"e86d2b84-af8e-43d9-97fa-0bc866345225\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n"
May 26 16:42:31.240850 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:31.240813    2572 generic.go:358] "Generic (PLEG): container finished" podID="7314872d-3df0-4439-828d-8c26d132733a" containerID="3bdd1ecbb734963dd2c57b6342dfdc3218b25d2c51c5cd3faac4ce50979b3979" exitCode=2
May 26 16:42:31.241059 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:31.240892    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm" event={"ID":"7314872d-3df0-4439-828d-8c26d132733a","Type":"ContainerDied","Data":"3bdd1ecbb734963dd2c57b6342dfdc3218b25d2c51c5cd3faac4ce50979b3979"}
May 26 16:42:31.359873 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:31.359823    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n"
May 26 16:42:31.495157 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:31.495025    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n"]
May 26 16:42:31.498069 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:42:31.498040    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode86d2b84_af8e_43d9_97fa_0bc866345225.slice/crio-2833c1063d7125caf0442c6e2a2a0d09f7a711e322a9c6d7333f64f55ae9df16 WatchSource:0}: Error finding container 2833c1063d7125caf0442c6e2a2a0d09f7a711e322a9c6d7333f64f55ae9df16: Status 404 returned error can't find the container with id 2833c1063d7125caf0442c6e2a2a0d09f7a711e322a9c6d7333f64f55ae9df16
May 26 16:42:32.244727 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:32.244691    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n" event={"ID":"e86d2b84-af8e-43d9-97fa-0bc866345225","Type":"ContainerStarted","Data":"ac0be53ad7818351b398e48347d16c0fcd4877b2f65f534f95ef568b29e580f3"}
May 26 16:42:32.244727 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:32.244728    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n" event={"ID":"e86d2b84-af8e-43d9-97fa-0bc866345225","Type":"ContainerStarted","Data":"2833c1063d7125caf0442c6e2a2a0d09f7a711e322a9c6d7333f64f55ae9df16"}
May 26 16:42:34.015718 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:34.015672    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm" podUID="7314872d-3df0-4439-828d-8c26d132733a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused"
May 26 16:42:34.021047 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:34.021023    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm" podUID="7314872d-3df0-4439-828d-8c26d132733a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.43:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.43:8080: connect: connection refused"
May 26 16:42:35.255676 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:35.255635    2572 generic.go:358] "Generic (PLEG): container finished" podID="e86d2b84-af8e-43d9-97fa-0bc866345225" containerID="ac0be53ad7818351b398e48347d16c0fcd4877b2f65f534f95ef568b29e580f3" exitCode=0
May 26 16:42:35.256094 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:35.255688    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n" event={"ID":"e86d2b84-af8e-43d9-97fa-0bc866345225","Type":"ContainerDied","Data":"ac0be53ad7818351b398e48347d16c0fcd4877b2f65f534f95ef568b29e580f3"}
May 26 16:42:35.942397 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:35.942369    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm"
May 26 16:42:36.010940 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:36.010861    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7314872d-3df0-4439-828d-8c26d132733a-kserve-provision-location\") pod \"7314872d-3df0-4439-828d-8c26d132733a\" (UID: \"7314872d-3df0-4439-828d-8c26d132733a\") "
May 26 16:42:36.010940 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:36.010925    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t64hl\" (UniqueName: \"kubernetes.io/projected/7314872d-3df0-4439-828d-8c26d132733a-kube-api-access-t64hl\") pod \"7314872d-3df0-4439-828d-8c26d132733a\" (UID: \"7314872d-3df0-4439-828d-8c26d132733a\") "
May 26 16:42:36.010940 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:36.010941    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7314872d-3df0-4439-828d-8c26d132733a-proxy-tls\") pod \"7314872d-3df0-4439-828d-8c26d132733a\" (UID: \"7314872d-3df0-4439-828d-8c26d132733a\") "
May 26 16:42:36.011168 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:36.010972    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7314872d-3df0-4439-828d-8c26d132733a-isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"7314872d-3df0-4439-828d-8c26d132733a\" (UID: \"7314872d-3df0-4439-828d-8c26d132733a\") "
May 26 16:42:36.011285 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:36.011257    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7314872d-3df0-4439-828d-8c26d132733a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7314872d-3df0-4439-828d-8c26d132733a" (UID: "7314872d-3df0-4439-828d-8c26d132733a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:42:36.011324 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:36.011307    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7314872d-3df0-4439-828d-8c26d132733a-isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config") pod "7314872d-3df0-4439-828d-8c26d132733a" (UID: "7314872d-3df0-4439-828d-8c26d132733a"). InnerVolumeSpecName "isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:42:36.013101 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:36.013078    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7314872d-3df0-4439-828d-8c26d132733a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7314872d-3df0-4439-828d-8c26d132733a" (UID: "7314872d-3df0-4439-828d-8c26d132733a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:42:36.013183 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:36.013118    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7314872d-3df0-4439-828d-8c26d132733a-kube-api-access-t64hl" (OuterVolumeSpecName: "kube-api-access-t64hl") pod "7314872d-3df0-4439-828d-8c26d132733a" (UID: "7314872d-3df0-4439-828d-8c26d132733a"). InnerVolumeSpecName "kube-api-access-t64hl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:42:36.112149 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:36.112112    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t64hl\" (UniqueName: \"kubernetes.io/projected/7314872d-3df0-4439-828d-8c26d132733a-kube-api-access-t64hl\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:42:36.112149 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:36.112144    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7314872d-3df0-4439-828d-8c26d132733a-proxy-tls\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:42:36.112337 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:36.112161    2572 reconciler_common.go:299] "Volume detached for volume \"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7314872d-3df0-4439-828d-8c26d132733a-isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:42:36.112337 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:36.112172    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7314872d-3df0-4439-828d-8c26d132733a-kserve-provision-location\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:42:36.260437 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:36.260395    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n" event={"ID":"e86d2b84-af8e-43d9-97fa-0bc866345225","Type":"ContainerStarted","Data":"9b3788a0f3b3cbac34ecc05df5212604490aabc873edf749ea28de781404092b"}
May 26 16:42:36.260437 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:36.260443    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n" event={"ID":"e86d2b84-af8e-43d9-97fa-0bc866345225","Type":"ContainerStarted","Data":"3aef16ebb17553dfddb7f80e32f47e6e7020332257394ddd089805ca6f23d491"}
May 26 16:42:36.260989 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:36.260671    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n"
May 26 16:42:36.262131 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:36.262104    2572 generic.go:358] "Generic (PLEG): container finished" podID="7314872d-3df0-4439-828d-8c26d132733a" containerID="f1b253e1f3c1aae76ac8628cc79b363973174de4f8399c4b8a29d90bee9d8a1f" exitCode=0
May 26 16:42:36.262256 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:36.262176    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm"
May 26 16:42:36.262256 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:36.262186    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm" event={"ID":"7314872d-3df0-4439-828d-8c26d132733a","Type":"ContainerDied","Data":"f1b253e1f3c1aae76ac8628cc79b363973174de4f8399c4b8a29d90bee9d8a1f"}
May 26 16:42:36.262256 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:36.262220    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm" event={"ID":"7314872d-3df0-4439-828d-8c26d132733a","Type":"ContainerDied","Data":"fdb6cabf154ff0f0e3035bbe1791283dd2dd5e39f8df4aec19d9d3ef384902b2"}
May 26 16:42:36.262256 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:36.262236    2572 scope.go:117] "RemoveContainer" containerID="3bdd1ecbb734963dd2c57b6342dfdc3218b25d2c51c5cd3faac4ce50979b3979"
May 26 16:42:36.270212 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:36.270193    2572 scope.go:117] "RemoveContainer" containerID="f1b253e1f3c1aae76ac8628cc79b363973174de4f8399c4b8a29d90bee9d8a1f"
May 26 16:42:36.276988 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:36.276967    2572 scope.go:117] "RemoveContainer" containerID="9a5251ef4b1507af6b492546cb4e29c26bb4f454aec2f1aa04a451e088eb6211"
May 26 16:42:36.282105 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:36.282057    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n" podStartSLOduration=5.282037084 podStartE2EDuration="5.282037084s" podCreationTimestamp="2026-05-26 16:42:31 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:42:36.280410846 +0000 UTC m=+2472.756138197" watchObservedRunningTime="2026-05-26 16:42:36.282037084 +0000 UTC m=+2472.757764435"
May 26 16:42:36.285046 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:36.285030    2572 scope.go:117] "RemoveContainer" containerID="3bdd1ecbb734963dd2c57b6342dfdc3218b25d2c51c5cd3faac4ce50979b3979"
May 26 16:42:36.285283 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:42:36.285266    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bdd1ecbb734963dd2c57b6342dfdc3218b25d2c51c5cd3faac4ce50979b3979\": container with ID starting with 3bdd1ecbb734963dd2c57b6342dfdc3218b25d2c51c5cd3faac4ce50979b3979 not found: ID does not exist" containerID="3bdd1ecbb734963dd2c57b6342dfdc3218b25d2c51c5cd3faac4ce50979b3979"
May 26 16:42:36.285325 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:36.285293    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bdd1ecbb734963dd2c57b6342dfdc3218b25d2c51c5cd3faac4ce50979b3979"} err="failed to get container status \"3bdd1ecbb734963dd2c57b6342dfdc3218b25d2c51c5cd3faac4ce50979b3979\": rpc error: code = NotFound desc = could not find container \"3bdd1ecbb734963dd2c57b6342dfdc3218b25d2c51c5cd3faac4ce50979b3979\": container with ID starting with 3bdd1ecbb734963dd2c57b6342dfdc3218b25d2c51c5cd3faac4ce50979b3979 not found: ID does not exist"
May 26 16:42:36.285325 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:36.285310    2572 scope.go:117] "RemoveContainer" containerID="f1b253e1f3c1aae76ac8628cc79b363973174de4f8399c4b8a29d90bee9d8a1f"
May 26 16:42:36.285501 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:42:36.285486    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1b253e1f3c1aae76ac8628cc79b363973174de4f8399c4b8a29d90bee9d8a1f\": container with ID starting with f1b253e1f3c1aae76ac8628cc79b363973174de4f8399c4b8a29d90bee9d8a1f not found: ID does not exist" containerID="f1b253e1f3c1aae76ac8628cc79b363973174de4f8399c4b8a29d90bee9d8a1f"
May 26 16:42:36.285548 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:36.285505    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1b253e1f3c1aae76ac8628cc79b363973174de4f8399c4b8a29d90bee9d8a1f"} err="failed to get container status \"f1b253e1f3c1aae76ac8628cc79b363973174de4f8399c4b8a29d90bee9d8a1f\": rpc error: code = NotFound desc = could not find container \"f1b253e1f3c1aae76ac8628cc79b363973174de4f8399c4b8a29d90bee9d8a1f\": container with ID starting with f1b253e1f3c1aae76ac8628cc79b363973174de4f8399c4b8a29d90bee9d8a1f not found: ID does not exist"
May 26 16:42:36.285548 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:36.285518    2572 scope.go:117] "RemoveContainer" containerID="9a5251ef4b1507af6b492546cb4e29c26bb4f454aec2f1aa04a451e088eb6211"
May 26 16:42:36.285774 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:42:36.285758    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a5251ef4b1507af6b492546cb4e29c26bb4f454aec2f1aa04a451e088eb6211\": container with ID starting with 9a5251ef4b1507af6b492546cb4e29c26bb4f454aec2f1aa04a451e088eb6211 not found: ID does not exist" containerID="9a5251ef4b1507af6b492546cb4e29c26bb4f454aec2f1aa04a451e088eb6211"
May 26 16:42:36.285827 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:36.285779    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a5251ef4b1507af6b492546cb4e29c26bb4f454aec2f1aa04a451e088eb6211"} err="failed to get container status \"9a5251ef4b1507af6b492546cb4e29c26bb4f454aec2f1aa04a451e088eb6211\": rpc error: code = NotFound desc = could not find container \"9a5251ef4b1507af6b492546cb4e29c26bb4f454aec2f1aa04a451e088eb6211\": container with ID starting with 9a5251ef4b1507af6b492546cb4e29c26bb4f454aec2f1aa04a451e088eb6211 not found: ID does not exist"
May 26 16:42:36.293509 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:36.293475    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm"]
May 26 16:42:36.298255 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:36.298231    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-b5d4f6b79-789jm"]
May 26 16:42:37.266184 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:37.266156    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n"
May 26 16:42:38.078973 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:38.078935    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7314872d-3df0-4439-828d-8c26d132733a" path="/var/lib/kubelet/pods/7314872d-3df0-4439-828d-8c26d132733a/volumes"
May 26 16:42:43.274501 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:42:43.274476    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n"
May 26 16:43:13.275911 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:43:13.275866    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n" podUID="e86d2b84-af8e-43d9-97fa-0bc866345225" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.44:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.44:8080: connect: connection refused"
May 26 16:43:23.275214 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:43:23.275166    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n" podUID="e86d2b84-af8e-43d9-97fa-0bc866345225" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.44:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.44:8080: connect: connection refused"
May 26 16:43:33.275853 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:43:33.275812    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n" podUID="e86d2b84-af8e-43d9-97fa-0bc866345225" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.44:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.44:8080: connect: connection refused"
May 26 16:43:43.275357 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:43:43.275320    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n" podUID="e86d2b84-af8e-43d9-97fa-0bc866345225" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.44:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.44:8080: connect: connection refused"
May 26 16:43:52.083952 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:43:52.083922    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n"
May 26 16:44:01.161846 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:01.161810    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n"]
May 26 16:44:01.162324 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:01.162296    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n" podUID="e86d2b84-af8e-43d9-97fa-0bc866345225" containerName="kserve-container" containerID="cri-o://3aef16ebb17553dfddb7f80e32f47e6e7020332257394ddd089805ca6f23d491" gracePeriod=30
May 26 16:44:01.162393 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:01.162345    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n" podUID="e86d2b84-af8e-43d9-97fa-0bc866345225" containerName="kube-rbac-proxy" containerID="cri-o://9b3788a0f3b3cbac34ecc05df5212604490aabc873edf749ea28de781404092b" gracePeriod=30
May 26 16:44:01.280930 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:01.280892    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx"]
May 26 16:44:01.281386 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:01.281299    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7314872d-3df0-4439-828d-8c26d132733a" containerName="kube-rbac-proxy"
May 26 16:44:01.281386 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:01.281319    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7314872d-3df0-4439-828d-8c26d132733a" containerName="kube-rbac-proxy"
May 26 16:44:01.281386 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:01.281343    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7314872d-3df0-4439-828d-8c26d132733a" containerName="storage-initializer"
May 26 16:44:01.281386 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:01.281352    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7314872d-3df0-4439-828d-8c26d132733a" containerName="storage-initializer"
May 26 16:44:01.281386 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:01.281373    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7314872d-3df0-4439-828d-8c26d132733a" containerName="kserve-container"
May 26 16:44:01.281386 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:01.281382    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7314872d-3df0-4439-828d-8c26d132733a" containerName="kserve-container"
May 26 16:44:01.281766 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:01.281456    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="7314872d-3df0-4439-828d-8c26d132733a" containerName="kube-rbac-proxy"
May 26 16:44:01.281766 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:01.281468    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="7314872d-3df0-4439-828d-8c26d132733a" containerName="kserve-container"
May 26 16:44:01.284761 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:01.284737    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx"
May 26 16:44:01.286941 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:01.286918    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
May 26 16:44:01.287169 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:01.287150    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
May 26 16:44:01.291806 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:01.291730    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx"]
May 26 16:44:01.365291 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:01.365252    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx\" (UID: \"f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx"
May 26 16:44:01.365469 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:01.365310    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44bvc\" (UniqueName: \"kubernetes.io/projected/f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a-kube-api-access-44bvc\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx\" (UID: \"f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx"
May 26 16:44:01.365469 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:01.365394    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx\" (UID: \"f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx"
May 26 16:44:01.365469 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:01.365431    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a-isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx\" (UID: \"f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx"
May 26 16:44:01.466785 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:01.466695    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-44bvc\" (UniqueName: \"kubernetes.io/projected/f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a-kube-api-access-44bvc\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx\" (UID: \"f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx"
May 26 16:44:01.466785 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:01.466763    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx\" (UID: \"f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx"
May 26 16:44:01.466991 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:01.466790    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a-isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx\" (UID: \"f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx"
May 26 16:44:01.466991 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:01.466817    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx\" (UID: \"f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx"
May 26 16:44:01.466991 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:44:01.466908    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-serving-cert: secret "isvc-predictive-lightgbm-v2-predictor-serving-cert" not found
May 26 16:44:01.466991 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:44:01.466981    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a-proxy-tls podName:f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a nodeName:}" failed. No retries permitted until 2026-05-26 16:44:01.966960333 +0000 UTC m=+2558.442687671 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a-proxy-tls") pod "isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx" (UID: "f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a") : secret "isvc-predictive-lightgbm-v2-predictor-serving-cert" not found
May 26 16:44:01.467241 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:01.467223    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx\" (UID: \"f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx"
May 26 16:44:01.467504 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:01.467484    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a-isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx\" (UID: \"f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx"
May 26 16:44:01.476390 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:01.476365    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44bvc\" (UniqueName: \"kubernetes.io/projected/f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a-kube-api-access-44bvc\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx\" (UID: \"f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx"
May 26 16:44:01.499020 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:01.498990    2572 generic.go:358] "Generic (PLEG): container finished" podID="e86d2b84-af8e-43d9-97fa-0bc866345225" containerID="9b3788a0f3b3cbac34ecc05df5212604490aabc873edf749ea28de781404092b" exitCode=2
May 26 16:44:01.499175 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:01.499066    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n" event={"ID":"e86d2b84-af8e-43d9-97fa-0bc866345225","Type":"ContainerDied","Data":"9b3788a0f3b3cbac34ecc05df5212604490aabc873edf749ea28de781404092b"}
May 26 16:44:01.972322 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:01.972284    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx\" (UID: \"f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx"
May 26 16:44:01.974764 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:01.974743    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx\" (UID: \"f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx"
May 26 16:44:02.076383 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:02.076329    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n" podUID="e86d2b84-af8e-43d9-97fa-0bc866345225" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.44:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.44:8080: connect: connection refused"
May 26 16:44:02.197859 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:02.197810    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx"
May 26 16:44:02.334659 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:02.334627    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx"]
May 26 16:44:02.337402 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:44:02.337361    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4eaa23d_9d87_4b48_8c5a_5f35ca115d8a.slice/crio-8d24d7f9a008e68a2f7c285dc7d32500f28f2de2e80ee07f6c60ecab2718c629 WatchSource:0}: Error finding container 8d24d7f9a008e68a2f7c285dc7d32500f28f2de2e80ee07f6c60ecab2718c629: Status 404 returned error can't find the container with id 8d24d7f9a008e68a2f7c285dc7d32500f28f2de2e80ee07f6c60ecab2718c629
May 26 16:44:02.503251 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:02.503158    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx" event={"ID":"f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a","Type":"ContainerStarted","Data":"4da63828b8bb40a003e8e0198d862cef0d1e9398124a56ba3381b7ede80bbef3"}
May 26 16:44:02.503251 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:02.503204    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx" event={"ID":"f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a","Type":"ContainerStarted","Data":"8d24d7f9a008e68a2f7c285dc7d32500f28f2de2e80ee07f6c60ecab2718c629"}
May 26 16:44:03.270924 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:03.270886    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n" podUID="e86d2b84-af8e-43d9-97fa-0bc866345225" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
May 26 16:44:06.104575 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:06.104549    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n"
May 26 16:44:06.203859 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:06.203772    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zbvp4\" (UniqueName: \"kubernetes.io/projected/e86d2b84-af8e-43d9-97fa-0bc866345225-kube-api-access-zbvp4\") pod \"e86d2b84-af8e-43d9-97fa-0bc866345225\" (UID: \"e86d2b84-af8e-43d9-97fa-0bc866345225\") "
May 26 16:44:06.204017 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:06.203872    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e86d2b84-af8e-43d9-97fa-0bc866345225-isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"e86d2b84-af8e-43d9-97fa-0bc866345225\" (UID: \"e86d2b84-af8e-43d9-97fa-0bc866345225\") "
May 26 16:44:06.204017 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:06.203903    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e86d2b84-af8e-43d9-97fa-0bc866345225-kserve-provision-location\") pod \"e86d2b84-af8e-43d9-97fa-0bc866345225\" (UID: \"e86d2b84-af8e-43d9-97fa-0bc866345225\") "
May 26 16:44:06.204017 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:06.203925    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e86d2b84-af8e-43d9-97fa-0bc866345225-proxy-tls\") pod \"e86d2b84-af8e-43d9-97fa-0bc866345225\" (UID: \"e86d2b84-af8e-43d9-97fa-0bc866345225\") "
May 26 16:44:06.204249 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:06.204220    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e86d2b84-af8e-43d9-97fa-0bc866345225-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e86d2b84-af8e-43d9-97fa-0bc866345225" (UID: "e86d2b84-af8e-43d9-97fa-0bc866345225"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:44:06.204249 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:06.204233    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e86d2b84-af8e-43d9-97fa-0bc866345225-isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config") pod "e86d2b84-af8e-43d9-97fa-0bc866345225" (UID: "e86d2b84-af8e-43d9-97fa-0bc866345225"). InnerVolumeSpecName "isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:44:06.205989 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:06.205962    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e86d2b84-af8e-43d9-97fa-0bc866345225-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e86d2b84-af8e-43d9-97fa-0bc866345225" (UID: "e86d2b84-af8e-43d9-97fa-0bc866345225"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:44:06.206080 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:06.206029    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e86d2b84-af8e-43d9-97fa-0bc866345225-kube-api-access-zbvp4" (OuterVolumeSpecName: "kube-api-access-zbvp4") pod "e86d2b84-af8e-43d9-97fa-0bc866345225" (UID: "e86d2b84-af8e-43d9-97fa-0bc866345225"). InnerVolumeSpecName "kube-api-access-zbvp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:44:06.304912 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:06.304869    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e86d2b84-af8e-43d9-97fa-0bc866345225-kserve-provision-location\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:44:06.304912 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:06.304905    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e86d2b84-af8e-43d9-97fa-0bc866345225-proxy-tls\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:44:06.304912 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:06.304917    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbvp4\" (UniqueName: \"kubernetes.io/projected/e86d2b84-af8e-43d9-97fa-0bc866345225-kube-api-access-zbvp4\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:44:06.305150 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:06.304927    2572 reconciler_common.go:299] "Volume detached for volume \"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e86d2b84-af8e-43d9-97fa-0bc866345225-isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:44:06.515041 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:06.514941    2572 generic.go:358] "Generic (PLEG): container finished" podID="f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a" containerID="4da63828b8bb40a003e8e0198d862cef0d1e9398124a56ba3381b7ede80bbef3" exitCode=0
May 26 16:44:06.515041 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:06.515015    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx" event={"ID":"f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a","Type":"ContainerDied","Data":"4da63828b8bb40a003e8e0198d862cef0d1e9398124a56ba3381b7ede80bbef3"}
May 26 16:44:06.517798 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:06.517767    2572 generic.go:358] "Generic (PLEG): container finished" podID="e86d2b84-af8e-43d9-97fa-0bc866345225" containerID="3aef16ebb17553dfddb7f80e32f47e6e7020332257394ddd089805ca6f23d491" exitCode=0
May 26 16:44:06.517953 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:06.517840    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n" event={"ID":"e86d2b84-af8e-43d9-97fa-0bc866345225","Type":"ContainerDied","Data":"3aef16ebb17553dfddb7f80e32f47e6e7020332257394ddd089805ca6f23d491"}
May 26 16:44:06.517953 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:06.517872    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n" event={"ID":"e86d2b84-af8e-43d9-97fa-0bc866345225","Type":"ContainerDied","Data":"2833c1063d7125caf0442c6e2a2a0d09f7a711e322a9c6d7333f64f55ae9df16"}
May 26 16:44:06.517953 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:06.517905    2572 scope.go:117] "RemoveContainer" containerID="9b3788a0f3b3cbac34ecc05df5212604490aabc873edf749ea28de781404092b"
May 26 16:44:06.518310 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:06.518223    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n"
May 26 16:44:06.527873 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:06.527851    2572 scope.go:117] "RemoveContainer" containerID="3aef16ebb17553dfddb7f80e32f47e6e7020332257394ddd089805ca6f23d491"
May 26 16:44:06.539362 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:06.539342    2572 scope.go:117] "RemoveContainer" containerID="ac0be53ad7818351b398e48347d16c0fcd4877b2f65f534f95ef568b29e580f3"
May 26 16:44:06.550025 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:06.549998    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n"]
May 26 16:44:06.551503 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:06.551478    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-6577c65fd8-7xn4n"]
May 26 16:44:06.556306 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:06.556283    2572 scope.go:117] "RemoveContainer" containerID="9b3788a0f3b3cbac34ecc05df5212604490aabc873edf749ea28de781404092b"
May 26 16:44:06.556620 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:44:06.556580    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b3788a0f3b3cbac34ecc05df5212604490aabc873edf749ea28de781404092b\": container with ID starting with 9b3788a0f3b3cbac34ecc05df5212604490aabc873edf749ea28de781404092b not found: ID does not exist" containerID="9b3788a0f3b3cbac34ecc05df5212604490aabc873edf749ea28de781404092b"
May 26 16:44:06.556706 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:06.556634    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b3788a0f3b3cbac34ecc05df5212604490aabc873edf749ea28de781404092b"} err="failed to get container status \"9b3788a0f3b3cbac34ecc05df5212604490aabc873edf749ea28de781404092b\": rpc error: code = NotFound desc = could not find container \"9b3788a0f3b3cbac34ecc05df5212604490aabc873edf749ea28de781404092b\": container with ID starting with 9b3788a0f3b3cbac34ecc05df5212604490aabc873edf749ea28de781404092b not found: ID does not exist"
May 26 16:44:06.556706 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:06.556666    2572 scope.go:117] "RemoveContainer" containerID="3aef16ebb17553dfddb7f80e32f47e6e7020332257394ddd089805ca6f23d491"
May 26 16:44:06.556973 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:44:06.556956    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3aef16ebb17553dfddb7f80e32f47e6e7020332257394ddd089805ca6f23d491\": container with ID starting with 3aef16ebb17553dfddb7f80e32f47e6e7020332257394ddd089805ca6f23d491 not found: ID does not exist" containerID="3aef16ebb17553dfddb7f80e32f47e6e7020332257394ddd089805ca6f23d491"
May 26 16:44:06.557016 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:06.556982    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aef16ebb17553dfddb7f80e32f47e6e7020332257394ddd089805ca6f23d491"} err="failed to get container status \"3aef16ebb17553dfddb7f80e32f47e6e7020332257394ddd089805ca6f23d491\": rpc error: code = NotFound desc = could not find container \"3aef16ebb17553dfddb7f80e32f47e6e7020332257394ddd089805ca6f23d491\": container with ID starting with 3aef16ebb17553dfddb7f80e32f47e6e7020332257394ddd089805ca6f23d491 not found: ID does not exist"
May 26 16:44:06.557016 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:06.557000    2572 scope.go:117] "RemoveContainer" containerID="ac0be53ad7818351b398e48347d16c0fcd4877b2f65f534f95ef568b29e580f3"
May 26 16:44:06.557267 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:44:06.557246    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac0be53ad7818351b398e48347d16c0fcd4877b2f65f534f95ef568b29e580f3\": container with ID starting with ac0be53ad7818351b398e48347d16c0fcd4877b2f65f534f95ef568b29e580f3 not found: ID does not exist" containerID="ac0be53ad7818351b398e48347d16c0fcd4877b2f65f534f95ef568b29e580f3"
May 26 16:44:06.557316 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:06.557277    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac0be53ad7818351b398e48347d16c0fcd4877b2f65f534f95ef568b29e580f3"} err="failed to get container status \"ac0be53ad7818351b398e48347d16c0fcd4877b2f65f534f95ef568b29e580f3\": rpc error: code = NotFound desc = could not find container \"ac0be53ad7818351b398e48347d16c0fcd4877b2f65f534f95ef568b29e580f3\": container with ID starting with ac0be53ad7818351b398e48347d16c0fcd4877b2f65f534f95ef568b29e580f3 not found: ID does not exist"
May 26 16:44:07.522526 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:07.522489    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx" event={"ID":"f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a","Type":"ContainerStarted","Data":"17671a06baf0c3345d704235b62a5272ca0b8e85fbfd6dda61e838b81a3485fb"}
May 26 16:44:07.522965 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:07.522537    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx" event={"ID":"f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a","Type":"ContainerStarted","Data":"476e94bc9be28422323a265e0945505916c31ec7b7ebac1802ca8cfc68ac4125"}
May 26 16:44:07.522965 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:07.522868    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx"
May 26 16:44:07.522965 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:07.522902    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx"
May 26 16:44:07.544738 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:07.544685    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx" podStartSLOduration=6.544668442 podStartE2EDuration="6.544668442s" podCreationTimestamp="2026-05-26 16:44:01 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:44:07.542733669 +0000 UTC m=+2564.018461032" watchObservedRunningTime="2026-05-26 16:44:07.544668442 +0000 UTC m=+2564.020395791"
May 26 16:44:08.080375 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:08.080344    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e86d2b84-af8e-43d9-97fa-0bc866345225" path="/var/lib/kubelet/pods/e86d2b84-af8e-43d9-97fa-0bc866345225/volumes"
May 26 16:44:13.531989 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:13.531962    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx"
May 26 16:44:43.533509 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:43.533413    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx" podUID="f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.45:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.45:8080: connect: connection refused"
May 26 16:44:53.533333 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:44:53.533291    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx" podUID="f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.45:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.45:8080: connect: connection refused"
May 26 16:45:03.532811 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:03.532773    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx" podUID="f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.45:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.45:8080: connect: connection refused"
May 26 16:45:13.533511 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:13.533467    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx" podUID="f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.45:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.45:8080: connect: connection refused"
May 26 16:45:23.535747 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:23.535720    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx"
May 26 16:45:31.339085 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:31.339041    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx"]
May 26 16:45:31.339544 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:31.339458    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx" podUID="f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a" containerName="kserve-container" containerID="cri-o://476e94bc9be28422323a265e0945505916c31ec7b7ebac1802ca8cfc68ac4125" gracePeriod=30
May 26 16:45:31.339642 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:31.339507    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx" podUID="f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a" containerName="kube-rbac-proxy" containerID="cri-o://17671a06baf0c3345d704235b62a5272ca0b8e85fbfd6dda61e838b81a3485fb" gracePeriod=30
May 26 16:45:31.763754 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:31.763717    2572 generic.go:358] "Generic (PLEG): container finished" podID="f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a" containerID="17671a06baf0c3345d704235b62a5272ca0b8e85fbfd6dda61e838b81a3485fb" exitCode=2
May 26 16:45:31.763943 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:31.763794    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx" event={"ID":"f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a","Type":"ContainerDied","Data":"17671a06baf0c3345d704235b62a5272ca0b8e85fbfd6dda61e838b81a3485fb"}
May 26 16:45:33.498353 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:33.498313    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-k9h2x"]
May 26 16:45:33.498787 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:33.498574    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e86d2b84-af8e-43d9-97fa-0bc866345225" containerName="storage-initializer"
May 26 16:45:33.498787 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:33.498584    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e86d2b84-af8e-43d9-97fa-0bc866345225" containerName="storage-initializer"
May 26 16:45:33.498787 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:33.498592    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e86d2b84-af8e-43d9-97fa-0bc866345225" containerName="kserve-container"
May 26 16:45:33.498787 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:33.498598    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e86d2b84-af8e-43d9-97fa-0bc866345225" containerName="kserve-container"
May 26 16:45:33.498787 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:33.498622    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e86d2b84-af8e-43d9-97fa-0bc866345225" containerName="kube-rbac-proxy"
May 26 16:45:33.498787 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:33.498628    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e86d2b84-af8e-43d9-97fa-0bc866345225" containerName="kube-rbac-proxy"
May 26 16:45:33.498787 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:33.498675    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="e86d2b84-af8e-43d9-97fa-0bc866345225" containerName="kserve-container"
May 26 16:45:33.498787 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:33.498687    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="e86d2b84-af8e-43d9-97fa-0bc866345225" containerName="kube-rbac-proxy"
May 26 16:45:33.501708 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:33.501677    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-k9h2x"
May 26 16:45:33.503773 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:33.503751    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
May 26 16:45:33.503906 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:33.503752    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-predictor-serving-cert\""
May 26 16:45:33.510516 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:33.510487    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-k9h2x"]
May 26 16:45:33.527530 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:33.527493    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx" podUID="f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused"
May 26 16:45:33.532426 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:33.532396    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx" podUID="f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.45:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.45:8080: connect: connection refused"
May 26 16:45:33.537937 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:33.537912    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d15d642a-e575-44e1-894d-1a58c2e1e283-proxy-tls\") pod \"isvc-sklearn-predictor-bc9cf9b57-k9h2x\" (UID: \"d15d642a-e575-44e1-894d-1a58c2e1e283\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-k9h2x"
May 26 16:45:33.538081 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:33.537970    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h9rh\" (UniqueName: \"kubernetes.io/projected/d15d642a-e575-44e1-894d-1a58c2e1e283-kube-api-access-2h9rh\") pod \"isvc-sklearn-predictor-bc9cf9b57-k9h2x\" (UID: \"d15d642a-e575-44e1-894d-1a58c2e1e283\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-k9h2x"
May 26 16:45:33.538081 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:33.538003    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d15d642a-e575-44e1-894d-1a58c2e1e283-isvc-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-bc9cf9b57-k9h2x\" (UID: \"d15d642a-e575-44e1-894d-1a58c2e1e283\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-k9h2x"
May 26 16:45:33.538081 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:33.538036    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d15d642a-e575-44e1-894d-1a58c2e1e283-kserve-provision-location\") pod \"isvc-sklearn-predictor-bc9cf9b57-k9h2x\" (UID: \"d15d642a-e575-44e1-894d-1a58c2e1e283\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-k9h2x"
May 26 16:45:33.639009 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:33.638966    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d15d642a-e575-44e1-894d-1a58c2e1e283-kserve-provision-location\") pod \"isvc-sklearn-predictor-bc9cf9b57-k9h2x\" (UID: \"d15d642a-e575-44e1-894d-1a58c2e1e283\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-k9h2x"
May 26 16:45:33.639215 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:33.639019    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d15d642a-e575-44e1-894d-1a58c2e1e283-proxy-tls\") pod \"isvc-sklearn-predictor-bc9cf9b57-k9h2x\" (UID: \"d15d642a-e575-44e1-894d-1a58c2e1e283\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-k9h2x"
May 26 16:45:33.639215 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:33.639084    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2h9rh\" (UniqueName: \"kubernetes.io/projected/d15d642a-e575-44e1-894d-1a58c2e1e283-kube-api-access-2h9rh\") pod \"isvc-sklearn-predictor-bc9cf9b57-k9h2x\" (UID: \"d15d642a-e575-44e1-894d-1a58c2e1e283\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-k9h2x"
May 26 16:45:33.639215 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:33.639116    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d15d642a-e575-44e1-894d-1a58c2e1e283-isvc-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-bc9cf9b57-k9h2x\" (UID: \"d15d642a-e575-44e1-894d-1a58c2e1e283\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-k9h2x"
May 26 16:45:33.639385 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:45:33.639234    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-predictor-serving-cert: secret "isvc-sklearn-predictor-serving-cert" not found
May 26 16:45:33.639385 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:45:33.639311    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d15d642a-e575-44e1-894d-1a58c2e1e283-proxy-tls podName:d15d642a-e575-44e1-894d-1a58c2e1e283 nodeName:}" failed. No retries permitted until 2026-05-26 16:45:34.139289109 +0000 UTC m=+2650.615016450 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d15d642a-e575-44e1-894d-1a58c2e1e283-proxy-tls") pod "isvc-sklearn-predictor-bc9cf9b57-k9h2x" (UID: "d15d642a-e575-44e1-894d-1a58c2e1e283") : secret "isvc-sklearn-predictor-serving-cert" not found
May 26 16:45:33.639552 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:33.639530    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d15d642a-e575-44e1-894d-1a58c2e1e283-kserve-provision-location\") pod \"isvc-sklearn-predictor-bc9cf9b57-k9h2x\" (UID: \"d15d642a-e575-44e1-894d-1a58c2e1e283\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-k9h2x"
May 26 16:45:33.639893 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:33.639876    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d15d642a-e575-44e1-894d-1a58c2e1e283-isvc-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-bc9cf9b57-k9h2x\" (UID: \"d15d642a-e575-44e1-894d-1a58c2e1e283\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-k9h2x"
May 26 16:45:33.647633 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:33.647582    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h9rh\" (UniqueName: \"kubernetes.io/projected/d15d642a-e575-44e1-894d-1a58c2e1e283-kube-api-access-2h9rh\") pod \"isvc-sklearn-predictor-bc9cf9b57-k9h2x\" (UID: \"d15d642a-e575-44e1-894d-1a58c2e1e283\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-k9h2x"
May 26 16:45:34.142682 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:34.142638    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d15d642a-e575-44e1-894d-1a58c2e1e283-proxy-tls\") pod \"isvc-sklearn-predictor-bc9cf9b57-k9h2x\" (UID: \"d15d642a-e575-44e1-894d-1a58c2e1e283\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-k9h2x"
May 26 16:45:34.145053 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:34.145021    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d15d642a-e575-44e1-894d-1a58c2e1e283-proxy-tls\") pod \"isvc-sklearn-predictor-bc9cf9b57-k9h2x\" (UID: \"d15d642a-e575-44e1-894d-1a58c2e1e283\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-k9h2x"
May 26 16:45:34.412866 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:34.412828    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-k9h2x"
May 26 16:45:34.540483 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:34.540453    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-k9h2x"]
May 26 16:45:34.543569 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:45:34.543539    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd15d642a_e575_44e1_894d_1a58c2e1e283.slice/crio-180b0d3d45b76d5c7844edd5c8fd4f7195898db0db20d31b7976e38ad5c018ad WatchSource:0}: Error finding container 180b0d3d45b76d5c7844edd5c8fd4f7195898db0db20d31b7976e38ad5c018ad: Status 404 returned error can't find the container with id 180b0d3d45b76d5c7844edd5c8fd4f7195898db0db20d31b7976e38ad5c018ad
May 26 16:45:34.545343 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:34.545326    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 16:45:34.773244 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:34.773155    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-k9h2x" event={"ID":"d15d642a-e575-44e1-894d-1a58c2e1e283","Type":"ContainerStarted","Data":"ffdfe770fb1c6903c3f051338587ab62201c88ffa1e64f07ca458ef0b222b3a4"}
May 26 16:45:34.773244 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:34.773197    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-k9h2x" event={"ID":"d15d642a-e575-44e1-894d-1a58c2e1e283","Type":"ContainerStarted","Data":"180b0d3d45b76d5c7844edd5c8fd4f7195898db0db20d31b7976e38ad5c018ad"}
May 26 16:45:36.682372 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:36.682343    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx"
May 26 16:45:36.759481 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:36.759447    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a-proxy-tls\") pod \"f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a\" (UID: \"f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a\") "
May 26 16:45:36.759700 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:36.759490    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44bvc\" (UniqueName: \"kubernetes.io/projected/f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a-kube-api-access-44bvc\") pod \"f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a\" (UID: \"f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a\") "
May 26 16:45:36.759700 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:36.759519    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a-kserve-provision-location\") pod \"f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a\" (UID: \"f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a\") "
May 26 16:45:36.759700 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:36.759553    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a-isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\") pod \"f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a\" (UID: \"f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a\") "
May 26 16:45:36.759978 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:36.759947    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a" (UID: "f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:45:36.760041 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:36.759980    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a-isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config") pod "f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a" (UID: "f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a"). InnerVolumeSpecName "isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:45:36.761629 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:36.761587    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a-kube-api-access-44bvc" (OuterVolumeSpecName: "kube-api-access-44bvc") pod "f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a" (UID: "f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a"). InnerVolumeSpecName "kube-api-access-44bvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:45:36.761715 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:36.761633    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a" (UID: "f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:45:36.780412 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:36.780383    2572 generic.go:358] "Generic (PLEG): container finished" podID="f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a" containerID="476e94bc9be28422323a265e0945505916c31ec7b7ebac1802ca8cfc68ac4125" exitCode=0
May 26 16:45:36.780550 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:36.780455    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx" event={"ID":"f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a","Type":"ContainerDied","Data":"476e94bc9be28422323a265e0945505916c31ec7b7ebac1802ca8cfc68ac4125"}
May 26 16:45:36.780550 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:36.780458    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx"
May 26 16:45:36.780550 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:36.780485    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx" event={"ID":"f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a","Type":"ContainerDied","Data":"8d24d7f9a008e68a2f7c285dc7d32500f28f2de2e80ee07f6c60ecab2718c629"}
May 26 16:45:36.780550 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:36.780505    2572 scope.go:117] "RemoveContainer" containerID="17671a06baf0c3345d704235b62a5272ca0b8e85fbfd6dda61e838b81a3485fb"
May 26 16:45:36.789850 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:36.789783    2572 scope.go:117] "RemoveContainer" containerID="476e94bc9be28422323a265e0945505916c31ec7b7ebac1802ca8cfc68ac4125"
May 26 16:45:36.796838 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:36.796817    2572 scope.go:117] "RemoveContainer" containerID="4da63828b8bb40a003e8e0198d862cef0d1e9398124a56ba3381b7ede80bbef3"
May 26 16:45:36.802147 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:36.802118    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx"]
May 26 16:45:36.806087 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:36.806065    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-865b4598f7-fnwwx"]
May 26 16:45:36.806160 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:36.806126    2572 scope.go:117] "RemoveContainer" containerID="17671a06baf0c3345d704235b62a5272ca0b8e85fbfd6dda61e838b81a3485fb"
May 26 16:45:36.806463 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:45:36.806436    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17671a06baf0c3345d704235b62a5272ca0b8e85fbfd6dda61e838b81a3485fb\": container with ID starting with 17671a06baf0c3345d704235b62a5272ca0b8e85fbfd6dda61e838b81a3485fb not found: ID does not exist" containerID="17671a06baf0c3345d704235b62a5272ca0b8e85fbfd6dda61e838b81a3485fb"
May 26 16:45:36.806552 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:36.806472    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17671a06baf0c3345d704235b62a5272ca0b8e85fbfd6dda61e838b81a3485fb"} err="failed to get container status \"17671a06baf0c3345d704235b62a5272ca0b8e85fbfd6dda61e838b81a3485fb\": rpc error: code = NotFound desc = could not find container \"17671a06baf0c3345d704235b62a5272ca0b8e85fbfd6dda61e838b81a3485fb\": container with ID starting with 17671a06baf0c3345d704235b62a5272ca0b8e85fbfd6dda61e838b81a3485fb not found: ID does not exist"
May 26 16:45:36.806552 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:36.806498    2572 scope.go:117] "RemoveContainer" containerID="476e94bc9be28422323a265e0945505916c31ec7b7ebac1802ca8cfc68ac4125"
May 26 16:45:36.806992 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:45:36.806965    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"476e94bc9be28422323a265e0945505916c31ec7b7ebac1802ca8cfc68ac4125\": container with ID starting with 476e94bc9be28422323a265e0945505916c31ec7b7ebac1802ca8cfc68ac4125 not found: ID does not exist" containerID="476e94bc9be28422323a265e0945505916c31ec7b7ebac1802ca8cfc68ac4125"
May 26 16:45:36.807076 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:36.806999    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"476e94bc9be28422323a265e0945505916c31ec7b7ebac1802ca8cfc68ac4125"} err="failed to get container status \"476e94bc9be28422323a265e0945505916c31ec7b7ebac1802ca8cfc68ac4125\": rpc error: code = NotFound desc = could not find container \"476e94bc9be28422323a265e0945505916c31ec7b7ebac1802ca8cfc68ac4125\": container with ID starting with 476e94bc9be28422323a265e0945505916c31ec7b7ebac1802ca8cfc68ac4125 not found: ID does not exist"
May 26 16:45:36.807076 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:36.807024    2572 scope.go:117] "RemoveContainer" containerID="4da63828b8bb40a003e8e0198d862cef0d1e9398124a56ba3381b7ede80bbef3"
May 26 16:45:36.807282 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:45:36.807265    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4da63828b8bb40a003e8e0198d862cef0d1e9398124a56ba3381b7ede80bbef3\": container with ID starting with 4da63828b8bb40a003e8e0198d862cef0d1e9398124a56ba3381b7ede80bbef3 not found: ID does not exist" containerID="4da63828b8bb40a003e8e0198d862cef0d1e9398124a56ba3381b7ede80bbef3"
May 26 16:45:36.807327 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:36.807289    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4da63828b8bb40a003e8e0198d862cef0d1e9398124a56ba3381b7ede80bbef3"} err="failed to get container status \"4da63828b8bb40a003e8e0198d862cef0d1e9398124a56ba3381b7ede80bbef3\": rpc error: code = NotFound desc = could not find container \"4da63828b8bb40a003e8e0198d862cef0d1e9398124a56ba3381b7ede80bbef3\": container with ID starting with 4da63828b8bb40a003e8e0198d862cef0d1e9398124a56ba3381b7ede80bbef3 not found: ID does not exist"
May 26 16:45:36.860772 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:36.860687    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a-proxy-tls\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:45:36.860772 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:36.860718    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44bvc\" (UniqueName: \"kubernetes.io/projected/f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a-kube-api-access-44bvc\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:45:36.860772 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:36.860728    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a-kserve-provision-location\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:45:36.860772 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:36.860738    2572 reconciler_common.go:299] "Volume detached for volume \"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a-isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:45:38.078756 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:38.078725    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a" path="/var/lib/kubelet/pods/f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a/volumes"
May 26 16:45:38.788307 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:38.788217    2572 generic.go:358] "Generic (PLEG): container finished" podID="d15d642a-e575-44e1-894d-1a58c2e1e283" containerID="ffdfe770fb1c6903c3f051338587ab62201c88ffa1e64f07ca458ef0b222b3a4" exitCode=0
May 26 16:45:38.788307 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:38.788295    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-k9h2x" event={"ID":"d15d642a-e575-44e1-894d-1a58c2e1e283","Type":"ContainerDied","Data":"ffdfe770fb1c6903c3f051338587ab62201c88ffa1e64f07ca458ef0b222b3a4"}
May 26 16:45:39.793169 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:39.793129    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-k9h2x" event={"ID":"d15d642a-e575-44e1-894d-1a58c2e1e283","Type":"ContainerStarted","Data":"a95d81d9a23e0cd4603a4bc286f34bfa23e41466af3d93a58cb22e719835d299"}
May 26 16:45:39.793169 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:39.793172    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-k9h2x" event={"ID":"d15d642a-e575-44e1-894d-1a58c2e1e283","Type":"ContainerStarted","Data":"f057f3875dd20a6bbc4ff5e2726d28cdf5c264cf6e0a67ce31d75bef6833d0b8"}
May 26 16:45:39.793597 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:39.793418    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-k9h2x"
May 26 16:45:39.793597 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:39.793528    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-k9h2x"
May 26 16:45:39.794794 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:39.794765    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-k9h2x" podUID="d15d642a-e575-44e1-894d-1a58c2e1e283" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
May 26 16:45:39.812448 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:39.812394    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-k9h2x" podStartSLOduration=6.812374322 podStartE2EDuration="6.812374322s" podCreationTimestamp="2026-05-26 16:45:33 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:45:39.810929161 +0000 UTC m=+2656.286656510" watchObservedRunningTime="2026-05-26 16:45:39.812374322 +0000 UTC m=+2656.288101672"
May 26 16:45:40.795872 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:40.795825    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-k9h2x" podUID="d15d642a-e575-44e1-894d-1a58c2e1e283" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
May 26 16:45:45.800079 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:45.800049    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-k9h2x"
May 26 16:45:45.800503 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:45.800478    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-k9h2x" podUID="d15d642a-e575-44e1-894d-1a58c2e1e283" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
May 26 16:45:55.801086 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:45:55.801040    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-k9h2x" podUID="d15d642a-e575-44e1-894d-1a58c2e1e283" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
May 26 16:46:05.800844 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:05.800592    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-k9h2x" podUID="d15d642a-e575-44e1-894d-1a58c2e1e283" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
May 26 16:46:15.801264 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:15.801177    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-k9h2x" podUID="d15d642a-e575-44e1-894d-1a58c2e1e283" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
May 26 16:46:25.801260 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:25.801222    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-k9h2x" podUID="d15d642a-e575-44e1-894d-1a58c2e1e283" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
May 26 16:46:35.800812 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:35.800773    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-k9h2x" podUID="d15d642a-e575-44e1-894d-1a58c2e1e283" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
May 26 16:46:45.801768 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:45.801738    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-k9h2x"
May 26 16:46:53.598717 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:53.598676    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-k9h2x"]
May 26 16:46:53.599178 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:53.599092    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-k9h2x" podUID="d15d642a-e575-44e1-894d-1a58c2e1e283" containerName="kserve-container" containerID="cri-o://f057f3875dd20a6bbc4ff5e2726d28cdf5c264cf6e0a67ce31d75bef6833d0b8" gracePeriod=30
May 26 16:46:53.599250 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:53.599178    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-k9h2x" podUID="d15d642a-e575-44e1-894d-1a58c2e1e283" containerName="kube-rbac-proxy" containerID="cri-o://a95d81d9a23e0cd4603a4bc286f34bfa23e41466af3d93a58cb22e719835d299" gracePeriod=30
May 26 16:46:53.681232 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:53.681197    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-llw97"]
May 26 16:46:53.681464 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:53.681452    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a" containerName="storage-initializer"
May 26 16:46:53.681519 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:53.681465    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a" containerName="storage-initializer"
May 26 16:46:53.681519 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:53.681481    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a" containerName="kube-rbac-proxy"
May 26 16:46:53.681519 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:53.681487    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a" containerName="kube-rbac-proxy"
May 26 16:46:53.681519 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:53.681498    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a" containerName="kserve-container"
May 26 16:46:53.681519 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:53.681504    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a" containerName="kserve-container"
May 26 16:46:53.681726 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:53.681554    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a" containerName="kserve-container"
May 26 16:46:53.681726 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:53.681561    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="f4eaa23d-9d87-4b48-8c5a-5f35ca115d8a" containerName="kube-rbac-proxy"
May 26 16:46:53.684804 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:53.684783    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-llw97"
May 26 16:46:53.686963 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:53.686938    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
May 26 16:46:53.687070 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:53.687009    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
May 26 16:46:53.695703 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:53.695679    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-llw97"]
May 26 16:46:53.811246 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:53.811207    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btncg\" (UniqueName: \"kubernetes.io/projected/0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd-kube-api-access-btncg\") pod \"sklearn-v2-mlserver-predictor-65d8664766-llw97\" (UID: \"0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-llw97"
May 26 16:46:53.811416 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:53.811279    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-65d8664766-llw97\" (UID: \"0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-llw97"
May 26 16:46:53.811416 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:53.811303    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd-sklearn-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-65d8664766-llw97\" (UID: \"0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-llw97"
May 26 16:46:53.811416 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:53.811321    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-65d8664766-llw97\" (UID: \"0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-llw97"
May 26 16:46:53.911846 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:53.911813    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-65d8664766-llw97\" (UID: \"0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-llw97"
May 26 16:46:53.912010 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:53.911853    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd-sklearn-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-65d8664766-llw97\" (UID: \"0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-llw97"
May 26 16:46:53.912010 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:53.911876    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-65d8664766-llw97\" (UID: \"0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-llw97"
May 26 16:46:53.912010 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:53.911927    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-btncg\" (UniqueName: \"kubernetes.io/projected/0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd-kube-api-access-btncg\") pod \"sklearn-v2-mlserver-predictor-65d8664766-llw97\" (UID: \"0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-llw97"
May 26 16:46:53.912163 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:46:53.912047    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-serving-cert: secret "sklearn-v2-mlserver-predictor-serving-cert" not found
May 26 16:46:53.912163 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:46:53.912155    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd-proxy-tls podName:0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd nodeName:}" failed. No retries permitted until 2026-05-26 16:46:54.412130948 +0000 UTC m=+2730.887858290 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd-proxy-tls") pod "sklearn-v2-mlserver-predictor-65d8664766-llw97" (UID: "0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd") : secret "sklearn-v2-mlserver-predictor-serving-cert" not found
May 26 16:46:53.912291 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:53.912272    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-65d8664766-llw97\" (UID: \"0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-llw97"
May 26 16:46:53.912508 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:53.912490    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd-sklearn-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-65d8664766-llw97\" (UID: \"0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-llw97"
May 26 16:46:53.920776 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:53.920753    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btncg\" (UniqueName: \"kubernetes.io/projected/0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd-kube-api-access-btncg\") pod \"sklearn-v2-mlserver-predictor-65d8664766-llw97\" (UID: \"0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-llw97"
May 26 16:46:53.993764 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:53.993731    2572 generic.go:358] "Generic (PLEG): container finished" podID="d15d642a-e575-44e1-894d-1a58c2e1e283" containerID="a95d81d9a23e0cd4603a4bc286f34bfa23e41466af3d93a58cb22e719835d299" exitCode=2
May 26 16:46:53.993943 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:53.993802    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-k9h2x" event={"ID":"d15d642a-e575-44e1-894d-1a58c2e1e283","Type":"ContainerDied","Data":"a95d81d9a23e0cd4603a4bc286f34bfa23e41466af3d93a58cb22e719835d299"}
May 26 16:46:54.417596 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:54.417569    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-65d8664766-llw97\" (UID: \"0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-llw97"
May 26 16:46:54.419886 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:54.419867    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-65d8664766-llw97\" (UID: \"0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-llw97"
May 26 16:46:54.595362 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:54.595324    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-llw97"
May 26 16:46:54.722353 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:54.722323    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-llw97"]
May 26 16:46:54.725550 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:46:54.725518    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e92609a_9092_4e0a_9e3b_e1ad31d0dfdd.slice/crio-66db086d50125f898b42ba1d64484c693f0e22f4c2afa1d3ca6f70427ae83996 WatchSource:0}: Error finding container 66db086d50125f898b42ba1d64484c693f0e22f4c2afa1d3ca6f70427ae83996: Status 404 returned error can't find the container with id 66db086d50125f898b42ba1d64484c693f0e22f4c2afa1d3ca6f70427ae83996
May 26 16:46:54.998550 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:54.998460    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-llw97" event={"ID":"0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd","Type":"ContainerStarted","Data":"c9045b7275f84a6e800e8669130be1e66941a9bd43956dda1e80dcdf312fbc76"}
May 26 16:46:54.998550 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:54.998507    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-llw97" event={"ID":"0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd","Type":"ContainerStarted","Data":"66db086d50125f898b42ba1d64484c693f0e22f4c2afa1d3ca6f70427ae83996"}
May 26 16:46:55.796317 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:55.796257    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-k9h2x" podUID="d15d642a-e575-44e1-894d-1a58c2e1e283" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.46:8643/healthz\": dial tcp 10.132.0.46:8643: connect: connection refused"
May 26 16:46:55.801004 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:55.800965    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-k9h2x" podUID="d15d642a-e575-44e1-894d-1a58c2e1e283" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
May 26 16:46:58.012422 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:58.012386    2572 generic.go:358] "Generic (PLEG): container finished" podID="d15d642a-e575-44e1-894d-1a58c2e1e283" containerID="f057f3875dd20a6bbc4ff5e2726d28cdf5c264cf6e0a67ce31d75bef6833d0b8" exitCode=0
May 26 16:46:58.012803 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:58.012453    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-k9h2x" event={"ID":"d15d642a-e575-44e1-894d-1a58c2e1e283","Type":"ContainerDied","Data":"f057f3875dd20a6bbc4ff5e2726d28cdf5c264cf6e0a67ce31d75bef6833d0b8"}
May 26 16:46:58.034010 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:58.033989    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-k9h2x"
May 26 16:46:58.146526 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:58.146489    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d15d642a-e575-44e1-894d-1a58c2e1e283-proxy-tls\") pod \"d15d642a-e575-44e1-894d-1a58c2e1e283\" (UID: \"d15d642a-e575-44e1-894d-1a58c2e1e283\") "
May 26 16:46:58.146753 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:58.146568    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2h9rh\" (UniqueName: \"kubernetes.io/projected/d15d642a-e575-44e1-894d-1a58c2e1e283-kube-api-access-2h9rh\") pod \"d15d642a-e575-44e1-894d-1a58c2e1e283\" (UID: \"d15d642a-e575-44e1-894d-1a58c2e1e283\") "
May 26 16:46:58.146753 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:58.146596    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d15d642a-e575-44e1-894d-1a58c2e1e283-kserve-provision-location\") pod \"d15d642a-e575-44e1-894d-1a58c2e1e283\" (UID: \"d15d642a-e575-44e1-894d-1a58c2e1e283\") "
May 26 16:46:58.146753 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:58.146687    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d15d642a-e575-44e1-894d-1a58c2e1e283-isvc-sklearn-kube-rbac-proxy-sar-config\") pod \"d15d642a-e575-44e1-894d-1a58c2e1e283\" (UID: \"d15d642a-e575-44e1-894d-1a58c2e1e283\") "
May 26 16:46:58.146970 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:58.146940    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d15d642a-e575-44e1-894d-1a58c2e1e283-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d15d642a-e575-44e1-894d-1a58c2e1e283" (UID: "d15d642a-e575-44e1-894d-1a58c2e1e283"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:46:58.147069 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:58.147044    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d15d642a-e575-44e1-894d-1a58c2e1e283-isvc-sklearn-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-kube-rbac-proxy-sar-config") pod "d15d642a-e575-44e1-894d-1a58c2e1e283" (UID: "d15d642a-e575-44e1-894d-1a58c2e1e283"). InnerVolumeSpecName "isvc-sklearn-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:46:58.148651 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:58.148631    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d15d642a-e575-44e1-894d-1a58c2e1e283-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d15d642a-e575-44e1-894d-1a58c2e1e283" (UID: "d15d642a-e575-44e1-894d-1a58c2e1e283"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:46:58.148751 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:58.148687    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d15d642a-e575-44e1-894d-1a58c2e1e283-kube-api-access-2h9rh" (OuterVolumeSpecName: "kube-api-access-2h9rh") pod "d15d642a-e575-44e1-894d-1a58c2e1e283" (UID: "d15d642a-e575-44e1-894d-1a58c2e1e283"). InnerVolumeSpecName "kube-api-access-2h9rh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:46:58.247199 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:58.247162    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d15d642a-e575-44e1-894d-1a58c2e1e283-proxy-tls\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:46:58.247199 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:58.247191    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h9rh\" (UniqueName: \"kubernetes.io/projected/d15d642a-e575-44e1-894d-1a58c2e1e283-kube-api-access-2h9rh\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:46:58.247199 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:58.247201    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d15d642a-e575-44e1-894d-1a58c2e1e283-kserve-provision-location\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:46:58.247425 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:58.247211    2572 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d15d642a-e575-44e1-894d-1a58c2e1e283-isvc-sklearn-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:46:59.016920 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:59.016888    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-k9h2x" event={"ID":"d15d642a-e575-44e1-894d-1a58c2e1e283","Type":"ContainerDied","Data":"180b0d3d45b76d5c7844edd5c8fd4f7195898db0db20d31b7976e38ad5c018ad"}
May 26 16:46:59.017324 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:59.016930    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-k9h2x"
May 26 16:46:59.017324 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:59.016937    2572 scope.go:117] "RemoveContainer" containerID="a95d81d9a23e0cd4603a4bc286f34bfa23e41466af3d93a58cb22e719835d299"
May 26 16:46:59.018325 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:59.018305    2572 generic.go:358] "Generic (PLEG): container finished" podID="0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd" containerID="c9045b7275f84a6e800e8669130be1e66941a9bd43956dda1e80dcdf312fbc76" exitCode=0
May 26 16:46:59.018445 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:59.018383    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-llw97" event={"ID":"0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd","Type":"ContainerDied","Data":"c9045b7275f84a6e800e8669130be1e66941a9bd43956dda1e80dcdf312fbc76"}
May 26 16:46:59.025526 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:59.025505    2572 scope.go:117] "RemoveContainer" containerID="f057f3875dd20a6bbc4ff5e2726d28cdf5c264cf6e0a67ce31d75bef6833d0b8"
May 26 16:46:59.033111 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:59.033089    2572 scope.go:117] "RemoveContainer" containerID="ffdfe770fb1c6903c3f051338587ab62201c88ffa1e64f07ca458ef0b222b3a4"
May 26 16:46:59.053888 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:59.053846    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-k9h2x"]
May 26 16:46:59.058065 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:46:59.058038    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-bc9cf9b57-k9h2x"]
May 26 16:47:00.023347 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:47:00.023311    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-llw97" event={"ID":"0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd","Type":"ContainerStarted","Data":"981aa210ca9dc558b76fb828ae1d1e59f56dc1dae87566e76fe3d85b5778fb38"}
May 26 16:47:00.023347 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:47:00.023351    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-llw97" event={"ID":"0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd","Type":"ContainerStarted","Data":"eec9f41306e5f73755612cd268bd74450b099a9492025895f93dc1038b40382f"}
May 26 16:47:00.023831 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:47:00.023638    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-llw97"
May 26 16:47:00.023831 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:47:00.023675    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-llw97"
May 26 16:47:00.041176 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:47:00.041120    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-llw97" podStartSLOduration=7.041104133 podStartE2EDuration="7.041104133s" podCreationTimestamp="2026-05-26 16:46:53 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:47:00.039973903 +0000 UTC m=+2736.515701254" watchObservedRunningTime="2026-05-26 16:47:00.041104133 +0000 UTC m=+2736.516831485"
May 26 16:47:00.078871 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:47:00.078831    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d15d642a-e575-44e1-894d-1a58c2e1e283" path="/var/lib/kubelet/pods/d15d642a-e575-44e1-894d-1a58c2e1e283/volumes"
May 26 16:47:06.032774 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:47:06.032743    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-llw97"
May 26 16:47:36.036915 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:47:36.036873    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-llw97" podUID="0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
May 26 16:47:46.035788 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:47:46.035705    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-llw97"
May 26 16:47:53.836832 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:47:53.836799    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-llw97"]
May 26 16:47:53.837210 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:47:53.837118    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-llw97" podUID="0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd" containerName="kserve-container" containerID="cri-o://eec9f41306e5f73755612cd268bd74450b099a9492025895f93dc1038b40382f" gracePeriod=30
May 26 16:47:53.837275 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:47:53.837181    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-llw97" podUID="0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd" containerName="kube-rbac-proxy" containerID="cri-o://981aa210ca9dc558b76fb828ae1d1e59f56dc1dae87566e76fe3d85b5778fb38" gracePeriod=30
May 26 16:47:53.878356 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:47:53.878325    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl"]
May 26 16:47:53.878632 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:47:53.878595    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d15d642a-e575-44e1-894d-1a58c2e1e283" containerName="storage-initializer"
May 26 16:47:53.878632 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:47:53.878624    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d15d642a-e575-44e1-894d-1a58c2e1e283" containerName="storage-initializer"
May 26 16:47:53.878725 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:47:53.878637    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d15d642a-e575-44e1-894d-1a58c2e1e283" containerName="kube-rbac-proxy"
May 26 16:47:53.878725 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:47:53.878646    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d15d642a-e575-44e1-894d-1a58c2e1e283" containerName="kube-rbac-proxy"
May 26 16:47:53.878725 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:47:53.878662    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d15d642a-e575-44e1-894d-1a58c2e1e283" containerName="kserve-container"
May 26 16:47:53.878725 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:47:53.878667    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d15d642a-e575-44e1-894d-1a58c2e1e283" containerName="kserve-container"
May 26 16:47:53.878725 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:47:53.878710    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="d15d642a-e575-44e1-894d-1a58c2e1e283" containerName="kserve-container"
May 26 16:47:53.878725 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:47:53.878718    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="d15d642a-e575-44e1-894d-1a58c2e1e283" containerName="kube-rbac-proxy"
May 26 16:47:53.881853 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:47:53.881833    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl"
May 26 16:47:53.884086 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:47:53.884061    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
May 26 16:47:53.884216 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:47:53.884066    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
May 26 16:47:53.890547 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:47:53.890523    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl"]
May 26 16:47:53.959864 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:47:53.959827    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af4961cd-6443-4ccc-94e6-b688a9dfbf15-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl\" (UID: \"af4961cd-6443-4ccc-94e6-b688a9dfbf15\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl"
May 26 16:47:53.959864 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:47:53.959862    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzm99\" (UniqueName: \"kubernetes.io/projected/af4961cd-6443-4ccc-94e6-b688a9dfbf15-kube-api-access-bzm99\") pod \"isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl\" (UID: \"af4961cd-6443-4ccc-94e6-b688a9dfbf15\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl"
May 26 16:47:53.960056 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:47:53.959955    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af4961cd-6443-4ccc-94e6-b688a9dfbf15-isvc-sklearn-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl\" (UID: \"af4961cd-6443-4ccc-94e6-b688a9dfbf15\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl"
May 26 16:47:53.960056 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:47:53.959989    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af4961cd-6443-4ccc-94e6-b688a9dfbf15-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl\" (UID: \"af4961cd-6443-4ccc-94e6-b688a9dfbf15\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl"
May 26 16:47:54.061246 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:47:54.061200    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af4961cd-6443-4ccc-94e6-b688a9dfbf15-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl\" (UID: \"af4961cd-6443-4ccc-94e6-b688a9dfbf15\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl"
May 26 16:47:54.061246 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:47:54.061243    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bzm99\" (UniqueName: \"kubernetes.io/projected/af4961cd-6443-4ccc-94e6-b688a9dfbf15-kube-api-access-bzm99\") pod \"isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl\" (UID: \"af4961cd-6443-4ccc-94e6-b688a9dfbf15\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl"
May 26 16:47:54.061499 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:47:54.061291    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af4961cd-6443-4ccc-94e6-b688a9dfbf15-isvc-sklearn-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl\" (UID: \"af4961cd-6443-4ccc-94e6-b688a9dfbf15\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl"
May 26 16:47:54.061499 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:47:54.061308    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af4961cd-6443-4ccc-94e6-b688a9dfbf15-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl\" (UID: \"af4961cd-6443-4ccc-94e6-b688a9dfbf15\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl"
May 26 16:47:54.061499 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:47:54.061364    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-serving-cert: secret "isvc-sklearn-runtime-predictor-serving-cert" not found
May 26 16:47:54.061499 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:47:54.061438    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af4961cd-6443-4ccc-94e6-b688a9dfbf15-proxy-tls podName:af4961cd-6443-4ccc-94e6-b688a9dfbf15 nodeName:}" failed. No retries permitted until 2026-05-26 16:47:54.561421222 +0000 UTC m=+2791.037148554 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/af4961cd-6443-4ccc-94e6-b688a9dfbf15-proxy-tls") pod "isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl" (UID: "af4961cd-6443-4ccc-94e6-b688a9dfbf15") : secret "isvc-sklearn-runtime-predictor-serving-cert" not found
May 26 16:47:54.061836 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:47:54.061816    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af4961cd-6443-4ccc-94e6-b688a9dfbf15-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl\" (UID: \"af4961cd-6443-4ccc-94e6-b688a9dfbf15\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl"
May 26 16:47:54.062060 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:47:54.062043    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af4961cd-6443-4ccc-94e6-b688a9dfbf15-isvc-sklearn-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl\" (UID: \"af4961cd-6443-4ccc-94e6-b688a9dfbf15\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl"
May 26 16:47:54.072034 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:47:54.072003    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzm99\" (UniqueName: \"kubernetes.io/projected/af4961cd-6443-4ccc-94e6-b688a9dfbf15-kube-api-access-bzm99\") pod \"isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl\" (UID: \"af4961cd-6443-4ccc-94e6-b688a9dfbf15\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl"
May 26 16:47:54.172135 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:47:54.172100    2572 generic.go:358] "Generic (PLEG): container finished" podID="0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd" containerID="981aa210ca9dc558b76fb828ae1d1e59f56dc1dae87566e76fe3d85b5778fb38" exitCode=2
May 26 16:47:54.172315 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:47:54.172166    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-llw97" event={"ID":"0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd","Type":"ContainerDied","Data":"981aa210ca9dc558b76fb828ae1d1e59f56dc1dae87566e76fe3d85b5778fb38"}
May 26 16:47:54.564938 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:47:54.564853    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af4961cd-6443-4ccc-94e6-b688a9dfbf15-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl\" (UID: \"af4961cd-6443-4ccc-94e6-b688a9dfbf15\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl"
May 26 16:47:54.567240 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:47:54.567216    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af4961cd-6443-4ccc-94e6-b688a9dfbf15-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl\" (UID: \"af4961cd-6443-4ccc-94e6-b688a9dfbf15\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl"
May 26 16:47:54.793147 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:47:54.793111    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl"
May 26 16:47:54.921834 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:47:54.921807    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl"]
May 26 16:47:54.924622 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:47:54.924570    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf4961cd_6443_4ccc_94e6_b688a9dfbf15.slice/crio-a1e1e7031a224ea97e68345e1cf68b8fa526029382772939a8340258162d3d3e WatchSource:0}: Error finding container a1e1e7031a224ea97e68345e1cf68b8fa526029382772939a8340258162d3d3e: Status 404 returned error can't find the container with id a1e1e7031a224ea97e68345e1cf68b8fa526029382772939a8340258162d3d3e
May 26 16:47:55.177132 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:47:55.177095    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl" event={"ID":"af4961cd-6443-4ccc-94e6-b688a9dfbf15","Type":"ContainerStarted","Data":"c7f6e207dc8cf2ba336a2cff0cefc4120619f3c34b429093eb1b06d1e111218f"}
May 26 16:47:55.177132 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:47:55.177134    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl" event={"ID":"af4961cd-6443-4ccc-94e6-b688a9dfbf15","Type":"ContainerStarted","Data":"a1e1e7031a224ea97e68345e1cf68b8fa526029382772939a8340258162d3d3e"}
May 26 16:47:56.026955 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:47:56.026909    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-llw97" podUID="0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
May 26 16:47:57.073804 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:47:57.073755    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-llw97" podUID="0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.47:8080/v2/models/sklearn-v2-mlserver/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
May 26 16:48:01.027317 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:01.027267    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-llw97" podUID="0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
May 26 16:48:01.192999 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:01.192967    2572 generic.go:358] "Generic (PLEG): container finished" podID="af4961cd-6443-4ccc-94e6-b688a9dfbf15" containerID="c7f6e207dc8cf2ba336a2cff0cefc4120619f3c34b429093eb1b06d1e111218f" exitCode=0
May 26 16:48:01.193162 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:01.193046    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl" event={"ID":"af4961cd-6443-4ccc-94e6-b688a9dfbf15","Type":"ContainerDied","Data":"c7f6e207dc8cf2ba336a2cff0cefc4120619f3c34b429093eb1b06d1e111218f"}
May 26 16:48:01.680879 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:01.680854    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-llw97"
May 26 16:48:01.716843 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:01.716813    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-btncg\" (UniqueName: \"kubernetes.io/projected/0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd-kube-api-access-btncg\") pod \"0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd\" (UID: \"0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd\") "
May 26 16:48:01.717027 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:01.716853    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd-sklearn-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd\" (UID: \"0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd\") "
May 26 16:48:01.717027 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:01.716900    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd-kserve-provision-location\") pod \"0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd\" (UID: \"0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd\") "
May 26 16:48:01.717027 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:01.716947    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd-proxy-tls\") pod \"0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd\" (UID: \"0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd\") "
May 26 16:48:01.717266 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:01.717231    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd" (UID: "0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:48:01.717266 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:01.717253    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd-sklearn-v2-mlserver-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "sklearn-v2-mlserver-kube-rbac-proxy-sar-config") pod "0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd" (UID: "0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd"). InnerVolumeSpecName "sklearn-v2-mlserver-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:48:01.719267 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:01.719241    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd-kube-api-access-btncg" (OuterVolumeSpecName: "kube-api-access-btncg") pod "0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd" (UID: "0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd"). InnerVolumeSpecName "kube-api-access-btncg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:48:01.719408 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:01.719332    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd" (UID: "0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:48:01.817963 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:01.817862    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btncg\" (UniqueName: \"kubernetes.io/projected/0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd-kube-api-access-btncg\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:48:01.817963 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:01.817903    2572 reconciler_common.go:299] "Volume detached for volume \"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd-sklearn-v2-mlserver-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:48:01.817963 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:01.817924    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd-kserve-provision-location\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:48:01.817963 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:01.817938    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd-proxy-tls\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:48:02.197184 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:02.197145    2572 generic.go:358] "Generic (PLEG): container finished" podID="0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd" containerID="eec9f41306e5f73755612cd268bd74450b099a9492025895f93dc1038b40382f" exitCode=0
May 26 16:48:02.197689 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:02.197239    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-llw97"
May 26 16:48:02.197689 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:02.197236    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-llw97" event={"ID":"0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd","Type":"ContainerDied","Data":"eec9f41306e5f73755612cd268bd74450b099a9492025895f93dc1038b40382f"}
May 26 16:48:02.197689 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:02.197396    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-llw97" event={"ID":"0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd","Type":"ContainerDied","Data":"66db086d50125f898b42ba1d64484c693f0e22f4c2afa1d3ca6f70427ae83996"}
May 26 16:48:02.197689 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:02.197421    2572 scope.go:117] "RemoveContainer" containerID="981aa210ca9dc558b76fb828ae1d1e59f56dc1dae87566e76fe3d85b5778fb38"
May 26 16:48:02.199708 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:02.199631    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl" event={"ID":"af4961cd-6443-4ccc-94e6-b688a9dfbf15","Type":"ContainerStarted","Data":"1aecfd037b9c870b453f6e021b7243e286972c9fdf37cc568babc9195f282cc0"}
May 26 16:48:02.199708 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:02.199676    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl" event={"ID":"af4961cd-6443-4ccc-94e6-b688a9dfbf15","Type":"ContainerStarted","Data":"dc2a1604fdfd383f0a9bf1797bdb0fb5b7a39e7d27eda205178b9ef553ea3ad1"}
May 26 16:48:02.200090 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:02.200072    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl"
May 26 16:48:02.200191 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:02.200100    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl"
May 26 16:48:02.201576 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:02.201543    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl" podUID="af4961cd-6443-4ccc-94e6-b688a9dfbf15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
May 26 16:48:02.206018 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:02.206001    2572 scope.go:117] "RemoveContainer" containerID="eec9f41306e5f73755612cd268bd74450b099a9492025895f93dc1038b40382f"
May 26 16:48:02.213064 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:02.213040    2572 scope.go:117] "RemoveContainer" containerID="c9045b7275f84a6e800e8669130be1e66941a9bd43956dda1e80dcdf312fbc76"
May 26 16:48:02.214465 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:02.214428    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-llw97"]
May 26 16:48:02.219133 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:02.219105    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-llw97"]
May 26 16:48:02.222166 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:02.222126    2572 scope.go:117] "RemoveContainer" containerID="981aa210ca9dc558b76fb828ae1d1e59f56dc1dae87566e76fe3d85b5778fb38"
May 26 16:48:02.222484 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:48:02.222464    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"981aa210ca9dc558b76fb828ae1d1e59f56dc1dae87566e76fe3d85b5778fb38\": container with ID starting with 981aa210ca9dc558b76fb828ae1d1e59f56dc1dae87566e76fe3d85b5778fb38 not found: ID does not exist" containerID="981aa210ca9dc558b76fb828ae1d1e59f56dc1dae87566e76fe3d85b5778fb38"
May 26 16:48:02.222564 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:02.222492    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"981aa210ca9dc558b76fb828ae1d1e59f56dc1dae87566e76fe3d85b5778fb38"} err="failed to get container status \"981aa210ca9dc558b76fb828ae1d1e59f56dc1dae87566e76fe3d85b5778fb38\": rpc error: code = NotFound desc = could not find container \"981aa210ca9dc558b76fb828ae1d1e59f56dc1dae87566e76fe3d85b5778fb38\": container with ID starting with 981aa210ca9dc558b76fb828ae1d1e59f56dc1dae87566e76fe3d85b5778fb38 not found: ID does not exist"
May 26 16:48:02.222564 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:02.222513    2572 scope.go:117] "RemoveContainer" containerID="eec9f41306e5f73755612cd268bd74450b099a9492025895f93dc1038b40382f"
May 26 16:48:02.222822 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:48:02.222803    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eec9f41306e5f73755612cd268bd74450b099a9492025895f93dc1038b40382f\": container with ID starting with eec9f41306e5f73755612cd268bd74450b099a9492025895f93dc1038b40382f not found: ID does not exist" containerID="eec9f41306e5f73755612cd268bd74450b099a9492025895f93dc1038b40382f"
May 26 16:48:02.222889 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:02.222832    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eec9f41306e5f73755612cd268bd74450b099a9492025895f93dc1038b40382f"} err="failed to get container status \"eec9f41306e5f73755612cd268bd74450b099a9492025895f93dc1038b40382f\": rpc error: code = NotFound desc = could not find container \"eec9f41306e5f73755612cd268bd74450b099a9492025895f93dc1038b40382f\": container with ID starting with eec9f41306e5f73755612cd268bd74450b099a9492025895f93dc1038b40382f not found: ID does not exist"
May 26 16:48:02.222889 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:02.222856    2572 scope.go:117] "RemoveContainer" containerID="c9045b7275f84a6e800e8669130be1e66941a9bd43956dda1e80dcdf312fbc76"
May 26 16:48:02.223129 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:48:02.223114    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9045b7275f84a6e800e8669130be1e66941a9bd43956dda1e80dcdf312fbc76\": container with ID starting with c9045b7275f84a6e800e8669130be1e66941a9bd43956dda1e80dcdf312fbc76 not found: ID does not exist" containerID="c9045b7275f84a6e800e8669130be1e66941a9bd43956dda1e80dcdf312fbc76"
May 26 16:48:02.223181 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:02.223136    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9045b7275f84a6e800e8669130be1e66941a9bd43956dda1e80dcdf312fbc76"} err="failed to get container status \"c9045b7275f84a6e800e8669130be1e66941a9bd43956dda1e80dcdf312fbc76\": rpc error: code = NotFound desc = could not find container \"c9045b7275f84a6e800e8669130be1e66941a9bd43956dda1e80dcdf312fbc76\": container with ID starting with c9045b7275f84a6e800e8669130be1e66941a9bd43956dda1e80dcdf312fbc76 not found: ID does not exist"
May 26 16:48:02.236786 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:02.236735    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl" podStartSLOduration=9.236719304 podStartE2EDuration="9.236719304s" podCreationTimestamp="2026-05-26 16:47:53 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:48:02.235173266 +0000 UTC m=+2798.710900617" watchObservedRunningTime="2026-05-26 16:48:02.236719304 +0000 UTC m=+2798.712446654"
May 26 16:48:03.203828 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:03.203778    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl" podUID="af4961cd-6443-4ccc-94e6-b688a9dfbf15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
May 26 16:48:04.079128 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:04.079093    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd" path="/var/lib/kubelet/pods/0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd/volumes"
May 26 16:48:08.208212 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:08.208174    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl"
May 26 16:48:08.208847 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:08.208822    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl" podUID="af4961cd-6443-4ccc-94e6-b688a9dfbf15" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
May 26 16:48:18.209254 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:18.209222    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl"
May 26 16:48:30.755912 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:30.755878    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl_af4961cd-6443-4ccc-94e6-b688a9dfbf15/kserve-container/0.log"
May 26 16:48:30.888978 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:30.888940    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl"]
May 26 16:48:30.889352 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:30.889299    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl" podUID="af4961cd-6443-4ccc-94e6-b688a9dfbf15" containerName="kserve-container" containerID="cri-o://dc2a1604fdfd383f0a9bf1797bdb0fb5b7a39e7d27eda205178b9ef553ea3ad1" gracePeriod=30
May 26 16:48:30.889509 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:30.889340    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl" podUID="af4961cd-6443-4ccc-94e6-b688a9dfbf15" containerName="kube-rbac-proxy" containerID="cri-o://1aecfd037b9c870b453f6e021b7243e286972c9fdf37cc568babc9195f282cc0" gracePeriod=30
May 26 16:48:30.981702 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:30.981666    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf"]
May 26 16:48:30.982039 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:30.982021    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd" containerName="storage-initializer"
May 26 16:48:30.982134 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:30.982042    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd" containerName="storage-initializer"
May 26 16:48:30.982134 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:30.982061    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd" containerName="kube-rbac-proxy"
May 26 16:48:30.982134 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:30.982070    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd" containerName="kube-rbac-proxy"
May 26 16:48:30.982134 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:30.982081    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd" containerName="kserve-container"
May 26 16:48:30.982134 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:30.982092    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd" containerName="kserve-container"
May 26 16:48:30.982389 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:30.982151    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd" containerName="kserve-container"
May 26 16:48:30.982389 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:30.982168    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="0e92609a-9092-4e0a-9e3b-e1ad31d0dfdd" containerName="kube-rbac-proxy"
May 26 16:48:30.985038 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:30.985017    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf"
May 26 16:48:30.987151 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:30.987131    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
May 26 16:48:30.987269 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:30.987133    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
May 26 16:48:30.997732 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:30.997708    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf"]
May 26 16:48:31.049554 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:31.049512    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0bebfcbc-8019-4fda-9e1e-5cf33fa0194d-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf\" (UID: \"0bebfcbc-8019-4fda-9e1e-5cf33fa0194d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf"
May 26 16:48:31.049740 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:31.049572    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4k9c\" (UniqueName: \"kubernetes.io/projected/0bebfcbc-8019-4fda-9e1e-5cf33fa0194d-kube-api-access-n4k9c\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf\" (UID: \"0bebfcbc-8019-4fda-9e1e-5cf33fa0194d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf"
May 26 16:48:31.049740 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:31.049662    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0bebfcbc-8019-4fda-9e1e-5cf33fa0194d-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf\" (UID: \"0bebfcbc-8019-4fda-9e1e-5cf33fa0194d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf"
May 26 16:48:31.049740 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:31.049691    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0bebfcbc-8019-4fda-9e1e-5cf33fa0194d-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf\" (UID: \"0bebfcbc-8019-4fda-9e1e-5cf33fa0194d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf"
May 26 16:48:31.150282 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:31.150243    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0bebfcbc-8019-4fda-9e1e-5cf33fa0194d-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf\" (UID: \"0bebfcbc-8019-4fda-9e1e-5cf33fa0194d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf"
May 26 16:48:31.150470 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:31.150297    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0bebfcbc-8019-4fda-9e1e-5cf33fa0194d-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf\" (UID: \"0bebfcbc-8019-4fda-9e1e-5cf33fa0194d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf"
May 26 16:48:31.150470 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:31.150336    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0bebfcbc-8019-4fda-9e1e-5cf33fa0194d-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf\" (UID: \"0bebfcbc-8019-4fda-9e1e-5cf33fa0194d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf"
May 26 16:48:31.150470 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:31.150380    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-n4k9c\" (UniqueName: \"kubernetes.io/projected/0bebfcbc-8019-4fda-9e1e-5cf33fa0194d-kube-api-access-n4k9c\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf\" (UID: \"0bebfcbc-8019-4fda-9e1e-5cf33fa0194d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf"
May 26 16:48:31.150680 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:31.150657    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0bebfcbc-8019-4fda-9e1e-5cf33fa0194d-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf\" (UID: \"0bebfcbc-8019-4fda-9e1e-5cf33fa0194d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf"
May 26 16:48:31.150962 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:31.150944    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0bebfcbc-8019-4fda-9e1e-5cf33fa0194d-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf\" (UID: \"0bebfcbc-8019-4fda-9e1e-5cf33fa0194d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf"
May 26 16:48:31.152829 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:31.152805    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0bebfcbc-8019-4fda-9e1e-5cf33fa0194d-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf\" (UID: \"0bebfcbc-8019-4fda-9e1e-5cf33fa0194d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf"
May 26 16:48:31.158795 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:31.158771    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4k9c\" (UniqueName: \"kubernetes.io/projected/0bebfcbc-8019-4fda-9e1e-5cf33fa0194d-kube-api-access-n4k9c\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf\" (UID: \"0bebfcbc-8019-4fda-9e1e-5cf33fa0194d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf"
May 26 16:48:31.283002 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:31.282916    2572 generic.go:358] "Generic (PLEG): container finished" podID="af4961cd-6443-4ccc-94e6-b688a9dfbf15" containerID="1aecfd037b9c870b453f6e021b7243e286972c9fdf37cc568babc9195f282cc0" exitCode=2
May 26 16:48:31.283002 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:31.282980    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl" event={"ID":"af4961cd-6443-4ccc-94e6-b688a9dfbf15","Type":"ContainerDied","Data":"1aecfd037b9c870b453f6e021b7243e286972c9fdf37cc568babc9195f282cc0"}
May 26 16:48:31.295159 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:31.295127    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf"
May 26 16:48:31.432821 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:31.432686    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf"]
May 26 16:48:31.435380 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:48:31.435348    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bebfcbc_8019_4fda_9e1e_5cf33fa0194d.slice/crio-e9be94368c483ca4b082ea9d138e469900b9197bd2c8a3b888bf7edf6aa8f72b WatchSource:0}: Error finding container e9be94368c483ca4b082ea9d138e469900b9197bd2c8a3b888bf7edf6aa8f72b: Status 404 returned error can't find the container with id e9be94368c483ca4b082ea9d138e469900b9197bd2c8a3b888bf7edf6aa8f72b
May 26 16:48:31.950103 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:31.950081    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl"
May 26 16:48:32.057852 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:32.057736    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af4961cd-6443-4ccc-94e6-b688a9dfbf15-proxy-tls\") pod \"af4961cd-6443-4ccc-94e6-b688a9dfbf15\" (UID: \"af4961cd-6443-4ccc-94e6-b688a9dfbf15\") "
May 26 16:48:32.057852 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:32.057809    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bzm99\" (UniqueName: \"kubernetes.io/projected/af4961cd-6443-4ccc-94e6-b688a9dfbf15-kube-api-access-bzm99\") pod \"af4961cd-6443-4ccc-94e6-b688a9dfbf15\" (UID: \"af4961cd-6443-4ccc-94e6-b688a9dfbf15\") "
May 26 16:48:32.058097 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:32.057880    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af4961cd-6443-4ccc-94e6-b688a9dfbf15-isvc-sklearn-runtime-kube-rbac-proxy-sar-config\") pod \"af4961cd-6443-4ccc-94e6-b688a9dfbf15\" (UID: \"af4961cd-6443-4ccc-94e6-b688a9dfbf15\") "
May 26 16:48:32.058097 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:32.057913    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af4961cd-6443-4ccc-94e6-b688a9dfbf15-kserve-provision-location\") pod \"af4961cd-6443-4ccc-94e6-b688a9dfbf15\" (UID: \"af4961cd-6443-4ccc-94e6-b688a9dfbf15\") "
May 26 16:48:32.058248 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:32.058220    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af4961cd-6443-4ccc-94e6-b688a9dfbf15-isvc-sklearn-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-runtime-kube-rbac-proxy-sar-config") pod "af4961cd-6443-4ccc-94e6-b688a9dfbf15" (UID: "af4961cd-6443-4ccc-94e6-b688a9dfbf15"). InnerVolumeSpecName "isvc-sklearn-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:48:32.060005 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:32.059973    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af4961cd-6443-4ccc-94e6-b688a9dfbf15-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "af4961cd-6443-4ccc-94e6-b688a9dfbf15" (UID: "af4961cd-6443-4ccc-94e6-b688a9dfbf15"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:48:32.060005 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:32.059991    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af4961cd-6443-4ccc-94e6-b688a9dfbf15-kube-api-access-bzm99" (OuterVolumeSpecName: "kube-api-access-bzm99") pod "af4961cd-6443-4ccc-94e6-b688a9dfbf15" (UID: "af4961cd-6443-4ccc-94e6-b688a9dfbf15"). InnerVolumeSpecName "kube-api-access-bzm99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:48:32.083072 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:32.083040    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af4961cd-6443-4ccc-94e6-b688a9dfbf15-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "af4961cd-6443-4ccc-94e6-b688a9dfbf15" (UID: "af4961cd-6443-4ccc-94e6-b688a9dfbf15"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:48:32.158805 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:32.158771    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af4961cd-6443-4ccc-94e6-b688a9dfbf15-proxy-tls\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:48:32.158805 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:32.158801    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzm99\" (UniqueName: \"kubernetes.io/projected/af4961cd-6443-4ccc-94e6-b688a9dfbf15-kube-api-access-bzm99\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:48:32.158805 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:32.158811    2572 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af4961cd-6443-4ccc-94e6-b688a9dfbf15-isvc-sklearn-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:48:32.159019 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:32.158822    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af4961cd-6443-4ccc-94e6-b688a9dfbf15-kserve-provision-location\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:48:32.287366 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:32.287330    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf" event={"ID":"0bebfcbc-8019-4fda-9e1e-5cf33fa0194d","Type":"ContainerStarted","Data":"1cab6ff15ea1178be5564222cacf36532a251596e5b29287ce667f61bc7fab7b"}
May 26 16:48:32.287366 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:32.287370    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf" event={"ID":"0bebfcbc-8019-4fda-9e1e-5cf33fa0194d","Type":"ContainerStarted","Data":"e9be94368c483ca4b082ea9d138e469900b9197bd2c8a3b888bf7edf6aa8f72b"}
May 26 16:48:32.289005 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:32.288979    2572 generic.go:358] "Generic (PLEG): container finished" podID="af4961cd-6443-4ccc-94e6-b688a9dfbf15" containerID="dc2a1604fdfd383f0a9bf1797bdb0fb5b7a39e7d27eda205178b9ef553ea3ad1" exitCode=0
May 26 16:48:32.289127 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:32.289015    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl" event={"ID":"af4961cd-6443-4ccc-94e6-b688a9dfbf15","Type":"ContainerDied","Data":"dc2a1604fdfd383f0a9bf1797bdb0fb5b7a39e7d27eda205178b9ef553ea3ad1"}
May 26 16:48:32.289127 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:32.289034    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl" event={"ID":"af4961cd-6443-4ccc-94e6-b688a9dfbf15","Type":"ContainerDied","Data":"a1e1e7031a224ea97e68345e1cf68b8fa526029382772939a8340258162d3d3e"}
May 26 16:48:32.289127 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:32.289053    2572 scope.go:117] "RemoveContainer" containerID="1aecfd037b9c870b453f6e021b7243e286972c9fdf37cc568babc9195f282cc0"
May 26 16:48:32.289127 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:32.289093    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl"
May 26 16:48:32.296750 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:32.296729    2572 scope.go:117] "RemoveContainer" containerID="dc2a1604fdfd383f0a9bf1797bdb0fb5b7a39e7d27eda205178b9ef553ea3ad1"
May 26 16:48:32.303577 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:32.303518    2572 scope.go:117] "RemoveContainer" containerID="c7f6e207dc8cf2ba336a2cff0cefc4120619f3c34b429093eb1b06d1e111218f"
May 26 16:48:32.311647 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:32.311626    2572 scope.go:117] "RemoveContainer" containerID="1aecfd037b9c870b453f6e021b7243e286972c9fdf37cc568babc9195f282cc0"
May 26 16:48:32.311917 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:48:32.311899    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1aecfd037b9c870b453f6e021b7243e286972c9fdf37cc568babc9195f282cc0\": container with ID starting with 1aecfd037b9c870b453f6e021b7243e286972c9fdf37cc568babc9195f282cc0 not found: ID does not exist" containerID="1aecfd037b9c870b453f6e021b7243e286972c9fdf37cc568babc9195f282cc0"
May 26 16:48:32.311995 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:32.311928    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aecfd037b9c870b453f6e021b7243e286972c9fdf37cc568babc9195f282cc0"} err="failed to get container status \"1aecfd037b9c870b453f6e021b7243e286972c9fdf37cc568babc9195f282cc0\": rpc error: code = NotFound desc = could not find container \"1aecfd037b9c870b453f6e021b7243e286972c9fdf37cc568babc9195f282cc0\": container with ID starting with 1aecfd037b9c870b453f6e021b7243e286972c9fdf37cc568babc9195f282cc0 not found: ID does not exist"
May 26 16:48:32.311995 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:32.311948    2572 scope.go:117] "RemoveContainer" containerID="dc2a1604fdfd383f0a9bf1797bdb0fb5b7a39e7d27eda205178b9ef553ea3ad1"
May 26 16:48:32.312140 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:48:32.312125    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc2a1604fdfd383f0a9bf1797bdb0fb5b7a39e7d27eda205178b9ef553ea3ad1\": container with ID starting with dc2a1604fdfd383f0a9bf1797bdb0fb5b7a39e7d27eda205178b9ef553ea3ad1 not found: ID does not exist" containerID="dc2a1604fdfd383f0a9bf1797bdb0fb5b7a39e7d27eda205178b9ef553ea3ad1"
May 26 16:48:32.312181 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:32.312146    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc2a1604fdfd383f0a9bf1797bdb0fb5b7a39e7d27eda205178b9ef553ea3ad1"} err="failed to get container status \"dc2a1604fdfd383f0a9bf1797bdb0fb5b7a39e7d27eda205178b9ef553ea3ad1\": rpc error: code = NotFound desc = could not find container \"dc2a1604fdfd383f0a9bf1797bdb0fb5b7a39e7d27eda205178b9ef553ea3ad1\": container with ID starting with dc2a1604fdfd383f0a9bf1797bdb0fb5b7a39e7d27eda205178b9ef553ea3ad1 not found: ID does not exist"
May 26 16:48:32.312181 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:32.312159    2572 scope.go:117] "RemoveContainer" containerID="c7f6e207dc8cf2ba336a2cff0cefc4120619f3c34b429093eb1b06d1e111218f"
May 26 16:48:32.312405 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:48:32.312376    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7f6e207dc8cf2ba336a2cff0cefc4120619f3c34b429093eb1b06d1e111218f\": container with ID starting with c7f6e207dc8cf2ba336a2cff0cefc4120619f3c34b429093eb1b06d1e111218f not found: ID does not exist" containerID="c7f6e207dc8cf2ba336a2cff0cefc4120619f3c34b429093eb1b06d1e111218f"
May 26 16:48:32.312470 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:32.312405    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7f6e207dc8cf2ba336a2cff0cefc4120619f3c34b429093eb1b06d1e111218f"} err="failed to get container status \"c7f6e207dc8cf2ba336a2cff0cefc4120619f3c34b429093eb1b06d1e111218f\": rpc error: code = NotFound desc = could not find container \"c7f6e207dc8cf2ba336a2cff0cefc4120619f3c34b429093eb1b06d1e111218f\": container with ID starting with c7f6e207dc8cf2ba336a2cff0cefc4120619f3c34b429093eb1b06d1e111218f not found: ID does not exist"
May 26 16:48:32.315885 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:32.315864    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl"]
May 26 16:48:32.317881 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:32.317855    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56f6dfd9f9-nccwl"]
May 26 16:48:34.078619 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:34.078575    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af4961cd-6443-4ccc-94e6-b688a9dfbf15" path="/var/lib/kubelet/pods/af4961cd-6443-4ccc-94e6-b688a9dfbf15/volumes"
May 26 16:48:35.299874 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:35.299781    2572 generic.go:358] "Generic (PLEG): container finished" podID="0bebfcbc-8019-4fda-9e1e-5cf33fa0194d" containerID="1cab6ff15ea1178be5564222cacf36532a251596e5b29287ce667f61bc7fab7b" exitCode=0
May 26 16:48:35.299874 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:35.299857    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf" event={"ID":"0bebfcbc-8019-4fda-9e1e-5cf33fa0194d","Type":"ContainerDied","Data":"1cab6ff15ea1178be5564222cacf36532a251596e5b29287ce667f61bc7fab7b"}
May 26 16:48:36.303938 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:36.303901    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf" event={"ID":"0bebfcbc-8019-4fda-9e1e-5cf33fa0194d","Type":"ContainerStarted","Data":"c046add50ff5604817831292fe30eeb3f22820b4808a15d572294b16cb3c5e67"}
May 26 16:48:36.303938 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:36.303941    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf" event={"ID":"0bebfcbc-8019-4fda-9e1e-5cf33fa0194d","Type":"ContainerStarted","Data":"a307a16bce311498b8f3374e95a797ad79d23b65c78931b0447b5f04bc589fa9"}
May 26 16:48:36.304356 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:36.304151    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf"
May 26 16:48:36.304356 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:36.304270    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf"
May 26 16:48:36.324363 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:36.324312    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf" podStartSLOduration=6.324296756 podStartE2EDuration="6.324296756s" podCreationTimestamp="2026-05-26 16:48:30 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:48:36.322367048 +0000 UTC m=+2832.798094421" watchObservedRunningTime="2026-05-26 16:48:36.324296756 +0000 UTC m=+2832.800024106"
May 26 16:48:42.313263 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:48:42.313231    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf"
May 26 16:49:12.334734 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:12.334685    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf" podUID="0bebfcbc-8019-4fda-9e1e-5cf33fa0194d" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
May 26 16:49:22.316419 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:22.316339    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf"
May 26 16:49:31.036894 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:31.036846    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf"]
May 26 16:49:31.037388 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:31.037140    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf" podUID="0bebfcbc-8019-4fda-9e1e-5cf33fa0194d" containerName="kserve-container" containerID="cri-o://a307a16bce311498b8f3374e95a797ad79d23b65c78931b0447b5f04bc589fa9" gracePeriod=30
May 26 16:49:31.037388 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:31.037177    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf" podUID="0bebfcbc-8019-4fda-9e1e-5cf33fa0194d" containerName="kube-rbac-proxy" containerID="cri-o://c046add50ff5604817831292fe30eeb3f22820b4808a15d572294b16cb3c5e67" gracePeriod=30
May 26 16:49:31.116092 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:31.116058    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-862j2"]
May 26 16:49:31.116331 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:31.116319    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af4961cd-6443-4ccc-94e6-b688a9dfbf15" containerName="storage-initializer"
May 26 16:49:31.116393 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:31.116333    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="af4961cd-6443-4ccc-94e6-b688a9dfbf15" containerName="storage-initializer"
May 26 16:49:31.116393 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:31.116343    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af4961cd-6443-4ccc-94e6-b688a9dfbf15" containerName="kserve-container"
May 26 16:49:31.116393 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:31.116348    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="af4961cd-6443-4ccc-94e6-b688a9dfbf15" containerName="kserve-container"
May 26 16:49:31.116393 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:31.116358    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af4961cd-6443-4ccc-94e6-b688a9dfbf15" containerName="kube-rbac-proxy"
May 26 16:49:31.116393 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:31.116364    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="af4961cd-6443-4ccc-94e6-b688a9dfbf15" containerName="kube-rbac-proxy"
May 26 16:49:31.116567 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:31.116412    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="af4961cd-6443-4ccc-94e6-b688a9dfbf15" containerName="kube-rbac-proxy"
May 26 16:49:31.116567 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:31.116447    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="af4961cd-6443-4ccc-94e6-b688a9dfbf15" containerName="kserve-container"
May 26 16:49:31.120809 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:31.120788    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-862j2"
May 26 16:49:31.123062 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:31.123030    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-predictor-serving-cert\""
May 26 16:49:31.123196 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:31.123067    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
May 26 16:49:31.130891 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:31.130862    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-862j2"]
May 26 16:49:31.194708 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:31.194668    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx6l8\" (UniqueName: \"kubernetes.io/projected/3107d7e5-2bbf-491d-819b-f3fa592646f5-kube-api-access-tx6l8\") pod \"isvc-sklearn-v2-predictor-9df9c96c7-862j2\" (UID: \"3107d7e5-2bbf-491d-819b-f3fa592646f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-862j2"
May 26 16:49:31.194888 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:31.194716    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3107d7e5-2bbf-491d-819b-f3fa592646f5-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-9df9c96c7-862j2\" (UID: \"3107d7e5-2bbf-491d-819b-f3fa592646f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-862j2"
May 26 16:49:31.194888 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:31.194771    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3107d7e5-2bbf-491d-819b-f3fa592646f5-isvc-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-9df9c96c7-862j2\" (UID: \"3107d7e5-2bbf-491d-819b-f3fa592646f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-862j2"
May 26 16:49:31.194888 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:31.194804    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3107d7e5-2bbf-491d-819b-f3fa592646f5-proxy-tls\") pod \"isvc-sklearn-v2-predictor-9df9c96c7-862j2\" (UID: \"3107d7e5-2bbf-491d-819b-f3fa592646f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-862j2"
May 26 16:49:31.295966 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:31.295881    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tx6l8\" (UniqueName: \"kubernetes.io/projected/3107d7e5-2bbf-491d-819b-f3fa592646f5-kube-api-access-tx6l8\") pod \"isvc-sklearn-v2-predictor-9df9c96c7-862j2\" (UID: \"3107d7e5-2bbf-491d-819b-f3fa592646f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-862j2"
May 26 16:49:31.295966 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:31.295922    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3107d7e5-2bbf-491d-819b-f3fa592646f5-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-9df9c96c7-862j2\" (UID: \"3107d7e5-2bbf-491d-819b-f3fa592646f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-862j2"
May 26 16:49:31.296181 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:31.296051    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3107d7e5-2bbf-491d-819b-f3fa592646f5-isvc-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-9df9c96c7-862j2\" (UID: \"3107d7e5-2bbf-491d-819b-f3fa592646f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-862j2"
May 26 16:49:31.296181 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:31.296109    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3107d7e5-2bbf-491d-819b-f3fa592646f5-proxy-tls\") pod \"isvc-sklearn-v2-predictor-9df9c96c7-862j2\" (UID: \"3107d7e5-2bbf-491d-819b-f3fa592646f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-862j2"
May 26 16:49:31.296300 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:31.296246    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3107d7e5-2bbf-491d-819b-f3fa592646f5-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-9df9c96c7-862j2\" (UID: \"3107d7e5-2bbf-491d-819b-f3fa592646f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-862j2"
May 26 16:49:31.296665 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:31.296643    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3107d7e5-2bbf-491d-819b-f3fa592646f5-isvc-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-9df9c96c7-862j2\" (UID: \"3107d7e5-2bbf-491d-819b-f3fa592646f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-862j2"
May 26 16:49:31.298592 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:31.298572    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3107d7e5-2bbf-491d-819b-f3fa592646f5-proxy-tls\") pod \"isvc-sklearn-v2-predictor-9df9c96c7-862j2\" (UID: \"3107d7e5-2bbf-491d-819b-f3fa592646f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-862j2"
May 26 16:49:31.305204 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:31.305181    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx6l8\" (UniqueName: \"kubernetes.io/projected/3107d7e5-2bbf-491d-819b-f3fa592646f5-kube-api-access-tx6l8\") pod \"isvc-sklearn-v2-predictor-9df9c96c7-862j2\" (UID: \"3107d7e5-2bbf-491d-819b-f3fa592646f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-862j2"
May 26 16:49:31.432516 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:31.432475    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-862j2"
May 26 16:49:31.460674 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:31.460643    2572 generic.go:358] "Generic (PLEG): container finished" podID="0bebfcbc-8019-4fda-9e1e-5cf33fa0194d" containerID="c046add50ff5604817831292fe30eeb3f22820b4808a15d572294b16cb3c5e67" exitCode=2
May 26 16:49:31.460835 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:31.460714    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf" event={"ID":"0bebfcbc-8019-4fda-9e1e-5cf33fa0194d","Type":"ContainerDied","Data":"c046add50ff5604817831292fe30eeb3f22820b4808a15d572294b16cb3c5e67"}
May 26 16:49:31.562267 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:31.562239    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-862j2"]
May 26 16:49:31.564686 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:49:31.564657    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3107d7e5_2bbf_491d_819b_f3fa592646f5.slice/crio-0835d0f271b2e0b18750d9dab3a1fe6915211856b868ad4f2cc254c9db466207 WatchSource:0}: Error finding container 0835d0f271b2e0b18750d9dab3a1fe6915211856b868ad4f2cc254c9db466207: Status 404 returned error can't find the container with id 0835d0f271b2e0b18750d9dab3a1fe6915211856b868ad4f2cc254c9db466207
May 26 16:49:32.307627 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:32.307564    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf" podUID="0bebfcbc-8019-4fda-9e1e-5cf33fa0194d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.49:8643/healthz\": dial tcp 10.132.0.49:8643: connect: connection refused"
May 26 16:49:32.465029 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:32.464993    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-862j2" event={"ID":"3107d7e5-2bbf-491d-819b-f3fa592646f5","Type":"ContainerStarted","Data":"09a755b6557b084c521ba4fc8dcdfb80956f2f71ec2806a466127b4dd1cef3b4"}
May 26 16:49:32.465029 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:32.465033    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-862j2" event={"ID":"3107d7e5-2bbf-491d-819b-f3fa592646f5","Type":"ContainerStarted","Data":"0835d0f271b2e0b18750d9dab3a1fe6915211856b868ad4f2cc254c9db466207"}
May 26 16:49:33.354839 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:33.354751    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf" podUID="0bebfcbc-8019-4fda-9e1e-5cf33fa0194d" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.49:8080/v2/models/isvc-sklearn-v2-runtime/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
May 26 16:49:35.474297 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:35.474260    2572 generic.go:358] "Generic (PLEG): container finished" podID="3107d7e5-2bbf-491d-819b-f3fa592646f5" containerID="09a755b6557b084c521ba4fc8dcdfb80956f2f71ec2806a466127b4dd1cef3b4" exitCode=0
May 26 16:49:35.474702 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:35.474334    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-862j2" event={"ID":"3107d7e5-2bbf-491d-819b-f3fa592646f5","Type":"ContainerDied","Data":"09a755b6557b084c521ba4fc8dcdfb80956f2f71ec2806a466127b4dd1cef3b4"}
May 26 16:49:36.478927 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:36.478891    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-862j2" event={"ID":"3107d7e5-2bbf-491d-819b-f3fa592646f5","Type":"ContainerStarted","Data":"ae66c7199bb37827ca3f6688ed405c98351db250a47208b27185ee3194491dd3"}
May 26 16:49:36.478927 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:36.478929    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-862j2" event={"ID":"3107d7e5-2bbf-491d-819b-f3fa592646f5","Type":"ContainerStarted","Data":"c65f1574009d3218f3de9f954986658ed8177ecb15df303434d4600ae94ce592"}
May 26 16:49:36.479408 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:36.479212    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-862j2"
May 26 16:49:36.479408 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:36.479325    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-862j2"
May 26 16:49:36.480476 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:36.480456    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-862j2" podUID="3107d7e5-2bbf-491d-819b-f3fa592646f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
May 26 16:49:36.498834 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:36.498791    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-862j2" podStartSLOduration=5.498778958 podStartE2EDuration="5.498778958s" podCreationTimestamp="2026-05-26 16:49:31 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:49:36.497241038 +0000 UTC m=+2892.972968403" watchObservedRunningTime="2026-05-26 16:49:36.498778958 +0000 UTC m=+2892.974506369"
May 26 16:49:37.307725 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:37.307683    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf" podUID="0bebfcbc-8019-4fda-9e1e-5cf33fa0194d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.49:8643/healthz\": dial tcp 10.132.0.49:8643: connect: connection refused"
May 26 16:49:37.481425 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:37.481379    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-862j2" podUID="3107d7e5-2bbf-491d-819b-f3fa592646f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
May 26 16:49:38.672060 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:38.672036    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf"
May 26 16:49:38.758278 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:38.758194    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0bebfcbc-8019-4fda-9e1e-5cf33fa0194d-kserve-provision-location\") pod \"0bebfcbc-8019-4fda-9e1e-5cf33fa0194d\" (UID: \"0bebfcbc-8019-4fda-9e1e-5cf33fa0194d\") "
May 26 16:49:38.758278 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:38.758237    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0bebfcbc-8019-4fda-9e1e-5cf33fa0194d-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\") pod \"0bebfcbc-8019-4fda-9e1e-5cf33fa0194d\" (UID: \"0bebfcbc-8019-4fda-9e1e-5cf33fa0194d\") "
May 26 16:49:38.758278 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:38.758259    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0bebfcbc-8019-4fda-9e1e-5cf33fa0194d-proxy-tls\") pod \"0bebfcbc-8019-4fda-9e1e-5cf33fa0194d\" (UID: \"0bebfcbc-8019-4fda-9e1e-5cf33fa0194d\") "
May 26 16:49:38.758500 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:38.758326    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4k9c\" (UniqueName: \"kubernetes.io/projected/0bebfcbc-8019-4fda-9e1e-5cf33fa0194d-kube-api-access-n4k9c\") pod \"0bebfcbc-8019-4fda-9e1e-5cf33fa0194d\" (UID: \"0bebfcbc-8019-4fda-9e1e-5cf33fa0194d\") "
May 26 16:49:38.758582 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:38.758556    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bebfcbc-8019-4fda-9e1e-5cf33fa0194d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0bebfcbc-8019-4fda-9e1e-5cf33fa0194d" (UID: "0bebfcbc-8019-4fda-9e1e-5cf33fa0194d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:49:38.758652 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:38.758596    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bebfcbc-8019-4fda-9e1e-5cf33fa0194d-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config") pod "0bebfcbc-8019-4fda-9e1e-5cf33fa0194d" (UID: "0bebfcbc-8019-4fda-9e1e-5cf33fa0194d"). InnerVolumeSpecName "isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:49:38.760394 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:38.760363    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bebfcbc-8019-4fda-9e1e-5cf33fa0194d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0bebfcbc-8019-4fda-9e1e-5cf33fa0194d" (UID: "0bebfcbc-8019-4fda-9e1e-5cf33fa0194d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:49:38.760518 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:38.760435    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bebfcbc-8019-4fda-9e1e-5cf33fa0194d-kube-api-access-n4k9c" (OuterVolumeSpecName: "kube-api-access-n4k9c") pod "0bebfcbc-8019-4fda-9e1e-5cf33fa0194d" (UID: "0bebfcbc-8019-4fda-9e1e-5cf33fa0194d"). InnerVolumeSpecName "kube-api-access-n4k9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:49:38.859118 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:38.859080    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0bebfcbc-8019-4fda-9e1e-5cf33fa0194d-kserve-provision-location\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:49:38.859118 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:38.859112    2572 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0bebfcbc-8019-4fda-9e1e-5cf33fa0194d-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:49:38.859118 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:38.859124    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0bebfcbc-8019-4fda-9e1e-5cf33fa0194d-proxy-tls\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:49:38.859343 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:38.859135    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4k9c\" (UniqueName: \"kubernetes.io/projected/0bebfcbc-8019-4fda-9e1e-5cf33fa0194d-kube-api-access-n4k9c\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:49:39.487863 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:39.487822    2572 generic.go:358] "Generic (PLEG): container finished" podID="0bebfcbc-8019-4fda-9e1e-5cf33fa0194d" containerID="a307a16bce311498b8f3374e95a797ad79d23b65c78931b0447b5f04bc589fa9" exitCode=0
May 26 16:49:39.488039 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:39.487902    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf" event={"ID":"0bebfcbc-8019-4fda-9e1e-5cf33fa0194d","Type":"ContainerDied","Data":"a307a16bce311498b8f3374e95a797ad79d23b65c78931b0447b5f04bc589fa9"}
May 26 16:49:39.488039 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:39.487919    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf"
May 26 16:49:39.488039 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:39.487939    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf" event={"ID":"0bebfcbc-8019-4fda-9e1e-5cf33fa0194d","Type":"ContainerDied","Data":"e9be94368c483ca4b082ea9d138e469900b9197bd2c8a3b888bf7edf6aa8f72b"}
May 26 16:49:39.488039 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:39.487955    2572 scope.go:117] "RemoveContainer" containerID="c046add50ff5604817831292fe30eeb3f22820b4808a15d572294b16cb3c5e67"
May 26 16:49:39.495674 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:39.495593    2572 scope.go:117] "RemoveContainer" containerID="a307a16bce311498b8f3374e95a797ad79d23b65c78931b0447b5f04bc589fa9"
May 26 16:49:39.502650 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:39.502632    2572 scope.go:117] "RemoveContainer" containerID="1cab6ff15ea1178be5564222cacf36532a251596e5b29287ce667f61bc7fab7b"
May 26 16:49:39.509460 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:39.509435    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf"]
May 26 16:49:39.509871 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:39.509850    2572 scope.go:117] "RemoveContainer" containerID="c046add50ff5604817831292fe30eeb3f22820b4808a15d572294b16cb3c5e67"
May 26 16:49:39.510199 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:49:39.510174    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c046add50ff5604817831292fe30eeb3f22820b4808a15d572294b16cb3c5e67\": container with ID starting with c046add50ff5604817831292fe30eeb3f22820b4808a15d572294b16cb3c5e67 not found: ID does not exist" containerID="c046add50ff5604817831292fe30eeb3f22820b4808a15d572294b16cb3c5e67"
May 26 16:49:39.510262 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:39.510207    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c046add50ff5604817831292fe30eeb3f22820b4808a15d572294b16cb3c5e67"} err="failed to get container status \"c046add50ff5604817831292fe30eeb3f22820b4808a15d572294b16cb3c5e67\": rpc error: code = NotFound desc = could not find container \"c046add50ff5604817831292fe30eeb3f22820b4808a15d572294b16cb3c5e67\": container with ID starting with c046add50ff5604817831292fe30eeb3f22820b4808a15d572294b16cb3c5e67 not found: ID does not exist"
May 26 16:49:39.510262 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:39.510227    2572 scope.go:117] "RemoveContainer" containerID="a307a16bce311498b8f3374e95a797ad79d23b65c78931b0447b5f04bc589fa9"
May 26 16:49:39.510464 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:49:39.510444    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a307a16bce311498b8f3374e95a797ad79d23b65c78931b0447b5f04bc589fa9\": container with ID starting with a307a16bce311498b8f3374e95a797ad79d23b65c78931b0447b5f04bc589fa9 not found: ID does not exist" containerID="a307a16bce311498b8f3374e95a797ad79d23b65c78931b0447b5f04bc589fa9"
May 26 16:49:39.510507 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:39.510469    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a307a16bce311498b8f3374e95a797ad79d23b65c78931b0447b5f04bc589fa9"} err="failed to get container status \"a307a16bce311498b8f3374e95a797ad79d23b65c78931b0447b5f04bc589fa9\": rpc error: code = NotFound desc = could not find container \"a307a16bce311498b8f3374e95a797ad79d23b65c78931b0447b5f04bc589fa9\": container with ID starting with a307a16bce311498b8f3374e95a797ad79d23b65c78931b0447b5f04bc589fa9 not found: ID does not exist"
May 26 16:49:39.510507 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:39.510484    2572 scope.go:117] "RemoveContainer" containerID="1cab6ff15ea1178be5564222cacf36532a251596e5b29287ce667f61bc7fab7b"
May 26 16:49:39.510762 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:49:39.510744    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cab6ff15ea1178be5564222cacf36532a251596e5b29287ce667f61bc7fab7b\": container with ID starting with 1cab6ff15ea1178be5564222cacf36532a251596e5b29287ce667f61bc7fab7b not found: ID does not exist" containerID="1cab6ff15ea1178be5564222cacf36532a251596e5b29287ce667f61bc7fab7b"
May 26 16:49:39.510825 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:39.510769    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cab6ff15ea1178be5564222cacf36532a251596e5b29287ce667f61bc7fab7b"} err="failed to get container status \"1cab6ff15ea1178be5564222cacf36532a251596e5b29287ce667f61bc7fab7b\": rpc error: code = NotFound desc = could not find container \"1cab6ff15ea1178be5564222cacf36532a251596e5b29287ce667f61bc7fab7b\": container with ID starting with 1cab6ff15ea1178be5564222cacf36532a251596e5b29287ce667f61bc7fab7b not found: ID does not exist"
May 26 16:49:39.517771 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:39.517742    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-fl4vf"]
May 26 16:49:40.079791 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:40.079751    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bebfcbc-8019-4fda-9e1e-5cf33fa0194d" path="/var/lib/kubelet/pods/0bebfcbc-8019-4fda-9e1e-5cf33fa0194d/volumes"
May 26 16:49:42.485764 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:42.485736    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-862j2"
May 26 16:49:42.486275 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:42.486251    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-862j2" podUID="3107d7e5-2bbf-491d-819b-f3fa592646f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
May 26 16:49:52.486212 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:49:52.486171    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-862j2" podUID="3107d7e5-2bbf-491d-819b-f3fa592646f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
May 26 16:50:02.486753 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:02.486706    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-862j2" podUID="3107d7e5-2bbf-491d-819b-f3fa592646f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
May 26 16:50:12.486916 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:12.486871    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-862j2" podUID="3107d7e5-2bbf-491d-819b-f3fa592646f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
May 26 16:50:22.486533 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:22.486492    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-862j2" podUID="3107d7e5-2bbf-491d-819b-f3fa592646f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
May 26 16:50:32.486287 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:32.486248    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-862j2" podUID="3107d7e5-2bbf-491d-819b-f3fa592646f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
May 26 16:50:42.487559 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:42.487526    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-862j2"
May 26 16:50:51.287741 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:51.287702    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-862j2"]
May 26 16:50:51.288215 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:51.288145    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-862j2" podUID="3107d7e5-2bbf-491d-819b-f3fa592646f5" containerName="kserve-container" containerID="cri-o://c65f1574009d3218f3de9f954986658ed8177ecb15df303434d4600ae94ce592" gracePeriod=30
May 26 16:50:51.288293 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:51.288185    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-862j2" podUID="3107d7e5-2bbf-491d-819b-f3fa592646f5" containerName="kube-rbac-proxy" containerID="cri-o://ae66c7199bb37827ca3f6688ed405c98351db250a47208b27185ee3194491dd3" gracePeriod=30
May 26 16:50:51.395283 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:51.395250    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72"]
May 26 16:50:51.395519 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:51.395507    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bebfcbc-8019-4fda-9e1e-5cf33fa0194d" containerName="kserve-container"
May 26 16:50:51.395572 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:51.395521    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bebfcbc-8019-4fda-9e1e-5cf33fa0194d" containerName="kserve-container"
May 26 16:50:51.395572 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:51.395536    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bebfcbc-8019-4fda-9e1e-5cf33fa0194d" containerName="storage-initializer"
May 26 16:50:51.395572 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:51.395542    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bebfcbc-8019-4fda-9e1e-5cf33fa0194d" containerName="storage-initializer"
May 26 16:50:51.395572 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:51.395552    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bebfcbc-8019-4fda-9e1e-5cf33fa0194d" containerName="kube-rbac-proxy"
May 26 16:50:51.395572 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:51.395557    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bebfcbc-8019-4fda-9e1e-5cf33fa0194d" containerName="kube-rbac-proxy"
May 26 16:50:51.395762 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:51.395621    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="0bebfcbc-8019-4fda-9e1e-5cf33fa0194d" containerName="kserve-container"
May 26 16:50:51.395762 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:51.395630    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="0bebfcbc-8019-4fda-9e1e-5cf33fa0194d" containerName="kube-rbac-proxy"
May 26 16:50:51.398720 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:51.398698    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72"
May 26 16:50:51.400985 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:51.400966    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
May 26 16:50:51.401088 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:51.401030    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
May 26 16:50:51.405895 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:51.405865    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/96223991-68df-4e67-a4ed-77cdf6530f22-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72\" (UID: \"96223991-68df-4e67-a4ed-77cdf6530f22\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72"
May 26 16:50:51.406006 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:51.405925    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96223991-68df-4e67-a4ed-77cdf6530f22-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72\" (UID: \"96223991-68df-4e67-a4ed-77cdf6530f22\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72"
May 26 16:50:51.406006 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:51.405951    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96223991-68df-4e67-a4ed-77cdf6530f22-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72\" (UID: \"96223991-68df-4e67-a4ed-77cdf6530f22\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72"
May 26 16:50:51.406006 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:51.405973    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wkxx\" (UniqueName: \"kubernetes.io/projected/96223991-68df-4e67-a4ed-77cdf6530f22-kube-api-access-5wkxx\") pod \"isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72\" (UID: \"96223991-68df-4e67-a4ed-77cdf6530f22\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72"
May 26 16:50:51.412683 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:51.411631    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72"]
May 26 16:50:51.506310 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:51.506272    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96223991-68df-4e67-a4ed-77cdf6530f22-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72\" (UID: \"96223991-68df-4e67-a4ed-77cdf6530f22\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72"
May 26 16:50:51.506310 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:51.506316    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96223991-68df-4e67-a4ed-77cdf6530f22-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72\" (UID: \"96223991-68df-4e67-a4ed-77cdf6530f22\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72"
May 26 16:50:51.506578 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:51.506336    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5wkxx\" (UniqueName: \"kubernetes.io/projected/96223991-68df-4e67-a4ed-77cdf6530f22-kube-api-access-5wkxx\") pod \"isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72\" (UID: \"96223991-68df-4e67-a4ed-77cdf6530f22\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72"
May 26 16:50:51.506578 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:51.506386    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/96223991-68df-4e67-a4ed-77cdf6530f22-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72\" (UID: \"96223991-68df-4e67-a4ed-77cdf6530f22\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72"
May 26 16:50:51.506734 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:51.506710    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96223991-68df-4e67-a4ed-77cdf6530f22-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72\" (UID: \"96223991-68df-4e67-a4ed-77cdf6530f22\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72"
May 26 16:50:51.506979 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:51.506959    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/96223991-68df-4e67-a4ed-77cdf6530f22-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72\" (UID: \"96223991-68df-4e67-a4ed-77cdf6530f22\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72"
May 26 16:50:51.508834 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:51.508815    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96223991-68df-4e67-a4ed-77cdf6530f22-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72\" (UID: \"96223991-68df-4e67-a4ed-77cdf6530f22\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72"
May 26 16:50:51.515486 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:51.515464    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wkxx\" (UniqueName: \"kubernetes.io/projected/96223991-68df-4e67-a4ed-77cdf6530f22-kube-api-access-5wkxx\") pod \"isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72\" (UID: \"96223991-68df-4e67-a4ed-77cdf6530f22\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72"
May 26 16:50:51.680083 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:51.680044    2572 generic.go:358] "Generic (PLEG): container finished" podID="3107d7e5-2bbf-491d-819b-f3fa592646f5" containerID="ae66c7199bb37827ca3f6688ed405c98351db250a47208b27185ee3194491dd3" exitCode=2
May 26 16:50:51.680264 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:51.680107    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-862j2" event={"ID":"3107d7e5-2bbf-491d-819b-f3fa592646f5","Type":"ContainerDied","Data":"ae66c7199bb37827ca3f6688ed405c98351db250a47208b27185ee3194491dd3"}
May 26 16:50:51.713497 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:51.713466    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72"
May 26 16:50:51.843188 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:51.843160    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72"]
May 26 16:50:51.845536 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:50:51.845509    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96223991_68df_4e67_a4ed_77cdf6530f22.slice/crio-4a80bb3831b338d91875da4e4ec567225eae70491f0fc10ff4ce542ee43bac57 WatchSource:0}: Error finding container 4a80bb3831b338d91875da4e4ec567225eae70491f0fc10ff4ce542ee43bac57: Status 404 returned error can't find the container with id 4a80bb3831b338d91875da4e4ec567225eae70491f0fc10ff4ce542ee43bac57
May 26 16:50:51.847250 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:51.847235    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 16:50:52.481762 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:52.481718    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-862j2" podUID="3107d7e5-2bbf-491d-819b-f3fa592646f5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.50:8643/healthz\": dial tcp 10.132.0.50:8643: connect: connection refused"
May 26 16:50:52.487096 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:52.487068    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-862j2" podUID="3107d7e5-2bbf-491d-819b-f3fa592646f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
May 26 16:50:52.683888 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:52.683848    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72" event={"ID":"96223991-68df-4e67-a4ed-77cdf6530f22","Type":"ContainerStarted","Data":"2ffe0f5892b4b10743ebe097b963b658edb4408b1b7294b1e04bccc0029fc36e"}
May 26 16:50:52.683888 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:52.683889    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72" event={"ID":"96223991-68df-4e67-a4ed-77cdf6530f22","Type":"ContainerStarted","Data":"4a80bb3831b338d91875da4e4ec567225eae70491f0fc10ff4ce542ee43bac57"}
May 26 16:50:55.693985 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:55.693953    2572 generic.go:358] "Generic (PLEG): container finished" podID="3107d7e5-2bbf-491d-819b-f3fa592646f5" containerID="c65f1574009d3218f3de9f954986658ed8177ecb15df303434d4600ae94ce592" exitCode=0
May 26 16:50:55.694386 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:55.693984    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-862j2" event={"ID":"3107d7e5-2bbf-491d-819b-f3fa592646f5","Type":"ContainerDied","Data":"c65f1574009d3218f3de9f954986658ed8177ecb15df303434d4600ae94ce592"}
May 26 16:50:55.697986 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:55.697952    2572 generic.go:358] "Generic (PLEG): container finished" podID="96223991-68df-4e67-a4ed-77cdf6530f22" containerID="2ffe0f5892b4b10743ebe097b963b658edb4408b1b7294b1e04bccc0029fc36e" exitCode=0
May 26 16:50:55.698147 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:55.698025    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72" event={"ID":"96223991-68df-4e67-a4ed-77cdf6530f22","Type":"ContainerDied","Data":"2ffe0f5892b4b10743ebe097b963b658edb4408b1b7294b1e04bccc0029fc36e"}
May 26 16:50:55.747251 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:55.747225    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-862j2"
May 26 16:50:55.835688 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:55.835659    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3107d7e5-2bbf-491d-819b-f3fa592646f5-kserve-provision-location\") pod \"3107d7e5-2bbf-491d-819b-f3fa592646f5\" (UID: \"3107d7e5-2bbf-491d-819b-f3fa592646f5\") "
May 26 16:50:55.835814 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:55.835763    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3107d7e5-2bbf-491d-819b-f3fa592646f5-isvc-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"3107d7e5-2bbf-491d-819b-f3fa592646f5\" (UID: \"3107d7e5-2bbf-491d-819b-f3fa592646f5\") "
May 26 16:50:55.835814 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:55.835798    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tx6l8\" (UniqueName: \"kubernetes.io/projected/3107d7e5-2bbf-491d-819b-f3fa592646f5-kube-api-access-tx6l8\") pod \"3107d7e5-2bbf-491d-819b-f3fa592646f5\" (UID: \"3107d7e5-2bbf-491d-819b-f3fa592646f5\") "
May 26 16:50:55.835933 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:55.835874    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3107d7e5-2bbf-491d-819b-f3fa592646f5-proxy-tls\") pod \"3107d7e5-2bbf-491d-819b-f3fa592646f5\" (UID: \"3107d7e5-2bbf-491d-819b-f3fa592646f5\") "
May 26 16:50:55.836066 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:55.836036    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3107d7e5-2bbf-491d-819b-f3fa592646f5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3107d7e5-2bbf-491d-819b-f3fa592646f5" (UID: "3107d7e5-2bbf-491d-819b-f3fa592646f5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:50:55.836135 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:55.836124    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3107d7e5-2bbf-491d-819b-f3fa592646f5-kserve-provision-location\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:50:55.836201 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:55.836119    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3107d7e5-2bbf-491d-819b-f3fa592646f5-isvc-sklearn-v2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-v2-kube-rbac-proxy-sar-config") pod "3107d7e5-2bbf-491d-819b-f3fa592646f5" (UID: "3107d7e5-2bbf-491d-819b-f3fa592646f5"). InnerVolumeSpecName "isvc-sklearn-v2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:50:55.837885 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:55.837846    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3107d7e5-2bbf-491d-819b-f3fa592646f5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3107d7e5-2bbf-491d-819b-f3fa592646f5" (UID: "3107d7e5-2bbf-491d-819b-f3fa592646f5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:50:55.837976 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:55.837899    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3107d7e5-2bbf-491d-819b-f3fa592646f5-kube-api-access-tx6l8" (OuterVolumeSpecName: "kube-api-access-tx6l8") pod "3107d7e5-2bbf-491d-819b-f3fa592646f5" (UID: "3107d7e5-2bbf-491d-819b-f3fa592646f5"). InnerVolumeSpecName "kube-api-access-tx6l8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:50:55.936496 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:55.936456    2572 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3107d7e5-2bbf-491d-819b-f3fa592646f5-isvc-sklearn-v2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:50:55.936496 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:55.936498    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tx6l8\" (UniqueName: \"kubernetes.io/projected/3107d7e5-2bbf-491d-819b-f3fa592646f5-kube-api-access-tx6l8\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:50:55.936496 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:55.936513    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3107d7e5-2bbf-491d-819b-f3fa592646f5-proxy-tls\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:50:56.702453 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:56.702415    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-862j2" event={"ID":"3107d7e5-2bbf-491d-819b-f3fa592646f5","Type":"ContainerDied","Data":"0835d0f271b2e0b18750d9dab3a1fe6915211856b868ad4f2cc254c9db466207"}
May 26 16:50:56.702903 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:56.702467    2572 scope.go:117] "RemoveContainer" containerID="ae66c7199bb37827ca3f6688ed405c98351db250a47208b27185ee3194491dd3"
May 26 16:50:56.702903 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:56.702474    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-862j2"
May 26 16:50:56.704363 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:56.704338    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72" event={"ID":"96223991-68df-4e67-a4ed-77cdf6530f22","Type":"ContainerStarted","Data":"5f9471b46dd1bb1b350d6bfd075d80a30fe9dcfa7b3b40ab7f537acf06bd81fe"}
May 26 16:50:56.704363 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:56.704377    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72" event={"ID":"96223991-68df-4e67-a4ed-77cdf6530f22","Type":"ContainerStarted","Data":"6ad7541dddf477fdc2023371b87eafd768c5125eb410c62741e98a4468b0503e"}
May 26 16:50:56.704649 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:56.704634    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72"
May 26 16:50:56.710088 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:56.710071    2572 scope.go:117] "RemoveContainer" containerID="c65f1574009d3218f3de9f954986658ed8177ecb15df303434d4600ae94ce592"
May 26 16:50:56.716928 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:56.716906    2572 scope.go:117] "RemoveContainer" containerID="09a755b6557b084c521ba4fc8dcdfb80956f2f71ec2806a466127b4dd1cef3b4"
May 26 16:50:56.722751 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:56.722726    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-862j2"]
May 26 16:50:56.723718 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:56.723700    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-9df9c96c7-862j2"]
May 26 16:50:56.741033 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:56.740975    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72" podStartSLOduration=5.740956725 podStartE2EDuration="5.740956725s" podCreationTimestamp="2026-05-26 16:50:51 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:50:56.739578139 +0000 UTC m=+2973.215305490" watchObservedRunningTime="2026-05-26 16:50:56.740956725 +0000 UTC m=+2973.216684076"
May 26 16:50:57.708629 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:57.708586    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72"
May 26 16:50:57.709534 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:57.709507    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72" podUID="96223991-68df-4e67-a4ed-77cdf6530f22" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
May 26 16:50:58.079158 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:58.079073    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3107d7e5-2bbf-491d-819b-f3fa592646f5" path="/var/lib/kubelet/pods/3107d7e5-2bbf-491d-819b-f3fa592646f5/volumes"
May 26 16:50:58.711303 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:50:58.711258    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72" podUID="96223991-68df-4e67-a4ed-77cdf6530f22" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
May 26 16:51:03.715868 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:51:03.715838    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72"
May 26 16:51:03.716312 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:51:03.716285    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72" podUID="96223991-68df-4e67-a4ed-77cdf6530f22" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
May 26 16:51:13.717057 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:51:13.717014    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72" podUID="96223991-68df-4e67-a4ed-77cdf6530f22" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
May 26 16:51:23.717134 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:51:23.717093    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72" podUID="96223991-68df-4e67-a4ed-77cdf6530f22" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
May 26 16:51:33.717233 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:51:33.717192    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72" podUID="96223991-68df-4e67-a4ed-77cdf6530f22" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
May 26 16:51:43.717170 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:51:43.717126    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72" podUID="96223991-68df-4e67-a4ed-77cdf6530f22" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
May 26 16:51:53.717110 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:51:53.717069    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72" podUID="96223991-68df-4e67-a4ed-77cdf6530f22" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
May 26 16:52:03.716768 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:03.716728    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72"
May 26 16:52:11.452368 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:11.452330    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72"]
May 26 16:52:11.452852 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:11.452656    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72" podUID="96223991-68df-4e67-a4ed-77cdf6530f22" containerName="kserve-container" containerID="cri-o://6ad7541dddf477fdc2023371b87eafd768c5125eb410c62741e98a4468b0503e" gracePeriod=30
May 26 16:52:11.452852 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:11.452730    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72" podUID="96223991-68df-4e67-a4ed-77cdf6530f22" containerName="kube-rbac-proxy" containerID="cri-o://5f9471b46dd1bb1b350d6bfd075d80a30fe9dcfa7b3b40ab7f537acf06bd81fe" gracePeriod=30
May 26 16:52:11.540624 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:11.540568    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-78wr2"]
May 26 16:52:11.540956 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:11.540938    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3107d7e5-2bbf-491d-819b-f3fa592646f5" containerName="kube-rbac-proxy"
May 26 16:52:11.541048 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:11.540958    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3107d7e5-2bbf-491d-819b-f3fa592646f5" containerName="kube-rbac-proxy"
May 26 16:52:11.541048 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:11.540982    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3107d7e5-2bbf-491d-819b-f3fa592646f5" containerName="kserve-container"
May 26 16:52:11.541048 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:11.540991    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3107d7e5-2bbf-491d-819b-f3fa592646f5" containerName="kserve-container"
May 26 16:52:11.541048 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:11.541005    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3107d7e5-2bbf-491d-819b-f3fa592646f5" containerName="storage-initializer"
May 26 16:52:11.541048 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:11.541015    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3107d7e5-2bbf-491d-819b-f3fa592646f5" containerName="storage-initializer"
May 26 16:52:11.541325 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:11.541083    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="3107d7e5-2bbf-491d-819b-f3fa592646f5" containerName="kube-rbac-proxy"
May 26 16:52:11.541325 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:11.541096    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="3107d7e5-2bbf-491d-819b-f3fa592646f5" containerName="kserve-container"
May 26 16:52:11.544429 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:11.544407    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-78wr2"
May 26 16:52:11.546473 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:11.546447    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-predictor-serving-cert\""
May 26 16:52:11.546578 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:11.546487    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
May 26 16:52:11.554386 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:11.554355    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-78wr2"]
May 26 16:52:11.701214 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:11.701171    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-tensorflow-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cb5ec3c-9e44-4282-813e-79e54e5ab493-isvc-tensorflow-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-6756f669d7-78wr2\" (UID: \"5cb5ec3c-9e44-4282-813e-79e54e5ab493\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-78wr2"
May 26 16:52:11.701402 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:11.701230    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5cb5ec3c-9e44-4282-813e-79e54e5ab493-kserve-provision-location\") pod \"isvc-tensorflow-predictor-6756f669d7-78wr2\" (UID: \"5cb5ec3c-9e44-4282-813e-79e54e5ab493\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-78wr2"
May 26 16:52:11.701402 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:11.701256    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nsfn\" (UniqueName: \"kubernetes.io/projected/5cb5ec3c-9e44-4282-813e-79e54e5ab493-kube-api-access-6nsfn\") pod \"isvc-tensorflow-predictor-6756f669d7-78wr2\" (UID: \"5cb5ec3c-9e44-4282-813e-79e54e5ab493\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-78wr2"
May 26 16:52:11.701402 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:11.701301    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cb5ec3c-9e44-4282-813e-79e54e5ab493-proxy-tls\") pod \"isvc-tensorflow-predictor-6756f669d7-78wr2\" (UID: \"5cb5ec3c-9e44-4282-813e-79e54e5ab493\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-78wr2"
May 26 16:52:11.802667 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:11.802549    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-tensorflow-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cb5ec3c-9e44-4282-813e-79e54e5ab493-isvc-tensorflow-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-6756f669d7-78wr2\" (UID: \"5cb5ec3c-9e44-4282-813e-79e54e5ab493\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-78wr2"
May 26 16:52:11.802667 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:11.802631    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5cb5ec3c-9e44-4282-813e-79e54e5ab493-kserve-provision-location\") pod \"isvc-tensorflow-predictor-6756f669d7-78wr2\" (UID: \"5cb5ec3c-9e44-4282-813e-79e54e5ab493\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-78wr2"
May 26 16:52:11.802667 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:11.802662    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6nsfn\" (UniqueName: \"kubernetes.io/projected/5cb5ec3c-9e44-4282-813e-79e54e5ab493-kube-api-access-6nsfn\") pod \"isvc-tensorflow-predictor-6756f669d7-78wr2\" (UID: \"5cb5ec3c-9e44-4282-813e-79e54e5ab493\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-78wr2"
May 26 16:52:11.802910 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:11.802732    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cb5ec3c-9e44-4282-813e-79e54e5ab493-proxy-tls\") pod \"isvc-tensorflow-predictor-6756f669d7-78wr2\" (UID: \"5cb5ec3c-9e44-4282-813e-79e54e5ab493\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-78wr2"
May 26 16:52:11.803094 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:11.803076    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5cb5ec3c-9e44-4282-813e-79e54e5ab493-kserve-provision-location\") pod \"isvc-tensorflow-predictor-6756f669d7-78wr2\" (UID: \"5cb5ec3c-9e44-4282-813e-79e54e5ab493\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-78wr2"
May 26 16:52:11.803360 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:11.803343    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-tensorflow-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cb5ec3c-9e44-4282-813e-79e54e5ab493-isvc-tensorflow-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-6756f669d7-78wr2\" (UID: \"5cb5ec3c-9e44-4282-813e-79e54e5ab493\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-78wr2"
May 26 16:52:11.805245 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:11.805212    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cb5ec3c-9e44-4282-813e-79e54e5ab493-proxy-tls\") pod \"isvc-tensorflow-predictor-6756f669d7-78wr2\" (UID: \"5cb5ec3c-9e44-4282-813e-79e54e5ab493\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-78wr2"
May 26 16:52:11.811129 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:11.811106    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nsfn\" (UniqueName: \"kubernetes.io/projected/5cb5ec3c-9e44-4282-813e-79e54e5ab493-kube-api-access-6nsfn\") pod \"isvc-tensorflow-predictor-6756f669d7-78wr2\" (UID: \"5cb5ec3c-9e44-4282-813e-79e54e5ab493\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-78wr2"
May 26 16:52:11.855780 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:11.855742    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-78wr2"
May 26 16:52:11.910495 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:11.910445    2572 generic.go:358] "Generic (PLEG): container finished" podID="96223991-68df-4e67-a4ed-77cdf6530f22" containerID="5f9471b46dd1bb1b350d6bfd075d80a30fe9dcfa7b3b40ab7f537acf06bd81fe" exitCode=2
May 26 16:52:11.910680 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:11.910522    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72" event={"ID":"96223991-68df-4e67-a4ed-77cdf6530f22","Type":"ContainerDied","Data":"5f9471b46dd1bb1b350d6bfd075d80a30fe9dcfa7b3b40ab7f537acf06bd81fe"}
May 26 16:52:11.985626 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:11.985572    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-78wr2"]
May 26 16:52:11.988831 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:52:11.988803    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cb5ec3c_9e44_4282_813e_79e54e5ab493.slice/crio-3721f32af38db456b84829e70111832163e64cb8805acc8755d1976460acb72d WatchSource:0}: Error finding container 3721f32af38db456b84829e70111832163e64cb8805acc8755d1976460acb72d: Status 404 returned error can't find the container with id 3721f32af38db456b84829e70111832163e64cb8805acc8755d1976460acb72d
May 26 16:52:12.914312 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:12.914276    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-78wr2" event={"ID":"5cb5ec3c-9e44-4282-813e-79e54e5ab493","Type":"ContainerStarted","Data":"4bd26832a4ca7366f582fbb3f8c62522fc17a560b09208444df74a1f52c16b6c"}
May 26 16:52:12.914312 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:12.914313    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-78wr2" event={"ID":"5cb5ec3c-9e44-4282-813e-79e54e5ab493","Type":"ContainerStarted","Data":"3721f32af38db456b84829e70111832163e64cb8805acc8755d1976460acb72d"}
May 26 16:52:13.711939 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:13.711895    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72" podUID="96223991-68df-4e67-a4ed-77cdf6530f22" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.51:8643/healthz\": dial tcp 10.132.0.51:8643: connect: connection refused"
May 26 16:52:13.717302 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:13.717272    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72" podUID="96223991-68df-4e67-a4ed-77cdf6530f22" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
May 26 16:52:15.890461 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:15.890435    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72"
May 26 16:52:15.924306 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:15.924273    2572 generic.go:358] "Generic (PLEG): container finished" podID="96223991-68df-4e67-a4ed-77cdf6530f22" containerID="6ad7541dddf477fdc2023371b87eafd768c5125eb410c62741e98a4468b0503e" exitCode=0
May 26 16:52:15.924496 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:15.924314    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72" event={"ID":"96223991-68df-4e67-a4ed-77cdf6530f22","Type":"ContainerDied","Data":"6ad7541dddf477fdc2023371b87eafd768c5125eb410c62741e98a4468b0503e"}
May 26 16:52:15.924496 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:15.924360    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72" event={"ID":"96223991-68df-4e67-a4ed-77cdf6530f22","Type":"ContainerDied","Data":"4a80bb3831b338d91875da4e4ec567225eae70491f0fc10ff4ce542ee43bac57"}
May 26 16:52:15.924496 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:15.924363    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72"
May 26 16:52:15.924496 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:15.924380    2572 scope.go:117] "RemoveContainer" containerID="5f9471b46dd1bb1b350d6bfd075d80a30fe9dcfa7b3b40ab7f537acf06bd81fe"
May 26 16:52:15.932827 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:15.932804    2572 scope.go:117] "RemoveContainer" containerID="6ad7541dddf477fdc2023371b87eafd768c5125eb410c62741e98a4468b0503e"
May 26 16:52:15.940276 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:15.940247    2572 scope.go:117] "RemoveContainer" containerID="2ffe0f5892b4b10743ebe097b963b658edb4408b1b7294b1e04bccc0029fc36e"
May 26 16:52:15.947353 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:15.947333    2572 scope.go:117] "RemoveContainer" containerID="5f9471b46dd1bb1b350d6bfd075d80a30fe9dcfa7b3b40ab7f537acf06bd81fe"
May 26 16:52:15.947621 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:52:15.947588    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f9471b46dd1bb1b350d6bfd075d80a30fe9dcfa7b3b40ab7f537acf06bd81fe\": container with ID starting with 5f9471b46dd1bb1b350d6bfd075d80a30fe9dcfa7b3b40ab7f537acf06bd81fe not found: ID does not exist" containerID="5f9471b46dd1bb1b350d6bfd075d80a30fe9dcfa7b3b40ab7f537acf06bd81fe"
May 26 16:52:15.947720 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:15.947628    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f9471b46dd1bb1b350d6bfd075d80a30fe9dcfa7b3b40ab7f537acf06bd81fe"} err="failed to get container status \"5f9471b46dd1bb1b350d6bfd075d80a30fe9dcfa7b3b40ab7f537acf06bd81fe\": rpc error: code = NotFound desc = could not find container \"5f9471b46dd1bb1b350d6bfd075d80a30fe9dcfa7b3b40ab7f537acf06bd81fe\": container with ID starting with 5f9471b46dd1bb1b350d6bfd075d80a30fe9dcfa7b3b40ab7f537acf06bd81fe not found: ID does not exist"
May 26 16:52:15.947720 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:15.947647    2572 scope.go:117] "RemoveContainer" containerID="6ad7541dddf477fdc2023371b87eafd768c5125eb410c62741e98a4468b0503e"
May 26 16:52:15.948001 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:52:15.947985    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ad7541dddf477fdc2023371b87eafd768c5125eb410c62741e98a4468b0503e\": container with ID starting with 6ad7541dddf477fdc2023371b87eafd768c5125eb410c62741e98a4468b0503e not found: ID does not exist" containerID="6ad7541dddf477fdc2023371b87eafd768c5125eb410c62741e98a4468b0503e"
May 26 16:52:15.948041 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:15.948006    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ad7541dddf477fdc2023371b87eafd768c5125eb410c62741e98a4468b0503e"} err="failed to get container status \"6ad7541dddf477fdc2023371b87eafd768c5125eb410c62741e98a4468b0503e\": rpc error: code = NotFound desc = could not find container \"6ad7541dddf477fdc2023371b87eafd768c5125eb410c62741e98a4468b0503e\": container with ID starting with 6ad7541dddf477fdc2023371b87eafd768c5125eb410c62741e98a4468b0503e not found: ID does not exist"
May 26 16:52:15.948041 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:15.948019    2572 scope.go:117] "RemoveContainer" containerID="2ffe0f5892b4b10743ebe097b963b658edb4408b1b7294b1e04bccc0029fc36e"
May 26 16:52:15.948223 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:52:15.948204    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ffe0f5892b4b10743ebe097b963b658edb4408b1b7294b1e04bccc0029fc36e\": container with ID starting with 2ffe0f5892b4b10743ebe097b963b658edb4408b1b7294b1e04bccc0029fc36e not found: ID does not exist" containerID="2ffe0f5892b4b10743ebe097b963b658edb4408b1b7294b1e04bccc0029fc36e"
May 26 16:52:15.948266 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:15.948226    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ffe0f5892b4b10743ebe097b963b658edb4408b1b7294b1e04bccc0029fc36e"} err="failed to get container status \"2ffe0f5892b4b10743ebe097b963b658edb4408b1b7294b1e04bccc0029fc36e\": rpc error: code = NotFound desc = could not find container \"2ffe0f5892b4b10743ebe097b963b658edb4408b1b7294b1e04bccc0029fc36e\": container with ID starting with 2ffe0f5892b4b10743ebe097b963b658edb4408b1b7294b1e04bccc0029fc36e not found: ID does not exist"
May 26 16:52:16.031898 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:16.031810    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96223991-68df-4e67-a4ed-77cdf6530f22-proxy-tls\") pod \"96223991-68df-4e67-a4ed-77cdf6530f22\" (UID: \"96223991-68df-4e67-a4ed-77cdf6530f22\") "
May 26 16:52:16.031898 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:16.031860    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96223991-68df-4e67-a4ed-77cdf6530f22-kserve-provision-location\") pod \"96223991-68df-4e67-a4ed-77cdf6530f22\" (UID: \"96223991-68df-4e67-a4ed-77cdf6530f22\") "
May 26 16:52:16.032101 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:16.031907    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/96223991-68df-4e67-a4ed-77cdf6530f22-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\") pod \"96223991-68df-4e67-a4ed-77cdf6530f22\" (UID: \"96223991-68df-4e67-a4ed-77cdf6530f22\") "
May 26 16:52:16.032101 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:16.031925    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5wkxx\" (UniqueName: \"kubernetes.io/projected/96223991-68df-4e67-a4ed-77cdf6530f22-kube-api-access-5wkxx\") pod \"96223991-68df-4e67-a4ed-77cdf6530f22\" (UID: \"96223991-68df-4e67-a4ed-77cdf6530f22\") "
May 26 16:52:16.032310 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:16.032282    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96223991-68df-4e67-a4ed-77cdf6530f22-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config") pod "96223991-68df-4e67-a4ed-77cdf6530f22" (UID: "96223991-68df-4e67-a4ed-77cdf6530f22"). InnerVolumeSpecName "isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:52:16.032415 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:16.032321    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96223991-68df-4e67-a4ed-77cdf6530f22-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "96223991-68df-4e67-a4ed-77cdf6530f22" (UID: "96223991-68df-4e67-a4ed-77cdf6530f22"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:52:16.034069 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:16.034039    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96223991-68df-4e67-a4ed-77cdf6530f22-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "96223991-68df-4e67-a4ed-77cdf6530f22" (UID: "96223991-68df-4e67-a4ed-77cdf6530f22"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:52:16.034180 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:16.034141    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96223991-68df-4e67-a4ed-77cdf6530f22-kube-api-access-5wkxx" (OuterVolumeSpecName: "kube-api-access-5wkxx") pod "96223991-68df-4e67-a4ed-77cdf6530f22" (UID: "96223991-68df-4e67-a4ed-77cdf6530f22"). InnerVolumeSpecName "kube-api-access-5wkxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:52:16.132685 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:16.132648    2572 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/96223991-68df-4e67-a4ed-77cdf6530f22-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:52:16.132685 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:16.132678    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wkxx\" (UniqueName: \"kubernetes.io/projected/96223991-68df-4e67-a4ed-77cdf6530f22-kube-api-access-5wkxx\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:52:16.132685 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:16.132689    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96223991-68df-4e67-a4ed-77cdf6530f22-proxy-tls\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:52:16.132921 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:16.132700    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96223991-68df-4e67-a4ed-77cdf6530f22-kserve-provision-location\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:52:16.239626 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:16.239575    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72"]
May 26 16:52:16.244408 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:16.244380    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5cd9489bbd-nnn72"]
May 26 16:52:16.929876 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:16.929835    2572 generic.go:358] "Generic (PLEG): container finished" podID="5cb5ec3c-9e44-4282-813e-79e54e5ab493" containerID="4bd26832a4ca7366f582fbb3f8c62522fc17a560b09208444df74a1f52c16b6c" exitCode=0
May 26 16:52:16.929876 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:16.929879    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-78wr2" event={"ID":"5cb5ec3c-9e44-4282-813e-79e54e5ab493","Type":"ContainerDied","Data":"4bd26832a4ca7366f582fbb3f8c62522fc17a560b09208444df74a1f52c16b6c"}
May 26 16:52:18.080186 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:18.079946    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96223991-68df-4e67-a4ed-77cdf6530f22" path="/var/lib/kubelet/pods/96223991-68df-4e67-a4ed-77cdf6530f22/volumes"
May 26 16:52:20.942863 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:20.942826    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-78wr2" event={"ID":"5cb5ec3c-9e44-4282-813e-79e54e5ab493","Type":"ContainerStarted","Data":"5cea6076fa1ad2df3eb38c071a6734bc5d3472ad0463d4a0939e00ba8571f532"}
May 26 16:52:20.943331 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:20.942871    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-78wr2" event={"ID":"5cb5ec3c-9e44-4282-813e-79e54e5ab493","Type":"ContainerStarted","Data":"23f1960eba0947cf0d193acfd7b4aecb2f9f048fa50dada6d3a5928d4e43590c"}
May 26 16:52:20.943331 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:20.943105    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-78wr2"
May 26 16:52:20.961895 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:20.961830    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-78wr2" podStartSLOduration=6.3336606060000005 podStartE2EDuration="9.961817415s" podCreationTimestamp="2026-05-26 16:52:11 +0000 UTC" firstStartedPulling="2026-05-26 16:52:16.93098884 +0000 UTC m=+3053.406716168" lastFinishedPulling="2026-05-26 16:52:20.559145633 +0000 UTC m=+3057.034872977" observedRunningTime="2026-05-26 16:52:20.960507786 +0000 UTC m=+3057.436235136" watchObservedRunningTime="2026-05-26 16:52:20.961817415 +0000 UTC m=+3057.437544766"
May 26 16:52:21.945924 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:21.945884    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-78wr2"
May 26 16:52:21.946936 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:21.946910    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-78wr2" podUID="5cb5ec3c-9e44-4282-813e-79e54e5ab493" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
May 26 16:52:22.948198 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:22.948156    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-78wr2" podUID="5cb5ec3c-9e44-4282-813e-79e54e5ab493" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
May 26 16:52:27.952100 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:27.952064    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-78wr2"
May 26 16:52:27.952675 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:27.952646    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-78wr2" podUID="5cb5ec3c-9e44-4282-813e-79e54e5ab493" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
May 26 16:52:37.952859 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:37.952826    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-78wr2"
May 26 16:52:52.774479 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:52.774448    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-78wr2"]
May 26 16:52:52.774911 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:52.774792    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-78wr2" podUID="5cb5ec3c-9e44-4282-813e-79e54e5ab493" containerName="kserve-container" containerID="cri-o://23f1960eba0947cf0d193acfd7b4aecb2f9f048fa50dada6d3a5928d4e43590c" gracePeriod=30
May 26 16:52:52.774911 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:52.774833    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-78wr2" podUID="5cb5ec3c-9e44-4282-813e-79e54e5ab493" containerName="kube-rbac-proxy" containerID="cri-o://5cea6076fa1ad2df3eb38c071a6734bc5d3472ad0463d4a0939e00ba8571f532" gracePeriod=30
May 26 16:52:52.850354 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:52.850315    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl"]
May 26 16:52:52.850624 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:52.850591    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96223991-68df-4e67-a4ed-77cdf6530f22" containerName="storage-initializer"
May 26 16:52:52.850624 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:52.850619    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="96223991-68df-4e67-a4ed-77cdf6530f22" containerName="storage-initializer"
May 26 16:52:52.850710 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:52.850634    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96223991-68df-4e67-a4ed-77cdf6530f22" containerName="kserve-container"
May 26 16:52:52.850710 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:52.850640    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="96223991-68df-4e67-a4ed-77cdf6530f22" containerName="kserve-container"
May 26 16:52:52.850710 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:52.850653    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96223991-68df-4e67-a4ed-77cdf6530f22" containerName="kube-rbac-proxy"
May 26 16:52:52.850710 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:52.850659    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="96223991-68df-4e67-a4ed-77cdf6530f22" containerName="kube-rbac-proxy"
May 26 16:52:52.850840 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:52.850717    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="96223991-68df-4e67-a4ed-77cdf6530f22" containerName="kube-rbac-proxy"
May 26 16:52:52.850840 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:52.850725    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="96223991-68df-4e67-a4ed-77cdf6530f22" containerName="kserve-container"
May 26 16:52:52.853885 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:52.853856    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl"
May 26 16:52:52.855971 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:52.855953    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\""
May 26 16:52:52.856191 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:52.856170    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-predictor-serving-cert\""
May 26 16:52:52.865229 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:52.865202    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl"]
May 26 16:52:52.916239 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:52.916205    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dffc9457-5179-4dd6-b015-ab504e05acfc-isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl\" (UID: \"dffc9457-5179-4dd6-b015-ab504e05acfc\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl"
May 26 16:52:52.916239 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:52.916248    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl6f4\" (UniqueName: \"kubernetes.io/projected/dffc9457-5179-4dd6-b015-ab504e05acfc-kube-api-access-sl6f4\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl\" (UID: \"dffc9457-5179-4dd6-b015-ab504e05acfc\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl"
May 26 16:52:52.916520 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:52.916268    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dffc9457-5179-4dd6-b015-ab504e05acfc-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl\" (UID: \"dffc9457-5179-4dd6-b015-ab504e05acfc\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl"
May 26 16:52:52.916520 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:52.916370    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dffc9457-5179-4dd6-b015-ab504e05acfc-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl\" (UID: \"dffc9457-5179-4dd6-b015-ab504e05acfc\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl"
May 26 16:52:52.948541 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:52.948498    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-78wr2" podUID="5cb5ec3c-9e44-4282-813e-79e54e5ab493" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.52:8643/healthz\": dial tcp 10.132.0.52:8643: connect: connection refused"
May 26 16:52:53.017803 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:53.017761    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dffc9457-5179-4dd6-b015-ab504e05acfc-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl\" (UID: \"dffc9457-5179-4dd6-b015-ab504e05acfc\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl"
May 26 16:52:53.018020 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:53.017821    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dffc9457-5179-4dd6-b015-ab504e05acfc-isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl\" (UID: \"dffc9457-5179-4dd6-b015-ab504e05acfc\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl"
May 26 16:52:53.018020 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:53.017859    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sl6f4\" (UniqueName: \"kubernetes.io/projected/dffc9457-5179-4dd6-b015-ab504e05acfc-kube-api-access-sl6f4\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl\" (UID: \"dffc9457-5179-4dd6-b015-ab504e05acfc\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl"
May 26 16:52:53.018020 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:53.017900    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dffc9457-5179-4dd6-b015-ab504e05acfc-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl\" (UID: \"dffc9457-5179-4dd6-b015-ab504e05acfc\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl"
May 26 16:52:53.018020 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:52:53.017915    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-serving-cert: secret "isvc-tensorflow-runtime-predictor-serving-cert" not found
May 26 16:52:53.018020 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:52:53.017984    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dffc9457-5179-4dd6-b015-ab504e05acfc-proxy-tls podName:dffc9457-5179-4dd6-b015-ab504e05acfc nodeName:}" failed. No retries permitted until 2026-05-26 16:52:53.517968695 +0000 UTC m=+3089.993696024 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/dffc9457-5179-4dd6-b015-ab504e05acfc-proxy-tls") pod "isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl" (UID: "dffc9457-5179-4dd6-b015-ab504e05acfc") : secret "isvc-tensorflow-runtime-predictor-serving-cert" not found
May 26 16:52:53.018350 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:53.018331    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dffc9457-5179-4dd6-b015-ab504e05acfc-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl\" (UID: \"dffc9457-5179-4dd6-b015-ab504e05acfc\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl"
May 26 16:52:53.018511 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:53.018491    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dffc9457-5179-4dd6-b015-ab504e05acfc-isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl\" (UID: \"dffc9457-5179-4dd6-b015-ab504e05acfc\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl"
May 26 16:52:53.026460 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:53.026396    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl6f4\" (UniqueName: \"kubernetes.io/projected/dffc9457-5179-4dd6-b015-ab504e05acfc-kube-api-access-sl6f4\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl\" (UID: \"dffc9457-5179-4dd6-b015-ab504e05acfc\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl"
May 26 16:52:53.029309 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:53.029280    2572 generic.go:358] "Generic (PLEG): container finished" podID="5cb5ec3c-9e44-4282-813e-79e54e5ab493" containerID="5cea6076fa1ad2df3eb38c071a6734bc5d3472ad0463d4a0939e00ba8571f532" exitCode=2
May 26 16:52:53.029424 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:53.029334    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-78wr2" event={"ID":"5cb5ec3c-9e44-4282-813e-79e54e5ab493","Type":"ContainerDied","Data":"5cea6076fa1ad2df3eb38c071a6734bc5d3472ad0463d4a0939e00ba8571f532"}
May 26 16:52:53.522467 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:53.522424    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dffc9457-5179-4dd6-b015-ab504e05acfc-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl\" (UID: \"dffc9457-5179-4dd6-b015-ab504e05acfc\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl"
May 26 16:52:53.524913 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:53.524886    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dffc9457-5179-4dd6-b015-ab504e05acfc-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl\" (UID: \"dffc9457-5179-4dd6-b015-ab504e05acfc\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl"
May 26 16:52:53.764387 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:53.764348    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl"
May 26 16:52:53.893948 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:53.893877    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl"]
May 26 16:52:53.896755 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:52:53.896728    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddffc9457_5179_4dd6_b015_ab504e05acfc.slice/crio-a8edff06b3c6d4d0c44d8d8a00fd06d7ac2ca6d5e853195a3c7f9bbabb62fc8b WatchSource:0}: Error finding container a8edff06b3c6d4d0c44d8d8a00fd06d7ac2ca6d5e853195a3c7f9bbabb62fc8b: Status 404 returned error can't find the container with id a8edff06b3c6d4d0c44d8d8a00fd06d7ac2ca6d5e853195a3c7f9bbabb62fc8b
May 26 16:52:54.033653 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:54.033560    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl" event={"ID":"dffc9457-5179-4dd6-b015-ab504e05acfc","Type":"ContainerStarted","Data":"e11457dfefde6edd4e57cbed14ed8abedf69c6dc62603ab81bad73e158353bf1"}
May 26 16:52:54.033653 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:54.033596    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl" event={"ID":"dffc9457-5179-4dd6-b015-ab504e05acfc","Type":"ContainerStarted","Data":"a8edff06b3c6d4d0c44d8d8a00fd06d7ac2ca6d5e853195a3c7f9bbabb62fc8b"}
May 26 16:52:57.948669 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:57.948596    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-78wr2" podUID="5cb5ec3c-9e44-4282-813e-79e54e5ab493" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.52:8643/healthz\": dial tcp 10.132.0.52:8643: connect: connection refused"
May 26 16:52:59.048826 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:59.048786    2572 generic.go:358] "Generic (PLEG): container finished" podID="dffc9457-5179-4dd6-b015-ab504e05acfc" containerID="e11457dfefde6edd4e57cbed14ed8abedf69c6dc62603ab81bad73e158353bf1" exitCode=0
May 26 16:52:59.049336 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:52:59.048861    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl" event={"ID":"dffc9457-5179-4dd6-b015-ab504e05acfc","Type":"ContainerDied","Data":"e11457dfefde6edd4e57cbed14ed8abedf69c6dc62603ab81bad73e158353bf1"}
May 26 16:53:00.053613 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:00.053562    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl" event={"ID":"dffc9457-5179-4dd6-b015-ab504e05acfc","Type":"ContainerStarted","Data":"80893d98fdd38fbfa29c1f0512e7d2d1d835008b6cc90935d839b9fe99c6ba4c"}
May 26 16:53:00.054080 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:00.053623    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl" event={"ID":"dffc9457-5179-4dd6-b015-ab504e05acfc","Type":"ContainerStarted","Data":"538202dea96d0e01865d2c1c74babbe10fc111f24e71bc08c67c0d7db4222abe"}
May 26 16:53:00.054080 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:00.053849    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl"
May 26 16:53:01.055992 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:01.055957    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl"
May 26 16:53:01.057087 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:01.057059    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl" podUID="dffc9457-5179-4dd6-b015-ab504e05acfc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
May 26 16:53:02.059568 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:02.059526    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl" podUID="dffc9457-5179-4dd6-b015-ab504e05acfc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
May 26 16:53:02.949335 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:02.949284    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-78wr2" podUID="5cb5ec3c-9e44-4282-813e-79e54e5ab493" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.52:8643/healthz\": dial tcp 10.132.0.52:8643: connect: connection refused"
May 26 16:53:02.949550 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:02.949482    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-78wr2"
May 26 16:53:02.967793 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:02.967728    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl" podStartSLOduration=10.967708099 podStartE2EDuration="10.967708099s" podCreationTimestamp="2026-05-26 16:52:52 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:53:00.072671664 +0000 UTC m=+3096.548399017" watchObservedRunningTime="2026-05-26 16:53:02.967708099 +0000 UTC m=+3099.443435450"
May 26 16:53:07.063754 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:07.063723    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl"
May 26 16:53:07.064327 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:07.064299    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl" podUID="dffc9457-5179-4dd6-b015-ab504e05acfc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
May 26 16:53:07.948666 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:07.948620    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-78wr2" podUID="5cb5ec3c-9e44-4282-813e-79e54e5ab493" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.52:8643/healthz\": dial tcp 10.132.0.52:8643: connect: connection refused"
May 26 16:53:12.948718 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:12.948664    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-78wr2" podUID="5cb5ec3c-9e44-4282-813e-79e54e5ab493" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.52:8643/healthz\": dial tcp 10.132.0.52:8643: connect: connection refused"
May 26 16:53:17.064971 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:17.064941    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl"
May 26 16:53:17.949205 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:17.949159    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-78wr2" podUID="5cb5ec3c-9e44-4282-813e-79e54e5ab493" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.52:8643/healthz\": dial tcp 10.132.0.52:8643: connect: connection refused"
May 26 16:53:22.949029 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:22.948981    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-78wr2" podUID="5cb5ec3c-9e44-4282-813e-79e54e5ab493" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.52:8643/healthz\": dial tcp 10.132.0.52:8643: connect: connection refused"
May 26 16:53:23.122545 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:23.122515    2572 generic.go:358] "Generic (PLEG): container finished" podID="5cb5ec3c-9e44-4282-813e-79e54e5ab493" containerID="23f1960eba0947cf0d193acfd7b4aecb2f9f048fa50dada6d3a5928d4e43590c" exitCode=137
May 26 16:53:23.122737 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:23.122572    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-78wr2" event={"ID":"5cb5ec3c-9e44-4282-813e-79e54e5ab493","Type":"ContainerDied","Data":"23f1960eba0947cf0d193acfd7b4aecb2f9f048fa50dada6d3a5928d4e43590c"}
May 26 16:53:23.411293 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:23.411270    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-78wr2"
May 26 16:53:23.566044 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:23.565998    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-tensorflow-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cb5ec3c-9e44-4282-813e-79e54e5ab493-isvc-tensorflow-kube-rbac-proxy-sar-config\") pod \"5cb5ec3c-9e44-4282-813e-79e54e5ab493\" (UID: \"5cb5ec3c-9e44-4282-813e-79e54e5ab493\") "
May 26 16:53:23.566215 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:23.566072    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5cb5ec3c-9e44-4282-813e-79e54e5ab493-kserve-provision-location\") pod \"5cb5ec3c-9e44-4282-813e-79e54e5ab493\" (UID: \"5cb5ec3c-9e44-4282-813e-79e54e5ab493\") "
May 26 16:53:23.566215 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:23.566105    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cb5ec3c-9e44-4282-813e-79e54e5ab493-proxy-tls\") pod \"5cb5ec3c-9e44-4282-813e-79e54e5ab493\" (UID: \"5cb5ec3c-9e44-4282-813e-79e54e5ab493\") "
May 26 16:53:23.566215 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:23.566180    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6nsfn\" (UniqueName: \"kubernetes.io/projected/5cb5ec3c-9e44-4282-813e-79e54e5ab493-kube-api-access-6nsfn\") pod \"5cb5ec3c-9e44-4282-813e-79e54e5ab493\" (UID: \"5cb5ec3c-9e44-4282-813e-79e54e5ab493\") "
May 26 16:53:23.566413 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:23.566390    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cb5ec3c-9e44-4282-813e-79e54e5ab493-isvc-tensorflow-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-tensorflow-kube-rbac-proxy-sar-config") pod "5cb5ec3c-9e44-4282-813e-79e54e5ab493" (UID: "5cb5ec3c-9e44-4282-813e-79e54e5ab493"). InnerVolumeSpecName "isvc-tensorflow-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:53:23.568230 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:23.568198    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cb5ec3c-9e44-4282-813e-79e54e5ab493-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5cb5ec3c-9e44-4282-813e-79e54e5ab493" (UID: "5cb5ec3c-9e44-4282-813e-79e54e5ab493"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:53:23.568361 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:23.568198    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cb5ec3c-9e44-4282-813e-79e54e5ab493-kube-api-access-6nsfn" (OuterVolumeSpecName: "kube-api-access-6nsfn") pod "5cb5ec3c-9e44-4282-813e-79e54e5ab493" (UID: "5cb5ec3c-9e44-4282-813e-79e54e5ab493"). InnerVolumeSpecName "kube-api-access-6nsfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:53:23.577262 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:23.577234    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cb5ec3c-9e44-4282-813e-79e54e5ab493-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5cb5ec3c-9e44-4282-813e-79e54e5ab493" (UID: "5cb5ec3c-9e44-4282-813e-79e54e5ab493"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:53:23.666963 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:23.666878    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5cb5ec3c-9e44-4282-813e-79e54e5ab493-kserve-provision-location\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:53:23.666963 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:23.666905    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cb5ec3c-9e44-4282-813e-79e54e5ab493-proxy-tls\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:53:23.666963 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:23.666941    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nsfn\" (UniqueName: \"kubernetes.io/projected/5cb5ec3c-9e44-4282-813e-79e54e5ab493-kube-api-access-6nsfn\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:53:23.666963 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:23.666953    2572 reconciler_common.go:299] "Volume detached for volume \"isvc-tensorflow-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cb5ec3c-9e44-4282-813e-79e54e5ab493-isvc-tensorflow-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:53:24.126883 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:24.126853    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-78wr2" event={"ID":"5cb5ec3c-9e44-4282-813e-79e54e5ab493","Type":"ContainerDied","Data":"3721f32af38db456b84829e70111832163e64cb8805acc8755d1976460acb72d"}
May 26 16:53:24.126883 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:24.126879    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-78wr2"
May 26 16:53:24.127427 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:24.126896    2572 scope.go:117] "RemoveContainer" containerID="5cea6076fa1ad2df3eb38c071a6734bc5d3472ad0463d4a0939e00ba8571f532"
May 26 16:53:24.134684 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:24.134665    2572 scope.go:117] "RemoveContainer" containerID="23f1960eba0947cf0d193acfd7b4aecb2f9f048fa50dada6d3a5928d4e43590c"
May 26 16:53:24.142249 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:24.142131    2572 scope.go:117] "RemoveContainer" containerID="4bd26832a4ca7366f582fbb3f8c62522fc17a560b09208444df74a1f52c16b6c"
May 26 16:53:24.143522 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:24.143500    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-78wr2"]
May 26 16:53:24.146838 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:24.146814    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-6756f669d7-78wr2"]
May 26 16:53:26.079227 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:26.079193    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cb5ec3c-9e44-4282-813e-79e54e5ab493" path="/var/lib/kubelet/pods/5cb5ec3c-9e44-4282-813e-79e54e5ab493/volumes"
May 26 16:53:34.071218 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:34.071182    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl"]
May 26 16:53:34.071803 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:34.071493    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl" podUID="dffc9457-5179-4dd6-b015-ab504e05acfc" containerName="kserve-container" containerID="cri-o://538202dea96d0e01865d2c1c74babbe10fc111f24e71bc08c67c0d7db4222abe" gracePeriod=30
May 26 16:53:34.071803 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:34.071541    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl" podUID="dffc9457-5179-4dd6-b015-ab504e05acfc" containerName="kube-rbac-proxy" containerID="cri-o://80893d98fdd38fbfa29c1f0512e7d2d1d835008b6cc90935d839b9fe99c6ba4c" gracePeriod=30
May 26 16:53:34.155740 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:34.155703    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-wg8gk"]
May 26 16:53:34.156041 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:34.156015    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cb5ec3c-9e44-4282-813e-79e54e5ab493" containerName="kserve-container"
May 26 16:53:34.156041 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:34.156033    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cb5ec3c-9e44-4282-813e-79e54e5ab493" containerName="kserve-container"
May 26 16:53:34.156237 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:34.156062    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cb5ec3c-9e44-4282-813e-79e54e5ab493" containerName="storage-initializer"
May 26 16:53:34.156237 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:34.156068    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cb5ec3c-9e44-4282-813e-79e54e5ab493" containerName="storage-initializer"
May 26 16:53:34.156237 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:34.156075    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cb5ec3c-9e44-4282-813e-79e54e5ab493" containerName="kube-rbac-proxy"
May 26 16:53:34.156237 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:34.156081    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cb5ec3c-9e44-4282-813e-79e54e5ab493" containerName="kube-rbac-proxy"
May 26 16:53:34.156237 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:34.156125    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="5cb5ec3c-9e44-4282-813e-79e54e5ab493" containerName="kube-rbac-proxy"
May 26 16:53:34.156237 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:34.156132    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="5cb5ec3c-9e44-4282-813e-79e54e5ab493" containerName="kserve-container"
May 26 16:53:34.159207 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:34.159188    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-wg8gk"
May 26 16:53:34.161130 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:34.161110    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-predictor-serving-cert\""
May 26 16:53:34.161253 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:34.161128    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-kube-rbac-proxy-sar-config\""
May 26 16:53:34.169118 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:34.169093    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-wg8gk"]
May 26 16:53:34.237532 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:34.237500    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16601799-343b-43b6-af7a-b712f0338895-proxy-tls\") pod \"isvc-triton-predictor-84bb65d94b-wg8gk\" (UID: \"16601799-343b-43b6-af7a-b712f0338895\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-wg8gk"
May 26 16:53:34.237724 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:34.237558    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16601799-343b-43b6-af7a-b712f0338895-kserve-provision-location\") pod \"isvc-triton-predictor-84bb65d94b-wg8gk\" (UID: \"16601799-343b-43b6-af7a-b712f0338895\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-wg8gk"
May 26 16:53:34.237724 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:34.237660    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-triton-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16601799-343b-43b6-af7a-b712f0338895-isvc-triton-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-84bb65d94b-wg8gk\" (UID: \"16601799-343b-43b6-af7a-b712f0338895\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-wg8gk"
May 26 16:53:34.237724 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:34.237689    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d8xz\" (UniqueName: \"kubernetes.io/projected/16601799-343b-43b6-af7a-b712f0338895-kube-api-access-5d8xz\") pod \"isvc-triton-predictor-84bb65d94b-wg8gk\" (UID: \"16601799-343b-43b6-af7a-b712f0338895\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-wg8gk"
May 26 16:53:34.338101 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:34.338005    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16601799-343b-43b6-af7a-b712f0338895-proxy-tls\") pod \"isvc-triton-predictor-84bb65d94b-wg8gk\" (UID: \"16601799-343b-43b6-af7a-b712f0338895\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-wg8gk"
May 26 16:53:34.338101 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:34.338063    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16601799-343b-43b6-af7a-b712f0338895-kserve-provision-location\") pod \"isvc-triton-predictor-84bb65d94b-wg8gk\" (UID: \"16601799-343b-43b6-af7a-b712f0338895\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-wg8gk"
May 26 16:53:34.338101 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:34.338100    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-triton-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16601799-343b-43b6-af7a-b712f0338895-isvc-triton-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-84bb65d94b-wg8gk\" (UID: \"16601799-343b-43b6-af7a-b712f0338895\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-wg8gk"
May 26 16:53:34.338404 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:34.338118    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5d8xz\" (UniqueName: \"kubernetes.io/projected/16601799-343b-43b6-af7a-b712f0338895-kube-api-access-5d8xz\") pod \"isvc-triton-predictor-84bb65d94b-wg8gk\" (UID: \"16601799-343b-43b6-af7a-b712f0338895\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-wg8gk"
May 26 16:53:34.338533 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:34.338514    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16601799-343b-43b6-af7a-b712f0338895-kserve-provision-location\") pod \"isvc-triton-predictor-84bb65d94b-wg8gk\" (UID: \"16601799-343b-43b6-af7a-b712f0338895\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-wg8gk"
May 26 16:53:34.338839 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:34.338817    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-triton-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16601799-343b-43b6-af7a-b712f0338895-isvc-triton-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-84bb65d94b-wg8gk\" (UID: \"16601799-343b-43b6-af7a-b712f0338895\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-wg8gk"
May 26 16:53:34.340421 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:34.340399    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16601799-343b-43b6-af7a-b712f0338895-proxy-tls\") pod \"isvc-triton-predictor-84bb65d94b-wg8gk\" (UID: \"16601799-343b-43b6-af7a-b712f0338895\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-wg8gk"
May 26 16:53:34.345975 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:34.345947    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d8xz\" (UniqueName: \"kubernetes.io/projected/16601799-343b-43b6-af7a-b712f0338895-kube-api-access-5d8xz\") pod \"isvc-triton-predictor-84bb65d94b-wg8gk\" (UID: \"16601799-343b-43b6-af7a-b712f0338895\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-wg8gk"
May 26 16:53:34.469910 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:34.469851    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-wg8gk"
May 26 16:53:34.596278 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:34.596197    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-wg8gk"]
May 26 16:53:34.599185 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:53:34.599151    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16601799_343b_43b6_af7a_b712f0338895.slice/crio-ee0fae2a2c674d11204369be18227b191047b887295195ca3ff7ad4d440718c2 WatchSource:0}: Error finding container ee0fae2a2c674d11204369be18227b191047b887295195ca3ff7ad4d440718c2: Status 404 returned error can't find the container with id ee0fae2a2c674d11204369be18227b191047b887295195ca3ff7ad4d440718c2
May 26 16:53:35.160612 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:35.160562    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-wg8gk" event={"ID":"16601799-343b-43b6-af7a-b712f0338895","Type":"ContainerStarted","Data":"2613afbbce3fdef03429c261c8f0d0588b249bbc2c77785fe0b78c829f742b60"}
May 26 16:53:35.161049 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:35.160626    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-wg8gk" event={"ID":"16601799-343b-43b6-af7a-b712f0338895","Type":"ContainerStarted","Data":"ee0fae2a2c674d11204369be18227b191047b887295195ca3ff7ad4d440718c2"}
May 26 16:53:35.162383 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:35.162358    2572 generic.go:358] "Generic (PLEG): container finished" podID="dffc9457-5179-4dd6-b015-ab504e05acfc" containerID="80893d98fdd38fbfa29c1f0512e7d2d1d835008b6cc90935d839b9fe99c6ba4c" exitCode=2
May 26 16:53:35.162488 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:35.162431    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl" event={"ID":"dffc9457-5179-4dd6-b015-ab504e05acfc","Type":"ContainerDied","Data":"80893d98fdd38fbfa29c1f0512e7d2d1d835008b6cc90935d839b9fe99c6ba4c"}
May 26 16:53:37.060762 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:37.060720    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl" podUID="dffc9457-5179-4dd6-b015-ab504e05acfc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.53:8643/healthz\": dial tcp 10.132.0.53:8643: connect: connection refused"
May 26 16:53:38.171615 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:38.171565    2572 generic.go:358] "Generic (PLEG): container finished" podID="16601799-343b-43b6-af7a-b712f0338895" containerID="2613afbbce3fdef03429c261c8f0d0588b249bbc2c77785fe0b78c829f742b60" exitCode=0
May 26 16:53:38.172075 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:38.171647    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-wg8gk" event={"ID":"16601799-343b-43b6-af7a-b712f0338895","Type":"ContainerDied","Data":"2613afbbce3fdef03429c261c8f0d0588b249bbc2c77785fe0b78c829f742b60"}
May 26 16:53:42.060740 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:42.060694    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl" podUID="dffc9457-5179-4dd6-b015-ab504e05acfc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.53:8643/healthz\": dial tcp 10.132.0.53:8643: connect: connection refused"
May 26 16:53:47.060680 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:47.060454    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl" podUID="dffc9457-5179-4dd6-b015-ab504e05acfc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.53:8643/healthz\": dial tcp 10.132.0.53:8643: connect: connection refused"
May 26 16:53:47.060680 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:47.060674    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl"
May 26 16:53:52.060617 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:52.060536    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl" podUID="dffc9457-5179-4dd6-b015-ab504e05acfc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.53:8643/healthz\": dial tcp 10.132.0.53:8643: connect: connection refused"
May 26 16:53:57.060757 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:53:57.060708    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl" podUID="dffc9457-5179-4dd6-b015-ab504e05acfc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.53:8643/healthz\": dial tcp 10.132.0.53:8643: connect: connection refused"
May 26 16:54:02.060613 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:54:02.060566    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl" podUID="dffc9457-5179-4dd6-b015-ab504e05acfc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.53:8643/healthz\": dial tcp 10.132.0.53:8643: connect: connection refused"
May 26 16:54:04.276913 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:54:04.276853    2572 generic.go:358] "Generic (PLEG): container finished" podID="dffc9457-5179-4dd6-b015-ab504e05acfc" containerID="538202dea96d0e01865d2c1c74babbe10fc111f24e71bc08c67c0d7db4222abe" exitCode=137
May 26 16:54:04.277405 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:54:04.276973    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl" event={"ID":"dffc9457-5179-4dd6-b015-ab504e05acfc","Type":"ContainerDied","Data":"538202dea96d0e01865d2c1c74babbe10fc111f24e71bc08c67c0d7db4222abe"}
May 26 16:54:04.770092 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:54:04.770064    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl"
May 26 16:54:04.904908 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:54:04.904823    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sl6f4\" (UniqueName: \"kubernetes.io/projected/dffc9457-5179-4dd6-b015-ab504e05acfc-kube-api-access-sl6f4\") pod \"dffc9457-5179-4dd6-b015-ab504e05acfc\" (UID: \"dffc9457-5179-4dd6-b015-ab504e05acfc\") "
May 26 16:54:04.904908 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:54:04.904883    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dffc9457-5179-4dd6-b015-ab504e05acfc-kserve-provision-location\") pod \"dffc9457-5179-4dd6-b015-ab504e05acfc\" (UID: \"dffc9457-5179-4dd6-b015-ab504e05acfc\") "
May 26 16:54:04.905179 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:54:04.904918    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dffc9457-5179-4dd6-b015-ab504e05acfc-isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\") pod \"dffc9457-5179-4dd6-b015-ab504e05acfc\" (UID: \"dffc9457-5179-4dd6-b015-ab504e05acfc\") "
May 26 16:54:04.905179 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:54:04.904946    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dffc9457-5179-4dd6-b015-ab504e05acfc-proxy-tls\") pod \"dffc9457-5179-4dd6-b015-ab504e05acfc\" (UID: \"dffc9457-5179-4dd6-b015-ab504e05acfc\") "
May 26 16:54:04.905396 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:54:04.905332    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dffc9457-5179-4dd6-b015-ab504e05acfc-isvc-tensorflow-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-tensorflow-runtime-kube-rbac-proxy-sar-config") pod "dffc9457-5179-4dd6-b015-ab504e05acfc" (UID: "dffc9457-5179-4dd6-b015-ab504e05acfc"). InnerVolumeSpecName "isvc-tensorflow-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:54:04.908480 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:54:04.908446    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dffc9457-5179-4dd6-b015-ab504e05acfc-kube-api-access-sl6f4" (OuterVolumeSpecName: "kube-api-access-sl6f4") pod "dffc9457-5179-4dd6-b015-ab504e05acfc" (UID: "dffc9457-5179-4dd6-b015-ab504e05acfc"). InnerVolumeSpecName "kube-api-access-sl6f4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:54:04.908829 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:54:04.908796    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dffc9457-5179-4dd6-b015-ab504e05acfc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dffc9457-5179-4dd6-b015-ab504e05acfc" (UID: "dffc9457-5179-4dd6-b015-ab504e05acfc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:54:04.908929 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:54:04.908862    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dffc9457-5179-4dd6-b015-ab504e05acfc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dffc9457-5179-4dd6-b015-ab504e05acfc" (UID: "dffc9457-5179-4dd6-b015-ab504e05acfc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:54:05.006438 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:54:05.006399    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sl6f4\" (UniqueName: \"kubernetes.io/projected/dffc9457-5179-4dd6-b015-ab504e05acfc-kube-api-access-sl6f4\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:54:05.006438 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:54:05.006438    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dffc9457-5179-4dd6-b015-ab504e05acfc-kserve-provision-location\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:54:05.006712 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:54:05.006457    2572 reconciler_common.go:299] "Volume detached for volume \"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dffc9457-5179-4dd6-b015-ab504e05acfc-isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:54:05.006712 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:54:05.006472    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dffc9457-5179-4dd6-b015-ab504e05acfc-proxy-tls\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:54:05.282084 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:54:05.282003    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl" event={"ID":"dffc9457-5179-4dd6-b015-ab504e05acfc","Type":"ContainerDied","Data":"a8edff06b3c6d4d0c44d8d8a00fd06d7ac2ca6d5e853195a3c7f9bbabb62fc8b"}
May 26 16:54:05.282084 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:54:05.282054    2572 scope.go:117] "RemoveContainer" containerID="80893d98fdd38fbfa29c1f0512e7d2d1d835008b6cc90935d839b9fe99c6ba4c"
May 26 16:54:05.282633 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:54:05.282102    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl"
May 26 16:54:05.292716 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:54:05.292520    2572 scope.go:117] "RemoveContainer" containerID="538202dea96d0e01865d2c1c74babbe10fc111f24e71bc08c67c0d7db4222abe"
May 26 16:54:05.303196 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:54:05.303172    2572 scope.go:117] "RemoveContainer" containerID="e11457dfefde6edd4e57cbed14ed8abedf69c6dc62603ab81bad73e158353bf1"
May 26 16:54:05.307791 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:54:05.307684    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl"]
May 26 16:54:05.312911 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:54:05.312885    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-8699d78cf-9s8wl"]
May 26 16:54:06.080085 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:54:06.080030    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dffc9457-5179-4dd6-b015-ab504e05acfc" path="/var/lib/kubelet/pods/dffc9457-5179-4dd6-b015-ab504e05acfc/volumes"
May 26 16:55:33.561319 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:33.561277    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-wg8gk" event={"ID":"16601799-343b-43b6-af7a-b712f0338895","Type":"ContainerStarted","Data":"7705b4f5e62a6ce1e6a1b23916e80d18016981838f1723febdcaf8e681b8b07b"}
May 26 16:55:33.561319 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:33.561322    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-wg8gk" event={"ID":"16601799-343b-43b6-af7a-b712f0338895","Type":"ContainerStarted","Data":"980d1ecaae5a0ae814b5f2560fa8082a9b3fcc337386455ec0cb7635d759a06b"}
May 26 16:55:33.561769 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:33.561460    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-wg8gk"
May 26 16:55:33.582878 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:33.582828    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-wg8gk" podStartSLOduration=5.052623082 podStartE2EDuration="1m59.582814371s" podCreationTimestamp="2026-05-26 16:53:34 +0000 UTC" firstStartedPulling="2026-05-26 16:53:38.172822391 +0000 UTC m=+3134.648549719" lastFinishedPulling="2026-05-26 16:55:32.703013681 +0000 UTC m=+3249.178741008" observedRunningTime="2026-05-26 16:55:33.580222803 +0000 UTC m=+3250.055950154" watchObservedRunningTime="2026-05-26 16:55:33.582814371 +0000 UTC m=+3250.058541721"
May 26 16:55:34.564008 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:34.563975    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-wg8gk"
May 26 16:55:34.565073 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:34.565045    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-wg8gk" podUID="16601799-343b-43b6-af7a-b712f0338895" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
May 26 16:55:35.566548 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:35.566511    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-wg8gk" podUID="16601799-343b-43b6-af7a-b712f0338895" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
May 26 16:55:40.571394 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:40.571363    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-wg8gk"
May 26 16:55:40.572194 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:40.572175    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-wg8gk"
May 26 16:55:45.823302 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:45.823269    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-wg8gk"]
May 26 16:55:45.823822 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:45.823722    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-wg8gk" podUID="16601799-343b-43b6-af7a-b712f0338895" containerName="kserve-container" containerID="cri-o://980d1ecaae5a0ae814b5f2560fa8082a9b3fcc337386455ec0cb7635d759a06b" gracePeriod=30
May 26 16:55:45.823822 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:45.823763    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-wg8gk" podUID="16601799-343b-43b6-af7a-b712f0338895" containerName="kube-rbac-proxy" containerID="cri-o://7705b4f5e62a6ce1e6a1b23916e80d18016981838f1723febdcaf8e681b8b07b" gracePeriod=30
May 26 16:55:45.963900 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:45.963868    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-rtqwj"]
May 26 16:55:45.964141 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:45.964129    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dffc9457-5179-4dd6-b015-ab504e05acfc" containerName="kserve-container"
May 26 16:55:45.964187 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:45.964142    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="dffc9457-5179-4dd6-b015-ab504e05acfc" containerName="kserve-container"
May 26 16:55:45.964187 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:45.964154    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dffc9457-5179-4dd6-b015-ab504e05acfc" containerName="kube-rbac-proxy"
May 26 16:55:45.964187 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:45.964160    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="dffc9457-5179-4dd6-b015-ab504e05acfc" containerName="kube-rbac-proxy"
May 26 16:55:45.964187 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:45.964166    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dffc9457-5179-4dd6-b015-ab504e05acfc" containerName="storage-initializer"
May 26 16:55:45.964187 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:45.964173    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="dffc9457-5179-4dd6-b015-ab504e05acfc" containerName="storage-initializer"
May 26 16:55:45.964344 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:45.964213    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="dffc9457-5179-4dd6-b015-ab504e05acfc" containerName="kube-rbac-proxy"
May 26 16:55:45.964344 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:45.964224    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="dffc9457-5179-4dd6-b015-ab504e05acfc" containerName="kserve-container"
May 26 16:55:45.976513 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:45.976490    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-rtqwj"
May 26 16:55:45.976690 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:45.976664    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-rtqwj"]
May 26 16:55:45.978692 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:45.978666    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-predictor-serving-cert\""
May 26 16:55:45.978822 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:45.978713    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\""
May 26 16:55:46.009974 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:46.009942    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e-isvc-xgboost-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-8689c4cfcc-rtqwj\" (UID: \"f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-rtqwj"
May 26 16:55:46.010158 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:46.009986    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6c9j\" (UniqueName: \"kubernetes.io/projected/f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e-kube-api-access-w6c9j\") pod \"isvc-xgboost-predictor-8689c4cfcc-rtqwj\" (UID: \"f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-rtqwj"
May 26 16:55:46.010158 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:46.010046    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e-kserve-provision-location\") pod \"isvc-xgboost-predictor-8689c4cfcc-rtqwj\" (UID: \"f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-rtqwj"
May 26 16:55:46.010158 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:46.010118    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e-proxy-tls\") pod \"isvc-xgboost-predictor-8689c4cfcc-rtqwj\" (UID: \"f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-rtqwj"
May 26 16:55:46.110681 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:46.110575    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e-isvc-xgboost-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-8689c4cfcc-rtqwj\" (UID: \"f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-rtqwj"
May 26 16:55:46.110681 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:46.110664    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-w6c9j\" (UniqueName: \"kubernetes.io/projected/f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e-kube-api-access-w6c9j\") pod \"isvc-xgboost-predictor-8689c4cfcc-rtqwj\" (UID: \"f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-rtqwj"
May 26 16:55:46.110941 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:46.110706    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e-kserve-provision-location\") pod \"isvc-xgboost-predictor-8689c4cfcc-rtqwj\" (UID: \"f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-rtqwj"
May 26 16:55:46.110941 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:46.110746    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e-proxy-tls\") pod \"isvc-xgboost-predictor-8689c4cfcc-rtqwj\" (UID: \"f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-rtqwj"
May 26 16:55:46.110941 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:55:46.110854    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-predictor-serving-cert: secret "isvc-xgboost-predictor-serving-cert" not found
May 26 16:55:46.110941 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:55:46.110910    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e-proxy-tls podName:f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e nodeName:}" failed. No retries permitted until 2026-05-26 16:55:46.610890476 +0000 UTC m=+3263.086617805 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e-proxy-tls") pod "isvc-xgboost-predictor-8689c4cfcc-rtqwj" (UID: "f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e") : secret "isvc-xgboost-predictor-serving-cert" not found
May 26 16:55:46.111151 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:46.111034    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e-kserve-provision-location\") pod \"isvc-xgboost-predictor-8689c4cfcc-rtqwj\" (UID: \"f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-rtqwj"
May 26 16:55:46.111261 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:46.111242    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e-isvc-xgboost-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-8689c4cfcc-rtqwj\" (UID: \"f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-rtqwj"
May 26 16:55:46.119274 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:46.119249    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6c9j\" (UniqueName: \"kubernetes.io/projected/f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e-kube-api-access-w6c9j\") pod \"isvc-xgboost-predictor-8689c4cfcc-rtqwj\" (UID: \"f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-rtqwj"
May 26 16:55:46.597971 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:46.597939    2572 generic.go:358] "Generic (PLEG): container finished" podID="16601799-343b-43b6-af7a-b712f0338895" containerID="7705b4f5e62a6ce1e6a1b23916e80d18016981838f1723febdcaf8e681b8b07b" exitCode=2
May 26 16:55:46.597971 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:46.597982    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-wg8gk" event={"ID":"16601799-343b-43b6-af7a-b712f0338895","Type":"ContainerDied","Data":"7705b4f5e62a6ce1e6a1b23916e80d18016981838f1723febdcaf8e681b8b07b"}
May 26 16:55:46.614468 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:46.614434    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e-proxy-tls\") pod \"isvc-xgboost-predictor-8689c4cfcc-rtqwj\" (UID: \"f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-rtqwj"
May 26 16:55:46.616978 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:46.616946    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e-proxy-tls\") pod \"isvc-xgboost-predictor-8689c4cfcc-rtqwj\" (UID: \"f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-rtqwj"
May 26 16:55:46.887594 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:46.887554    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-rtqwj"
May 26 16:55:47.185409 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:47.185318    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-rtqwj"]
May 26 16:55:47.188898 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:55:47.188869    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0e0fb8c_e6e0_4aa9_ba2f_b03965cbf59e.slice/crio-70f31ade2f0f8a0961bed77da37f098167464344210a028c1b34a8f9de2b8694 WatchSource:0}: Error finding container 70f31ade2f0f8a0961bed77da37f098167464344210a028c1b34a8f9de2b8694: Status 404 returned error can't find the container with id 70f31ade2f0f8a0961bed77da37f098167464344210a028c1b34a8f9de2b8694
May 26 16:55:47.602381 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:47.602299    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-rtqwj" event={"ID":"f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e","Type":"ContainerStarted","Data":"197888ba3d0350394b0948059e6992ac7afc4a3be6aa9737436dd8edd3548b79"}
May 26 16:55:47.602381 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:47.602336    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-rtqwj" event={"ID":"f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e","Type":"ContainerStarted","Data":"70f31ade2f0f8a0961bed77da37f098167464344210a028c1b34a8f9de2b8694"}
May 26 16:55:48.578679 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:48.578655    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-wg8gk"
May 26 16:55:48.608306 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:48.608224    2572 generic.go:358] "Generic (PLEG): container finished" podID="16601799-343b-43b6-af7a-b712f0338895" containerID="980d1ecaae5a0ae814b5f2560fa8082a9b3fcc337386455ec0cb7635d759a06b" exitCode=0
May 26 16:55:48.608306 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:48.608299    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-wg8gk"
May 26 16:55:48.608518 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:48.608300    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-wg8gk" event={"ID":"16601799-343b-43b6-af7a-b712f0338895","Type":"ContainerDied","Data":"980d1ecaae5a0ae814b5f2560fa8082a9b3fcc337386455ec0cb7635d759a06b"}
May 26 16:55:48.608518 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:48.608343    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-wg8gk" event={"ID":"16601799-343b-43b6-af7a-b712f0338895","Type":"ContainerDied","Data":"ee0fae2a2c674d11204369be18227b191047b887295195ca3ff7ad4d440718c2"}
May 26 16:55:48.608518 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:48.608360    2572 scope.go:117] "RemoveContainer" containerID="7705b4f5e62a6ce1e6a1b23916e80d18016981838f1723febdcaf8e681b8b07b"
May 26 16:55:48.615910 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:48.615893    2572 scope.go:117] "RemoveContainer" containerID="980d1ecaae5a0ae814b5f2560fa8082a9b3fcc337386455ec0cb7635d759a06b"
May 26 16:55:48.623100 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:48.623081    2572 scope.go:117] "RemoveContainer" containerID="2613afbbce3fdef03429c261c8f0d0588b249bbc2c77785fe0b78c829f742b60"
May 26 16:55:48.625353 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:48.625331    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-triton-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16601799-343b-43b6-af7a-b712f0338895-isvc-triton-kube-rbac-proxy-sar-config\") pod \"16601799-343b-43b6-af7a-b712f0338895\" (UID: \"16601799-343b-43b6-af7a-b712f0338895\") "
May 26 16:55:48.625463 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:48.625365    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16601799-343b-43b6-af7a-b712f0338895-kserve-provision-location\") pod \"16601799-343b-43b6-af7a-b712f0338895\" (UID: \"16601799-343b-43b6-af7a-b712f0338895\") "
May 26 16:55:48.625463 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:48.625422    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5d8xz\" (UniqueName: \"kubernetes.io/projected/16601799-343b-43b6-af7a-b712f0338895-kube-api-access-5d8xz\") pod \"16601799-343b-43b6-af7a-b712f0338895\" (UID: \"16601799-343b-43b6-af7a-b712f0338895\") "
May 26 16:55:48.625463 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:48.625443    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16601799-343b-43b6-af7a-b712f0338895-proxy-tls\") pod \"16601799-343b-43b6-af7a-b712f0338895\" (UID: \"16601799-343b-43b6-af7a-b712f0338895\") "
May 26 16:55:48.625858 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:48.625825    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16601799-343b-43b6-af7a-b712f0338895-isvc-triton-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-triton-kube-rbac-proxy-sar-config") pod "16601799-343b-43b6-af7a-b712f0338895" (UID: "16601799-343b-43b6-af7a-b712f0338895"). InnerVolumeSpecName "isvc-triton-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:55:48.625996 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:48.625968    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16601799-343b-43b6-af7a-b712f0338895-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "16601799-343b-43b6-af7a-b712f0338895" (UID: "16601799-343b-43b6-af7a-b712f0338895"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:55:48.627993 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:48.627965    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16601799-343b-43b6-af7a-b712f0338895-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "16601799-343b-43b6-af7a-b712f0338895" (UID: "16601799-343b-43b6-af7a-b712f0338895"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:55:48.628078 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:48.627988    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16601799-343b-43b6-af7a-b712f0338895-kube-api-access-5d8xz" (OuterVolumeSpecName: "kube-api-access-5d8xz") pod "16601799-343b-43b6-af7a-b712f0338895" (UID: "16601799-343b-43b6-af7a-b712f0338895"). InnerVolumeSpecName "kube-api-access-5d8xz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:55:48.630131 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:48.630109    2572 scope.go:117] "RemoveContainer" containerID="7705b4f5e62a6ce1e6a1b23916e80d18016981838f1723febdcaf8e681b8b07b"
May 26 16:55:48.630404 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:55:48.630380    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7705b4f5e62a6ce1e6a1b23916e80d18016981838f1723febdcaf8e681b8b07b\": container with ID starting with 7705b4f5e62a6ce1e6a1b23916e80d18016981838f1723febdcaf8e681b8b07b not found: ID does not exist" containerID="7705b4f5e62a6ce1e6a1b23916e80d18016981838f1723febdcaf8e681b8b07b"
May 26 16:55:48.630485 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:48.630412    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7705b4f5e62a6ce1e6a1b23916e80d18016981838f1723febdcaf8e681b8b07b"} err="failed to get container status \"7705b4f5e62a6ce1e6a1b23916e80d18016981838f1723febdcaf8e681b8b07b\": rpc error: code = NotFound desc = could not find container \"7705b4f5e62a6ce1e6a1b23916e80d18016981838f1723febdcaf8e681b8b07b\": container with ID starting with 7705b4f5e62a6ce1e6a1b23916e80d18016981838f1723febdcaf8e681b8b07b not found: ID does not exist"
May 26 16:55:48.630485 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:48.630432    2572 scope.go:117] "RemoveContainer" containerID="980d1ecaae5a0ae814b5f2560fa8082a9b3fcc337386455ec0cb7635d759a06b"
May 26 16:55:48.630712 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:55:48.630694    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"980d1ecaae5a0ae814b5f2560fa8082a9b3fcc337386455ec0cb7635d759a06b\": container with ID starting with 980d1ecaae5a0ae814b5f2560fa8082a9b3fcc337386455ec0cb7635d759a06b not found: ID does not exist" containerID="980d1ecaae5a0ae814b5f2560fa8082a9b3fcc337386455ec0cb7635d759a06b"
May 26 16:55:48.630780 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:48.630737    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"980d1ecaae5a0ae814b5f2560fa8082a9b3fcc337386455ec0cb7635d759a06b"} err="failed to get container status \"980d1ecaae5a0ae814b5f2560fa8082a9b3fcc337386455ec0cb7635d759a06b\": rpc error: code = NotFound desc = could not find container \"980d1ecaae5a0ae814b5f2560fa8082a9b3fcc337386455ec0cb7635d759a06b\": container with ID starting with 980d1ecaae5a0ae814b5f2560fa8082a9b3fcc337386455ec0cb7635d759a06b not found: ID does not exist"
May 26 16:55:48.630780 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:48.630752    2572 scope.go:117] "RemoveContainer" containerID="2613afbbce3fdef03429c261c8f0d0588b249bbc2c77785fe0b78c829f742b60"
May 26 16:55:48.631008 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:55:48.630987    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2613afbbce3fdef03429c261c8f0d0588b249bbc2c77785fe0b78c829f742b60\": container with ID starting with 2613afbbce3fdef03429c261c8f0d0588b249bbc2c77785fe0b78c829f742b60 not found: ID does not exist" containerID="2613afbbce3fdef03429c261c8f0d0588b249bbc2c77785fe0b78c829f742b60"
May 26 16:55:48.631084 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:48.631012    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2613afbbce3fdef03429c261c8f0d0588b249bbc2c77785fe0b78c829f742b60"} err="failed to get container status \"2613afbbce3fdef03429c261c8f0d0588b249bbc2c77785fe0b78c829f742b60\": rpc error: code = NotFound desc = could not find container \"2613afbbce3fdef03429c261c8f0d0588b249bbc2c77785fe0b78c829f742b60\": container with ID starting with 2613afbbce3fdef03429c261c8f0d0588b249bbc2c77785fe0b78c829f742b60 not found: ID does not exist"
May 26 16:55:48.726455 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:48.726419    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d8xz\" (UniqueName: \"kubernetes.io/projected/16601799-343b-43b6-af7a-b712f0338895-kube-api-access-5d8xz\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:55:48.726455 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:48.726450    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16601799-343b-43b6-af7a-b712f0338895-proxy-tls\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:55:48.726455 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:48.726464    2572 reconciler_common.go:299] "Volume detached for volume \"isvc-triton-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16601799-343b-43b6-af7a-b712f0338895-isvc-triton-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:55:48.726754 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:48.726473    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16601799-343b-43b6-af7a-b712f0338895-kserve-provision-location\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:55:48.929382 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:48.929345    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-wg8gk"]
May 26 16:55:48.933402 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:48.933376    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-84bb65d94b-wg8gk"]
May 26 16:55:50.079453 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:50.079421    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16601799-343b-43b6-af7a-b712f0338895" path="/var/lib/kubelet/pods/16601799-343b-43b6-af7a-b712f0338895/volumes"
May 26 16:55:51.618725 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:51.618693    2572 generic.go:358] "Generic (PLEG): container finished" podID="f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e" containerID="197888ba3d0350394b0948059e6992ac7afc4a3be6aa9737436dd8edd3548b79" exitCode=0
May 26 16:55:51.619189 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:55:51.618745    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-rtqwj" event={"ID":"f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e","Type":"ContainerDied","Data":"197888ba3d0350394b0948059e6992ac7afc4a3be6aa9737436dd8edd3548b79"}
May 26 16:56:11.163361 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:56:11.163337    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 16:56:11.677729 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:56:11.677689    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-rtqwj" event={"ID":"f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e","Type":"ContainerStarted","Data":"5e47c8b63e090236bf02439d4df825ce40857ac59e4651638b27061468aab865"}
May 26 16:56:11.677729 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:56:11.677732    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-rtqwj" event={"ID":"f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e","Type":"ContainerStarted","Data":"b3a4b9bfa35af7a8fbc5b9cf0102301ae00ebb1b2c8fff2e8b34db4a1bb093f8"}
May 26 16:56:11.678047 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:56:11.678029    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-rtqwj"
May 26 16:56:11.701053 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:56:11.701004    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-rtqwj" podStartSLOduration=7.29503516 podStartE2EDuration="26.700986715s" podCreationTimestamp="2026-05-26 16:55:45 +0000 UTC" firstStartedPulling="2026-05-26 16:55:51.620031876 +0000 UTC m=+3268.095759204" lastFinishedPulling="2026-05-26 16:56:11.025983416 +0000 UTC m=+3287.501710759" observedRunningTime="2026-05-26 16:56:11.699907306 +0000 UTC m=+3288.175634656" watchObservedRunningTime="2026-05-26 16:56:11.700986715 +0000 UTC m=+3288.176714065"
May 26 16:56:12.680845 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:56:12.680809    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-rtqwj"
May 26 16:56:12.681813 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:56:12.681787    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-rtqwj" podUID="f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
May 26 16:56:13.683748 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:56:13.683711    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-rtqwj" podUID="f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
May 26 16:56:18.687582 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:56:18.687552    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-rtqwj"
May 26 16:56:18.688121 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:56:18.688089    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-rtqwj" podUID="f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
May 26 16:56:28.688088 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:56:28.688050    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-rtqwj" podUID="f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
May 26 16:56:38.688489 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:56:38.688445    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-rtqwj" podUID="f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
May 26 16:56:48.688386 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:56:48.688306    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-rtqwj" podUID="f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
May 26 16:56:58.688338 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:56:58.688299    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-rtqwj" podUID="f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
May 26 16:57:08.688582 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:08.688540    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-rtqwj" podUID="f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
May 26 16:57:18.688754 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:18.688722    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-rtqwj"
May 26 16:57:25.996747 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:25.996708    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-rtqwj"]
May 26 16:57:25.997251 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:25.997036    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-rtqwj" podUID="f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e" containerName="kserve-container" containerID="cri-o://b3a4b9bfa35af7a8fbc5b9cf0102301ae00ebb1b2c8fff2e8b34db4a1bb093f8" gracePeriod=30
May 26 16:57:25.997251 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:25.997091    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-rtqwj" podUID="f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e" containerName="kube-rbac-proxy" containerID="cri-o://5e47c8b63e090236bf02439d4df825ce40857ac59e4651638b27061468aab865" gracePeriod=30
May 26 16:57:26.109696 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:26.109656    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd"]
May 26 16:57:26.109944 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:26.109932    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16601799-343b-43b6-af7a-b712f0338895" containerName="kserve-container"
May 26 16:57:26.109991 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:26.109946    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="16601799-343b-43b6-af7a-b712f0338895" containerName="kserve-container"
May 26 16:57:26.109991 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:26.109959    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16601799-343b-43b6-af7a-b712f0338895" containerName="kube-rbac-proxy"
May 26 16:57:26.109991 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:26.109966    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="16601799-343b-43b6-af7a-b712f0338895" containerName="kube-rbac-proxy"
May 26 16:57:26.109991 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:26.109975    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16601799-343b-43b6-af7a-b712f0338895" containerName="storage-initializer"
May 26 16:57:26.109991 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:26.109981    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="16601799-343b-43b6-af7a-b712f0338895" containerName="storage-initializer"
May 26 16:57:26.110169 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:26.110029    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="16601799-343b-43b6-af7a-b712f0338895" containerName="kserve-container"
May 26 16:57:26.110169 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:26.110039    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="16601799-343b-43b6-af7a-b712f0338895" containerName="kube-rbac-proxy"
May 26 16:57:26.111906 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:26.111888    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd"
May 26 16:57:26.114002 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:26.113960    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
May 26 16:57:26.114138 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:26.114096    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
May 26 16:57:26.121918 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:26.121895    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd"]
May 26 16:57:26.238034 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:26.238000    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39598be1-1d31-4551-be5a-34d544c84b7e-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd\" (UID: \"39598be1-1d31-4551-be5a-34d544c84b7e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd"
May 26 16:57:26.238034 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:26.238049    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39598be1-1d31-4551-be5a-34d544c84b7e-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd\" (UID: \"39598be1-1d31-4551-be5a-34d544c84b7e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd"
May 26 16:57:26.238308 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:26.238105    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39598be1-1d31-4551-be5a-34d544c84b7e-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd\" (UID: \"39598be1-1d31-4551-be5a-34d544c84b7e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd"
May 26 16:57:26.238308 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:26.238123    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh6zx\" (UniqueName: \"kubernetes.io/projected/39598be1-1d31-4551-be5a-34d544c84b7e-kube-api-access-vh6zx\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd\" (UID: \"39598be1-1d31-4551-be5a-34d544c84b7e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd"
May 26 16:57:26.339338 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:26.339225    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39598be1-1d31-4551-be5a-34d544c84b7e-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd\" (UID: \"39598be1-1d31-4551-be5a-34d544c84b7e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd"
May 26 16:57:26.339338 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:26.339267    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vh6zx\" (UniqueName: \"kubernetes.io/projected/39598be1-1d31-4551-be5a-34d544c84b7e-kube-api-access-vh6zx\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd\" (UID: \"39598be1-1d31-4551-be5a-34d544c84b7e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd"
May 26 16:57:26.339338 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:26.339298    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39598be1-1d31-4551-be5a-34d544c84b7e-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd\" (UID: \"39598be1-1d31-4551-be5a-34d544c84b7e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd"
May 26 16:57:26.339338 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:26.339333    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39598be1-1d31-4551-be5a-34d544c84b7e-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd\" (UID: \"39598be1-1d31-4551-be5a-34d544c84b7e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd"
May 26 16:57:26.339834 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:26.339810    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39598be1-1d31-4551-be5a-34d544c84b7e-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd\" (UID: \"39598be1-1d31-4551-be5a-34d544c84b7e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd"
May 26 16:57:26.340075 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:26.340054    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39598be1-1d31-4551-be5a-34d544c84b7e-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd\" (UID: \"39598be1-1d31-4551-be5a-34d544c84b7e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd"
May 26 16:57:26.341729 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:26.341707    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39598be1-1d31-4551-be5a-34d544c84b7e-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd\" (UID: \"39598be1-1d31-4551-be5a-34d544c84b7e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd"
May 26 16:57:26.347488 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:26.347467    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh6zx\" (UniqueName: \"kubernetes.io/projected/39598be1-1d31-4551-be5a-34d544c84b7e-kube-api-access-vh6zx\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd\" (UID: \"39598be1-1d31-4551-be5a-34d544c84b7e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd"
May 26 16:57:26.422591 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:26.422548    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd"
May 26 16:57:26.551467 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:26.551277    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd"]
May 26 16:57:26.554053 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:57:26.554024    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39598be1_1d31_4551_be5a_34d544c84b7e.slice/crio-7306923bb848fb938f8f94e3ac592bd853562ea5a6b3eaa033a86620a02c751a WatchSource:0}: Error finding container 7306923bb848fb938f8f94e3ac592bd853562ea5a6b3eaa033a86620a02c751a: Status 404 returned error can't find the container with id 7306923bb848fb938f8f94e3ac592bd853562ea5a6b3eaa033a86620a02c751a
May 26 16:57:26.872047 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:26.872016    2572 generic.go:358] "Generic (PLEG): container finished" podID="f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e" containerID="5e47c8b63e090236bf02439d4df825ce40857ac59e4651638b27061468aab865" exitCode=2
May 26 16:57:26.872264 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:26.872102    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-rtqwj" event={"ID":"f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e","Type":"ContainerDied","Data":"5e47c8b63e090236bf02439d4df825ce40857ac59e4651638b27061468aab865"}
May 26 16:57:26.873364 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:26.873342    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd" event={"ID":"39598be1-1d31-4551-be5a-34d544c84b7e","Type":"ContainerStarted","Data":"7d42d1a9d9339d47d9835e2ab4fe532c99cb4fdf3ce8ea26f21e4093999a7009"}
May 26 16:57:26.873503 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:26.873369    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd" event={"ID":"39598be1-1d31-4551-be5a-34d544c84b7e","Type":"ContainerStarted","Data":"7306923bb848fb938f8f94e3ac592bd853562ea5a6b3eaa033a86620a02c751a"}
May 26 16:57:28.684821 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:28.684782    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-rtqwj" podUID="f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.55:8643/healthz\": dial tcp 10.132.0.55:8643: connect: connection refused"
May 26 16:57:28.688085 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:28.688057    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-rtqwj" podUID="f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
May 26 16:57:29.761839 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:29.761814    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-rtqwj"
May 26 16:57:29.867692 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:29.867662    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e-kserve-provision-location\") pod \"f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e\" (UID: \"f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e\") "
May 26 16:57:29.867845 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:29.867708    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e-isvc-xgboost-kube-rbac-proxy-sar-config\") pod \"f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e\" (UID: \"f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e\") "
May 26 16:57:29.867845 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:29.867745    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6c9j\" (UniqueName: \"kubernetes.io/projected/f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e-kube-api-access-w6c9j\") pod \"f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e\" (UID: \"f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e\") "
May 26 16:57:29.867845 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:29.867787    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e-proxy-tls\") pod \"f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e\" (UID: \"f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e\") "
May 26 16:57:29.868037 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:29.868015    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e" (UID: "f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:57:29.868125 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:29.868093    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e-isvc-xgboost-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-kube-rbac-proxy-sar-config") pod "f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e" (UID: "f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e"). InnerVolumeSpecName "isvc-xgboost-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:57:29.869808 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:29.869780    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e-kube-api-access-w6c9j" (OuterVolumeSpecName: "kube-api-access-w6c9j") pod "f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e" (UID: "f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e"). InnerVolumeSpecName "kube-api-access-w6c9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:57:29.869947 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:29.869924    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e" (UID: "f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:57:29.884062 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:29.884035    2572 generic.go:358] "Generic (PLEG): container finished" podID="f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e" containerID="b3a4b9bfa35af7a8fbc5b9cf0102301ae00ebb1b2c8fff2e8b34db4a1bb093f8" exitCode=0
May 26 16:57:29.884204 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:29.884077    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-rtqwj" event={"ID":"f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e","Type":"ContainerDied","Data":"b3a4b9bfa35af7a8fbc5b9cf0102301ae00ebb1b2c8fff2e8b34db4a1bb093f8"}
May 26 16:57:29.884204 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:29.884101    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-rtqwj" event={"ID":"f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e","Type":"ContainerDied","Data":"70f31ade2f0f8a0961bed77da37f098167464344210a028c1b34a8f9de2b8694"}
May 26 16:57:29.884204 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:29.884116    2572 scope.go:117] "RemoveContainer" containerID="5e47c8b63e090236bf02439d4df825ce40857ac59e4651638b27061468aab865"
May 26 16:57:29.884204 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:29.884116    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-rtqwj"
May 26 16:57:29.891825 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:29.891679    2572 scope.go:117] "RemoveContainer" containerID="b3a4b9bfa35af7a8fbc5b9cf0102301ae00ebb1b2c8fff2e8b34db4a1bb093f8"
May 26 16:57:29.898900 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:29.898885    2572 scope.go:117] "RemoveContainer" containerID="197888ba3d0350394b0948059e6992ac7afc4a3be6aa9737436dd8edd3548b79"
May 26 16:57:29.904391 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:29.904369    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-rtqwj"]
May 26 16:57:29.906248 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:29.906235    2572 scope.go:117] "RemoveContainer" containerID="5e47c8b63e090236bf02439d4df825ce40857ac59e4651638b27061468aab865"
May 26 16:57:29.906514 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:57:29.906496    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e47c8b63e090236bf02439d4df825ce40857ac59e4651638b27061468aab865\": container with ID starting with 5e47c8b63e090236bf02439d4df825ce40857ac59e4651638b27061468aab865 not found: ID does not exist" containerID="5e47c8b63e090236bf02439d4df825ce40857ac59e4651638b27061468aab865"
May 26 16:57:29.906559 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:29.906523    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e47c8b63e090236bf02439d4df825ce40857ac59e4651638b27061468aab865"} err="failed to get container status \"5e47c8b63e090236bf02439d4df825ce40857ac59e4651638b27061468aab865\": rpc error: code = NotFound desc = could not find container \"5e47c8b63e090236bf02439d4df825ce40857ac59e4651638b27061468aab865\": container with ID starting with 5e47c8b63e090236bf02439d4df825ce40857ac59e4651638b27061468aab865 not found: ID does not exist"
May 26 16:57:29.906559 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:29.906542    2572 scope.go:117] "RemoveContainer" containerID="b3a4b9bfa35af7a8fbc5b9cf0102301ae00ebb1b2c8fff2e8b34db4a1bb093f8"
May 26 16:57:29.906805 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:57:29.906788    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3a4b9bfa35af7a8fbc5b9cf0102301ae00ebb1b2c8fff2e8b34db4a1bb093f8\": container with ID starting with b3a4b9bfa35af7a8fbc5b9cf0102301ae00ebb1b2c8fff2e8b34db4a1bb093f8 not found: ID does not exist" containerID="b3a4b9bfa35af7a8fbc5b9cf0102301ae00ebb1b2c8fff2e8b34db4a1bb093f8"
May 26 16:57:29.906852 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:29.906811    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3a4b9bfa35af7a8fbc5b9cf0102301ae00ebb1b2c8fff2e8b34db4a1bb093f8"} err="failed to get container status \"b3a4b9bfa35af7a8fbc5b9cf0102301ae00ebb1b2c8fff2e8b34db4a1bb093f8\": rpc error: code = NotFound desc = could not find container \"b3a4b9bfa35af7a8fbc5b9cf0102301ae00ebb1b2c8fff2e8b34db4a1bb093f8\": container with ID starting with b3a4b9bfa35af7a8fbc5b9cf0102301ae00ebb1b2c8fff2e8b34db4a1bb093f8 not found: ID does not exist"
May 26 16:57:29.906852 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:29.906832    2572 scope.go:117] "RemoveContainer" containerID="197888ba3d0350394b0948059e6992ac7afc4a3be6aa9737436dd8edd3548b79"
May 26 16:57:29.907050 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:57:29.907034    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"197888ba3d0350394b0948059e6992ac7afc4a3be6aa9737436dd8edd3548b79\": container with ID starting with 197888ba3d0350394b0948059e6992ac7afc4a3be6aa9737436dd8edd3548b79 not found: ID does not exist" containerID="197888ba3d0350394b0948059e6992ac7afc4a3be6aa9737436dd8edd3548b79"
May 26 16:57:29.907092 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:29.907057    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"197888ba3d0350394b0948059e6992ac7afc4a3be6aa9737436dd8edd3548b79"} err="failed to get container status \"197888ba3d0350394b0948059e6992ac7afc4a3be6aa9737436dd8edd3548b79\": rpc error: code = NotFound desc = could not find container \"197888ba3d0350394b0948059e6992ac7afc4a3be6aa9737436dd8edd3548b79\": container with ID starting with 197888ba3d0350394b0948059e6992ac7afc4a3be6aa9737436dd8edd3548b79 not found: ID does not exist"
May 26 16:57:29.909888 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:29.909861    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-8689c4cfcc-rtqwj"]
May 26 16:57:29.968825 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:29.968791    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e-kserve-provision-location\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:57:29.968825 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:29.968821    2572 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e-isvc-xgboost-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:57:29.968825 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:29.968833    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6c9j\" (UniqueName: \"kubernetes.io/projected/f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e-kube-api-access-w6c9j\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:57:29.969042 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:29.968842    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e-proxy-tls\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:57:30.079080 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:30.079046    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e" path="/var/lib/kubelet/pods/f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e/volumes"
May 26 16:57:30.887732 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:30.887701    2572 generic.go:358] "Generic (PLEG): container finished" podID="39598be1-1d31-4551-be5a-34d544c84b7e" containerID="7d42d1a9d9339d47d9835e2ab4fe532c99cb4fdf3ce8ea26f21e4093999a7009" exitCode=0
May 26 16:57:30.888178 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:30.887772    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd" event={"ID":"39598be1-1d31-4551-be5a-34d544c84b7e","Type":"ContainerDied","Data":"7d42d1a9d9339d47d9835e2ab4fe532c99cb4fdf3ce8ea26f21e4093999a7009"}
May 26 16:57:31.892690 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:31.892653    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd" event={"ID":"39598be1-1d31-4551-be5a-34d544c84b7e","Type":"ContainerStarted","Data":"24e2d270ac31f2144e8bd503187ec85f6c3d8fec07958344db9bb450f6f012ae"}
May 26 16:57:31.892690 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:31.892693    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd" event={"ID":"39598be1-1d31-4551-be5a-34d544c84b7e","Type":"ContainerStarted","Data":"fb076cb25f6e8e9e90c116c26e6a2235f4725d5b6d2e1024c7e8765880631b5c"}
May 26 16:57:31.893118 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:31.892888    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd"
May 26 16:57:31.915128 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:31.915072    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd" podStartSLOduration=5.915059328 podStartE2EDuration="5.915059328s" podCreationTimestamp="2026-05-26 16:57:26 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:57:31.913422695 +0000 UTC m=+3368.389150046" watchObservedRunningTime="2026-05-26 16:57:31.915059328 +0000 UTC m=+3368.390786720"
May 26 16:57:32.895036 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:32.895004    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd"
May 26 16:57:38.903807 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:57:38.903778    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd"
May 26 16:58:08.907218 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:08.907189    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd"
May 26 16:58:16.160803 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:16.160708    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd"]
May 26 16:58:16.161248 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:16.161146    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd" podUID="39598be1-1d31-4551-be5a-34d544c84b7e" containerName="kserve-container" containerID="cri-o://fb076cb25f6e8e9e90c116c26e6a2235f4725d5b6d2e1024c7e8765880631b5c" gracePeriod=30
May 26 16:58:16.161341 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:16.161177    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd" podUID="39598be1-1d31-4551-be5a-34d544c84b7e" containerName="kube-rbac-proxy" containerID="cri-o://24e2d270ac31f2144e8bd503187ec85f6c3d8fec07958344db9bb450f6f012ae" gracePeriod=30
May 26 16:58:16.233251 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:16.233217    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-ppp6x"]
May 26 16:58:16.233486 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:16.233475    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e" containerName="kserve-container"
May 26 16:58:16.233538 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:16.233488    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e" containerName="kserve-container"
May 26 16:58:16.233538 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:16.233527    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e" containerName="kube-rbac-proxy"
May 26 16:58:16.233538 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:16.233533    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e" containerName="kube-rbac-proxy"
May 26 16:58:16.233694 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:16.233542    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e" containerName="storage-initializer"
May 26 16:58:16.233694 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:16.233547    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e" containerName="storage-initializer"
May 26 16:58:16.233694 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:16.233586    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e" containerName="kserve-container"
May 26 16:58:16.233694 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:16.233594    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="f0e0fb8c-e6e0-4aa9-ba2f-b03965cbf59e" containerName="kube-rbac-proxy"
May 26 16:58:16.235682 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:16.235657    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-ppp6x"
May 26 16:58:16.237919 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:16.237892    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
May 26 16:58:16.238061 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:16.237960    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
May 26 16:58:16.245712 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:16.245686    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-ppp6x"]
May 26 16:58:16.273971 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:16.273887    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjbpg\" (UniqueName: \"kubernetes.io/projected/0f31e5d5-9f66-4b20-ae0b-f932fcc63102-kube-api-access-vjbpg\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-ppp6x\" (UID: \"0f31e5d5-9f66-4b20-ae0b-f932fcc63102\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-ppp6x"
May 26 16:58:16.273971 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:16.273940    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f31e5d5-9f66-4b20-ae0b-f932fcc63102-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-ppp6x\" (UID: \"0f31e5d5-9f66-4b20-ae0b-f932fcc63102\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-ppp6x"
May 26 16:58:16.274271 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:16.274002    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f31e5d5-9f66-4b20-ae0b-f932fcc63102-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-ppp6x\" (UID: \"0f31e5d5-9f66-4b20-ae0b-f932fcc63102\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-ppp6x"
May 26 16:58:16.274271 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:16.274035    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f31e5d5-9f66-4b20-ae0b-f932fcc63102-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-ppp6x\" (UID: \"0f31e5d5-9f66-4b20-ae0b-f932fcc63102\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-ppp6x"
May 26 16:58:16.374740 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:16.374703    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f31e5d5-9f66-4b20-ae0b-f932fcc63102-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-ppp6x\" (UID: \"0f31e5d5-9f66-4b20-ae0b-f932fcc63102\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-ppp6x"
May 26 16:58:16.374923 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:16.374752    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f31e5d5-9f66-4b20-ae0b-f932fcc63102-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-ppp6x\" (UID: \"0f31e5d5-9f66-4b20-ae0b-f932fcc63102\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-ppp6x"
May 26 16:58:16.374923 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:58:16.374843    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-serving-cert: secret "xgboost-v2-mlserver-predictor-serving-cert" not found
May 26 16:58:16.374923 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:16.374867    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f31e5d5-9f66-4b20-ae0b-f932fcc63102-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-ppp6x\" (UID: \"0f31e5d5-9f66-4b20-ae0b-f932fcc63102\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-ppp6x"
May 26 16:58:16.374923 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:58:16.374898    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f31e5d5-9f66-4b20-ae0b-f932fcc63102-proxy-tls podName:0f31e5d5-9f66-4b20-ae0b-f932fcc63102 nodeName:}" failed. No retries permitted until 2026-05-26 16:58:16.874877092 +0000 UTC m=+3413.350604419 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0f31e5d5-9f66-4b20-ae0b-f932fcc63102-proxy-tls") pod "xgboost-v2-mlserver-predictor-7799869d6f-ppp6x" (UID: "0f31e5d5-9f66-4b20-ae0b-f932fcc63102") : secret "xgboost-v2-mlserver-predictor-serving-cert" not found
May 26 16:58:16.375097 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:16.374993    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vjbpg\" (UniqueName: \"kubernetes.io/projected/0f31e5d5-9f66-4b20-ae0b-f932fcc63102-kube-api-access-vjbpg\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-ppp6x\" (UID: \"0f31e5d5-9f66-4b20-ae0b-f932fcc63102\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-ppp6x"
May 26 16:58:16.375211 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:16.375192    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f31e5d5-9f66-4b20-ae0b-f932fcc63102-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-ppp6x\" (UID: \"0f31e5d5-9f66-4b20-ae0b-f932fcc63102\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-ppp6x"
May 26 16:58:16.375444 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:16.375427    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f31e5d5-9f66-4b20-ae0b-f932fcc63102-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-ppp6x\" (UID: \"0f31e5d5-9f66-4b20-ae0b-f932fcc63102\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-ppp6x"
May 26 16:58:16.383976 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:16.383951    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjbpg\" (UniqueName: \"kubernetes.io/projected/0f31e5d5-9f66-4b20-ae0b-f932fcc63102-kube-api-access-vjbpg\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-ppp6x\" (UID: \"0f31e5d5-9f66-4b20-ae0b-f932fcc63102\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-ppp6x"
May 26 16:58:16.878016 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:16.877976    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f31e5d5-9f66-4b20-ae0b-f932fcc63102-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-ppp6x\" (UID: \"0f31e5d5-9f66-4b20-ae0b-f932fcc63102\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-ppp6x"
May 26 16:58:16.880342 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:16.880324    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f31e5d5-9f66-4b20-ae0b-f932fcc63102-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-ppp6x\" (UID: \"0f31e5d5-9f66-4b20-ae0b-f932fcc63102\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-ppp6x"
May 26 16:58:17.013735 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:17.013697    2572 generic.go:358] "Generic (PLEG): container finished" podID="39598be1-1d31-4551-be5a-34d544c84b7e" containerID="24e2d270ac31f2144e8bd503187ec85f6c3d8fec07958344db9bb450f6f012ae" exitCode=2
May 26 16:58:17.013929 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:17.013751    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd" event={"ID":"39598be1-1d31-4551-be5a-34d544c84b7e","Type":"ContainerDied","Data":"24e2d270ac31f2144e8bd503187ec85f6c3d8fec07958344db9bb450f6f012ae"}
May 26 16:58:17.147173 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:17.147081    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-ppp6x"
May 26 16:58:17.274789 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:17.274577    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-ppp6x"]
May 26 16:58:17.277383 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:58:17.277354    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f31e5d5_9f66_4b20_ae0b_f932fcc63102.slice/crio-7d335cfbfc7211840d1d0d6292f2f36fee24b12563f640c7d8f0c48f3dff513d WatchSource:0}: Error finding container 7d335cfbfc7211840d1d0d6292f2f36fee24b12563f640c7d8f0c48f3dff513d: Status 404 returned error can't find the container with id 7d335cfbfc7211840d1d0d6292f2f36fee24b12563f640c7d8f0c48f3dff513d
May 26 16:58:18.018195 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:18.018158    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-ppp6x" event={"ID":"0f31e5d5-9f66-4b20-ae0b-f932fcc63102","Type":"ContainerStarted","Data":"dde7fbcb11439cc0cade95ada5f175890cf14114657a0a8880dab7336a061172"}
May 26 16:58:18.018195 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:18.018196    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-ppp6x" event={"ID":"0f31e5d5-9f66-4b20-ae0b-f932fcc63102","Type":"ContainerStarted","Data":"7d335cfbfc7211840d1d0d6292f2f36fee24b12563f640c7d8f0c48f3dff513d"}
May 26 16:58:18.898732 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:18.898682    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd" podUID="39598be1-1d31-4551-be5a-34d544c84b7e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.56:8643/healthz\": dial tcp 10.132.0.56:8643: connect: connection refused"
May 26 16:58:18.905188 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:18.905159    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd" podUID="39598be1-1d31-4551-be5a-34d544c84b7e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.56:8080/v2/models/isvc-xgboost-v2-mlserver/ready\": dial tcp 10.132.0.56:8080: connect: connection refused"
May 26 16:58:22.029418 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:22.029383    2572 generic.go:358] "Generic (PLEG): container finished" podID="0f31e5d5-9f66-4b20-ae0b-f932fcc63102" containerID="dde7fbcb11439cc0cade95ada5f175890cf14114657a0a8880dab7336a061172" exitCode=0
May 26 16:58:22.029418 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:22.029422    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-ppp6x" event={"ID":"0f31e5d5-9f66-4b20-ae0b-f932fcc63102","Type":"ContainerDied","Data":"dde7fbcb11439cc0cade95ada5f175890cf14114657a0a8880dab7336a061172"}
May 26 16:58:22.906570 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:22.906546    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd"
May 26 16:58:22.926570 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:22.926500    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39598be1-1d31-4551-be5a-34d544c84b7e-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"39598be1-1d31-4551-be5a-34d544c84b7e\" (UID: \"39598be1-1d31-4551-be5a-34d544c84b7e\") "
May 26 16:58:22.926987 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:22.926812    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39598be1-1d31-4551-be5a-34d544c84b7e-kserve-provision-location\") pod \"39598be1-1d31-4551-be5a-34d544c84b7e\" (UID: \"39598be1-1d31-4551-be5a-34d544c84b7e\") "
May 26 16:58:22.926987 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:22.926861    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39598be1-1d31-4551-be5a-34d544c84b7e-proxy-tls\") pod \"39598be1-1d31-4551-be5a-34d544c84b7e\" (UID: \"39598be1-1d31-4551-be5a-34d544c84b7e\") "
May 26 16:58:22.926987 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:22.926873    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39598be1-1d31-4551-be5a-34d544c84b7e-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config") pod "39598be1-1d31-4551-be5a-34d544c84b7e" (UID: "39598be1-1d31-4551-be5a-34d544c84b7e"). InnerVolumeSpecName "isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:58:22.926987 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:22.926901    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vh6zx\" (UniqueName: \"kubernetes.io/projected/39598be1-1d31-4551-be5a-34d544c84b7e-kube-api-access-vh6zx\") pod \"39598be1-1d31-4551-be5a-34d544c84b7e\" (UID: \"39598be1-1d31-4551-be5a-34d544c84b7e\") "
May 26 16:58:22.927370 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:22.927139    2572 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39598be1-1d31-4551-be5a-34d544c84b7e-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:58:22.927370 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:22.927310    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39598be1-1d31-4551-be5a-34d544c84b7e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "39598be1-1d31-4551-be5a-34d544c84b7e" (UID: "39598be1-1d31-4551-be5a-34d544c84b7e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:58:22.930277 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:22.930019    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39598be1-1d31-4551-be5a-34d544c84b7e-kube-api-access-vh6zx" (OuterVolumeSpecName: "kube-api-access-vh6zx") pod "39598be1-1d31-4551-be5a-34d544c84b7e" (UID: "39598be1-1d31-4551-be5a-34d544c84b7e"). InnerVolumeSpecName "kube-api-access-vh6zx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:58:22.930408 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:22.930197    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39598be1-1d31-4551-be5a-34d544c84b7e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "39598be1-1d31-4551-be5a-34d544c84b7e" (UID: "39598be1-1d31-4551-be5a-34d544c84b7e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:58:23.027665 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:23.027543    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh6zx\" (UniqueName: \"kubernetes.io/projected/39598be1-1d31-4551-be5a-34d544c84b7e-kube-api-access-vh6zx\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:58:23.027665 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:23.027578    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39598be1-1d31-4551-be5a-34d544c84b7e-kserve-provision-location\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:58:23.027665 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:23.027588    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39598be1-1d31-4551-be5a-34d544c84b7e-proxy-tls\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:58:23.034200 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:23.034168    2572 generic.go:358] "Generic (PLEG): container finished" podID="39598be1-1d31-4551-be5a-34d544c84b7e" containerID="fb076cb25f6e8e9e90c116c26e6a2235f4725d5b6d2e1024c7e8765880631b5c" exitCode=0
May 26 16:58:23.034651 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:23.034244    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd" event={"ID":"39598be1-1d31-4551-be5a-34d544c84b7e","Type":"ContainerDied","Data":"fb076cb25f6e8e9e90c116c26e6a2235f4725d5b6d2e1024c7e8765880631b5c"}
May 26 16:58:23.034651 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:23.034270    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd"
May 26 16:58:23.034651 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:23.034279    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd" event={"ID":"39598be1-1d31-4551-be5a-34d544c84b7e","Type":"ContainerDied","Data":"7306923bb848fb938f8f94e3ac592bd853562ea5a6b3eaa033a86620a02c751a"}
May 26 16:58:23.034651 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:23.034296    2572 scope.go:117] "RemoveContainer" containerID="24e2d270ac31f2144e8bd503187ec85f6c3d8fec07958344db9bb450f6f012ae"
May 26 16:58:23.036403 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:23.036379    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-ppp6x" event={"ID":"0f31e5d5-9f66-4b20-ae0b-f932fcc63102","Type":"ContainerStarted","Data":"6abb79d2c1cd0ee17e37695d6cc4aee4d45702eee642b08de57325a09d0b02d6"}
May 26 16:58:23.036539 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:23.036409    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-ppp6x" event={"ID":"0f31e5d5-9f66-4b20-ae0b-f932fcc63102","Type":"ContainerStarted","Data":"7273cf6c528ec47a35c1769cb15365f156b413d30c4535c56f820915ed96c0a7"}
May 26 16:58:23.036664 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:23.036644    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-ppp6x"
May 26 16:58:23.036741 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:23.036721    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-ppp6x"
May 26 16:58:23.045585 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:23.045565    2572 scope.go:117] "RemoveContainer" containerID="fb076cb25f6e8e9e90c116c26e6a2235f4725d5b6d2e1024c7e8765880631b5c"
May 26 16:58:23.052720 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:23.052701    2572 scope.go:117] "RemoveContainer" containerID="7d42d1a9d9339d47d9835e2ab4fe532c99cb4fdf3ce8ea26f21e4093999a7009"
May 26 16:58:23.059917 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:23.059867    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-ppp6x" podStartSLOduration=7.059848263 podStartE2EDuration="7.059848263s" podCreationTimestamp="2026-05-26 16:58:16 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:58:23.059066109 +0000 UTC m=+3419.534793459" watchObservedRunningTime="2026-05-26 16:58:23.059848263 +0000 UTC m=+3419.535575614"
May 26 16:58:23.060515 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:23.060495    2572 scope.go:117] "RemoveContainer" containerID="24e2d270ac31f2144e8bd503187ec85f6c3d8fec07958344db9bb450f6f012ae"
May 26 16:58:23.060836 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:58:23.060815    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24e2d270ac31f2144e8bd503187ec85f6c3d8fec07958344db9bb450f6f012ae\": container with ID starting with 24e2d270ac31f2144e8bd503187ec85f6c3d8fec07958344db9bb450f6f012ae not found: ID does not exist" containerID="24e2d270ac31f2144e8bd503187ec85f6c3d8fec07958344db9bb450f6f012ae"
May 26 16:58:23.060921 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:23.060847    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24e2d270ac31f2144e8bd503187ec85f6c3d8fec07958344db9bb450f6f012ae"} err="failed to get container status \"24e2d270ac31f2144e8bd503187ec85f6c3d8fec07958344db9bb450f6f012ae\": rpc error: code = NotFound desc = could not find container \"24e2d270ac31f2144e8bd503187ec85f6c3d8fec07958344db9bb450f6f012ae\": container with ID starting with 24e2d270ac31f2144e8bd503187ec85f6c3d8fec07958344db9bb450f6f012ae not found: ID does not exist"
May 26 16:58:23.060921 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:23.060871    2572 scope.go:117] "RemoveContainer" containerID="fb076cb25f6e8e9e90c116c26e6a2235f4725d5b6d2e1024c7e8765880631b5c"
May 26 16:58:23.061124 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:58:23.061109    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb076cb25f6e8e9e90c116c26e6a2235f4725d5b6d2e1024c7e8765880631b5c\": container with ID starting with fb076cb25f6e8e9e90c116c26e6a2235f4725d5b6d2e1024c7e8765880631b5c not found: ID does not exist" containerID="fb076cb25f6e8e9e90c116c26e6a2235f4725d5b6d2e1024c7e8765880631b5c"
May 26 16:58:23.061171 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:23.061130    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb076cb25f6e8e9e90c116c26e6a2235f4725d5b6d2e1024c7e8765880631b5c"} err="failed to get container status \"fb076cb25f6e8e9e90c116c26e6a2235f4725d5b6d2e1024c7e8765880631b5c\": rpc error: code = NotFound desc = could not find container \"fb076cb25f6e8e9e90c116c26e6a2235f4725d5b6d2e1024c7e8765880631b5c\": container with ID starting with fb076cb25f6e8e9e90c116c26e6a2235f4725d5b6d2e1024c7e8765880631b5c not found: ID does not exist"
May 26 16:58:23.061171 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:23.061148    2572 scope.go:117] "RemoveContainer" containerID="7d42d1a9d9339d47d9835e2ab4fe532c99cb4fdf3ce8ea26f21e4093999a7009"
May 26 16:58:23.061400 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:58:23.061377    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d42d1a9d9339d47d9835e2ab4fe532c99cb4fdf3ce8ea26f21e4093999a7009\": container with ID starting with 7d42d1a9d9339d47d9835e2ab4fe532c99cb4fdf3ce8ea26f21e4093999a7009 not found: ID does not exist" containerID="7d42d1a9d9339d47d9835e2ab4fe532c99cb4fdf3ce8ea26f21e4093999a7009"
May 26 16:58:23.061457 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:23.061406    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d42d1a9d9339d47d9835e2ab4fe532c99cb4fdf3ce8ea26f21e4093999a7009"} err="failed to get container status \"7d42d1a9d9339d47d9835e2ab4fe532c99cb4fdf3ce8ea26f21e4093999a7009\": rpc error: code = NotFound desc = could not find container \"7d42d1a9d9339d47d9835e2ab4fe532c99cb4fdf3ce8ea26f21e4093999a7009\": container with ID starting with 7d42d1a9d9339d47d9835e2ab4fe532c99cb4fdf3ce8ea26f21e4093999a7009 not found: ID does not exist"
May 26 16:58:23.074984 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:23.074957    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd"]
May 26 16:58:23.077160 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:23.077139    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-bqfqd"]
May 26 16:58:24.079436 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:24.079408    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39598be1-1d31-4551-be5a-34d544c84b7e" path="/var/lib/kubelet/pods/39598be1-1d31-4551-be5a-34d544c84b7e/volumes"
May 26 16:58:29.046560 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:29.046483    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-ppp6x"
May 26 16:58:59.050517 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:58:59.050486    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-ppp6x"
May 26 16:59:06.295192 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:06.295159    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-ppp6x"]
May 26 16:59:06.295736 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:06.295448    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-ppp6x" podUID="0f31e5d5-9f66-4b20-ae0b-f932fcc63102" containerName="kserve-container" containerID="cri-o://7273cf6c528ec47a35c1769cb15365f156b413d30c4535c56f820915ed96c0a7" gracePeriod=30
May 26 16:59:06.295736 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:06.295484    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-ppp6x" podUID="0f31e5d5-9f66-4b20-ae0b-f932fcc63102" containerName="kube-rbac-proxy" containerID="cri-o://6abb79d2c1cd0ee17e37695d6cc4aee4d45702eee642b08de57325a09d0b02d6" gracePeriod=30
May 26 16:59:06.394230 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:06.394198    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-fmnv4"]
May 26 16:59:06.394505 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:06.394492    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39598be1-1d31-4551-be5a-34d544c84b7e" containerName="kserve-container"
May 26 16:59:06.394556 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:06.394507    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="39598be1-1d31-4551-be5a-34d544c84b7e" containerName="kserve-container"
May 26 16:59:06.394556 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:06.394519    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39598be1-1d31-4551-be5a-34d544c84b7e" containerName="storage-initializer"
May 26 16:59:06.394556 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:06.394525    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="39598be1-1d31-4551-be5a-34d544c84b7e" containerName="storage-initializer"
May 26 16:59:06.394556 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:06.394531    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39598be1-1d31-4551-be5a-34d544c84b7e" containerName="kube-rbac-proxy"
May 26 16:59:06.394556 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:06.394537    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="39598be1-1d31-4551-be5a-34d544c84b7e" containerName="kube-rbac-proxy"
May 26 16:59:06.394766 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:06.394586    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="39598be1-1d31-4551-be5a-34d544c84b7e" containerName="kserve-container"
May 26 16:59:06.394766 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:06.394596    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="39598be1-1d31-4551-be5a-34d544c84b7e" containerName="kube-rbac-proxy"
May 26 16:59:06.396672 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:06.396653    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-fmnv4"
May 26 16:59:06.398717 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:06.398695    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
May 26 16:59:06.398826 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:06.398695    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
May 26 16:59:06.409414 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:06.407485    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-fmnv4"]
May 26 16:59:06.468913 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:06.468872    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a91cc1f5-fc7a-4c93-92fd-a6d551a180cf-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-779db84d9-fmnv4\" (UID: \"a91cc1f5-fc7a-4c93-92fd-a6d551a180cf\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-fmnv4"
May 26 16:59:06.469082 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:06.468921    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a91cc1f5-fc7a-4c93-92fd-a6d551a180cf-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-779db84d9-fmnv4\" (UID: \"a91cc1f5-fc7a-4c93-92fd-a6d551a180cf\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-fmnv4"
May 26 16:59:06.469082 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:06.468946    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw5xj\" (UniqueName: \"kubernetes.io/projected/a91cc1f5-fc7a-4c93-92fd-a6d551a180cf-kube-api-access-lw5xj\") pod \"isvc-xgboost-runtime-predictor-779db84d9-fmnv4\" (UID: \"a91cc1f5-fc7a-4c93-92fd-a6d551a180cf\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-fmnv4"
May 26 16:59:06.469082 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:06.468972    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a91cc1f5-fc7a-4c93-92fd-a6d551a180cf-isvc-xgboost-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-779db84d9-fmnv4\" (UID: \"a91cc1f5-fc7a-4c93-92fd-a6d551a180cf\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-fmnv4"
May 26 16:59:06.570095 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:06.570005    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lw5xj\" (UniqueName: \"kubernetes.io/projected/a91cc1f5-fc7a-4c93-92fd-a6d551a180cf-kube-api-access-lw5xj\") pod \"isvc-xgboost-runtime-predictor-779db84d9-fmnv4\" (UID: \"a91cc1f5-fc7a-4c93-92fd-a6d551a180cf\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-fmnv4"
May 26 16:59:06.570095 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:06.570056    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a91cc1f5-fc7a-4c93-92fd-a6d551a180cf-isvc-xgboost-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-779db84d9-fmnv4\" (UID: \"a91cc1f5-fc7a-4c93-92fd-a6d551a180cf\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-fmnv4"
May 26 16:59:06.570323 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:06.570111    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a91cc1f5-fc7a-4c93-92fd-a6d551a180cf-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-779db84d9-fmnv4\" (UID: \"a91cc1f5-fc7a-4c93-92fd-a6d551a180cf\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-fmnv4"
May 26 16:59:06.570323 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:06.570136    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a91cc1f5-fc7a-4c93-92fd-a6d551a180cf-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-779db84d9-fmnv4\" (UID: \"a91cc1f5-fc7a-4c93-92fd-a6d551a180cf\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-fmnv4"
May 26 16:59:06.570548 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:06.570525    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a91cc1f5-fc7a-4c93-92fd-a6d551a180cf-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-779db84d9-fmnv4\" (UID: \"a91cc1f5-fc7a-4c93-92fd-a6d551a180cf\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-fmnv4"
May 26 16:59:06.570763 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:06.570743    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a91cc1f5-fc7a-4c93-92fd-a6d551a180cf-isvc-xgboost-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-779db84d9-fmnv4\" (UID: \"a91cc1f5-fc7a-4c93-92fd-a6d551a180cf\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-fmnv4"
May 26 16:59:06.572560 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:06.572544    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a91cc1f5-fc7a-4c93-92fd-a6d551a180cf-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-779db84d9-fmnv4\" (UID: \"a91cc1f5-fc7a-4c93-92fd-a6d551a180cf\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-fmnv4"
May 26 16:59:06.579472 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:06.579441    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw5xj\" (UniqueName: \"kubernetes.io/projected/a91cc1f5-fc7a-4c93-92fd-a6d551a180cf-kube-api-access-lw5xj\") pod \"isvc-xgboost-runtime-predictor-779db84d9-fmnv4\" (UID: \"a91cc1f5-fc7a-4c93-92fd-a6d551a180cf\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-fmnv4"
May 26 16:59:06.710160 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:06.710121    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-fmnv4"
May 26 16:59:06.836279 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:06.836186    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-fmnv4"]
May 26 16:59:06.842164 ip-10-0-130-83 kubenswrapper[2572]: W0526 16:59:06.842132    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda91cc1f5_fc7a_4c93_92fd_a6d551a180cf.slice/crio-24d1e79f03efb1803a5f14669d23c2305bbc9fe2d850dfe28dfc9f813f9401c9 WatchSource:0}: Error finding container 24d1e79f03efb1803a5f14669d23c2305bbc9fe2d850dfe28dfc9f813f9401c9: Status 404 returned error can't find the container with id 24d1e79f03efb1803a5f14669d23c2305bbc9fe2d850dfe28dfc9f813f9401c9
May 26 16:59:07.154678 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:07.154636    2572 generic.go:358] "Generic (PLEG): container finished" podID="0f31e5d5-9f66-4b20-ae0b-f932fcc63102" containerID="6abb79d2c1cd0ee17e37695d6cc4aee4d45702eee642b08de57325a09d0b02d6" exitCode=2
May 26 16:59:07.154859 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:07.154698    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-ppp6x" event={"ID":"0f31e5d5-9f66-4b20-ae0b-f932fcc63102","Type":"ContainerDied","Data":"6abb79d2c1cd0ee17e37695d6cc4aee4d45702eee642b08de57325a09d0b02d6"}
May 26 16:59:07.156013 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:07.155991    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-fmnv4" event={"ID":"a91cc1f5-fc7a-4c93-92fd-a6d551a180cf","Type":"ContainerStarted","Data":"9ed51ee8e13bcd5552673f90c5c41a2700cbb48c3a20d457e931c52de4ee2d60"}
May 26 16:59:07.156113 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:07.156017    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-fmnv4" event={"ID":"a91cc1f5-fc7a-4c93-92fd-a6d551a180cf","Type":"ContainerStarted","Data":"24d1e79f03efb1803a5f14669d23c2305bbc9fe2d850dfe28dfc9f813f9401c9"}
May 26 16:59:09.041228 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:09.041185    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-ppp6x" podUID="0f31e5d5-9f66-4b20-ae0b-f932fcc63102" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.57:8643/healthz\": dial tcp 10.132.0.57:8643: connect: connection refused"
May 26 16:59:09.047471 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:09.047440    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-ppp6x" podUID="0f31e5d5-9f66-4b20-ae0b-f932fcc63102" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.57:8080/v2/models/xgboost-v2-mlserver/ready\": dial tcp 10.132.0.57:8080: connect: connection refused"
May 26 16:59:11.169918 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:11.169871    2572 generic.go:358] "Generic (PLEG): container finished" podID="a91cc1f5-fc7a-4c93-92fd-a6d551a180cf" containerID="9ed51ee8e13bcd5552673f90c5c41a2700cbb48c3a20d457e931c52de4ee2d60" exitCode=0
May 26 16:59:11.170327 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:11.169945    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-fmnv4" event={"ID":"a91cc1f5-fc7a-4c93-92fd-a6d551a180cf","Type":"ContainerDied","Data":"9ed51ee8e13bcd5552673f90c5c41a2700cbb48c3a20d457e931c52de4ee2d60"}
May 26 16:59:12.180257 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:12.180216    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-fmnv4" event={"ID":"a91cc1f5-fc7a-4c93-92fd-a6d551a180cf","Type":"ContainerStarted","Data":"8b442c332ca6493705df5c73d69a76f9bd23b539f31a5352d0e74a459de3054b"}
May 26 16:59:12.180661 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:12.180265    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-fmnv4" event={"ID":"a91cc1f5-fc7a-4c93-92fd-a6d551a180cf","Type":"ContainerStarted","Data":"21653f03bb10faf5e9166bb497ff3a4fc9f9fe2395af4b93529450a4008d7f19"}
May 26 16:59:12.180661 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:12.180491    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-fmnv4"
May 26 16:59:12.200938 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:12.200887    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-fmnv4" podStartSLOduration=6.2008726 podStartE2EDuration="6.2008726s" podCreationTimestamp="2026-05-26 16:59:06 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:59:12.199935046 +0000 UTC m=+3468.675662393" watchObservedRunningTime="2026-05-26 16:59:12.2008726 +0000 UTC m=+3468.676599950"
May 26 16:59:13.134894 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:13.134862    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-ppp6x"
May 26 16:59:13.184718 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:13.184689    2572 generic.go:358] "Generic (PLEG): container finished" podID="0f31e5d5-9f66-4b20-ae0b-f932fcc63102" containerID="7273cf6c528ec47a35c1769cb15365f156b413d30c4535c56f820915ed96c0a7" exitCode=0
May 26 16:59:13.185187 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:13.184766    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-ppp6x" event={"ID":"0f31e5d5-9f66-4b20-ae0b-f932fcc63102","Type":"ContainerDied","Data":"7273cf6c528ec47a35c1769cb15365f156b413d30c4535c56f820915ed96c0a7"}
May 26 16:59:13.185187 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:13.184777    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-ppp6x"
May 26 16:59:13.185187 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:13.184802    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-ppp6x" event={"ID":"0f31e5d5-9f66-4b20-ae0b-f932fcc63102","Type":"ContainerDied","Data":"7d335cfbfc7211840d1d0d6292f2f36fee24b12563f640c7d8f0c48f3dff513d"}
May 26 16:59:13.185187 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:13.184819    2572 scope.go:117] "RemoveContainer" containerID="6abb79d2c1cd0ee17e37695d6cc4aee4d45702eee642b08de57325a09d0b02d6"
May 26 16:59:13.185398 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:13.185216    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-fmnv4"
May 26 16:59:13.186514 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:13.186477    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-fmnv4" podUID="a91cc1f5-fc7a-4c93-92fd-a6d551a180cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
May 26 16:59:13.192397 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:13.192380    2572 scope.go:117] "RemoveContainer" containerID="7273cf6c528ec47a35c1769cb15365f156b413d30c4535c56f820915ed96c0a7"
May 26 16:59:13.199301 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:13.199282    2572 scope.go:117] "RemoveContainer" containerID="dde7fbcb11439cc0cade95ada5f175890cf14114657a0a8880dab7336a061172"
May 26 16:59:13.206042 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:13.206025    2572 scope.go:117] "RemoveContainer" containerID="6abb79d2c1cd0ee17e37695d6cc4aee4d45702eee642b08de57325a09d0b02d6"
May 26 16:59:13.206291 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:59:13.206273    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6abb79d2c1cd0ee17e37695d6cc4aee4d45702eee642b08de57325a09d0b02d6\": container with ID starting with 6abb79d2c1cd0ee17e37695d6cc4aee4d45702eee642b08de57325a09d0b02d6 not found: ID does not exist" containerID="6abb79d2c1cd0ee17e37695d6cc4aee4d45702eee642b08de57325a09d0b02d6"
May 26 16:59:13.206365 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:13.206303    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6abb79d2c1cd0ee17e37695d6cc4aee4d45702eee642b08de57325a09d0b02d6"} err="failed to get container status \"6abb79d2c1cd0ee17e37695d6cc4aee4d45702eee642b08de57325a09d0b02d6\": rpc error: code = NotFound desc = could not find container \"6abb79d2c1cd0ee17e37695d6cc4aee4d45702eee642b08de57325a09d0b02d6\": container with ID starting with 6abb79d2c1cd0ee17e37695d6cc4aee4d45702eee642b08de57325a09d0b02d6 not found: ID does not exist"
May 26 16:59:13.206365 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:13.206331    2572 scope.go:117] "RemoveContainer" containerID="7273cf6c528ec47a35c1769cb15365f156b413d30c4535c56f820915ed96c0a7"
May 26 16:59:13.206573 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:59:13.206558    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7273cf6c528ec47a35c1769cb15365f156b413d30c4535c56f820915ed96c0a7\": container with ID starting with 7273cf6c528ec47a35c1769cb15365f156b413d30c4535c56f820915ed96c0a7 not found: ID does not exist" containerID="7273cf6c528ec47a35c1769cb15365f156b413d30c4535c56f820915ed96c0a7"
May 26 16:59:13.206628 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:13.206578    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7273cf6c528ec47a35c1769cb15365f156b413d30c4535c56f820915ed96c0a7"} err="failed to get container status \"7273cf6c528ec47a35c1769cb15365f156b413d30c4535c56f820915ed96c0a7\": rpc error: code = NotFound desc = could not find container \"7273cf6c528ec47a35c1769cb15365f156b413d30c4535c56f820915ed96c0a7\": container with ID starting with 7273cf6c528ec47a35c1769cb15365f156b413d30c4535c56f820915ed96c0a7 not found: ID does not exist"
May 26 16:59:13.206628 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:13.206593    2572 scope.go:117] "RemoveContainer" containerID="dde7fbcb11439cc0cade95ada5f175890cf14114657a0a8880dab7336a061172"
May 26 16:59:13.206792 ip-10-0-130-83 kubenswrapper[2572]: E0526 16:59:13.206776    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dde7fbcb11439cc0cade95ada5f175890cf14114657a0a8880dab7336a061172\": container with ID starting with dde7fbcb11439cc0cade95ada5f175890cf14114657a0a8880dab7336a061172 not found: ID does not exist" containerID="dde7fbcb11439cc0cade95ada5f175890cf14114657a0a8880dab7336a061172"
May 26 16:59:13.206831 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:13.206796    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dde7fbcb11439cc0cade95ada5f175890cf14114657a0a8880dab7336a061172"} err="failed to get container status \"dde7fbcb11439cc0cade95ada5f175890cf14114657a0a8880dab7336a061172\": rpc error: code = NotFound desc = could not find container \"dde7fbcb11439cc0cade95ada5f175890cf14114657a0a8880dab7336a061172\": container with ID starting with dde7fbcb11439cc0cade95ada5f175890cf14114657a0a8880dab7336a061172 not found: ID does not exist"
May 26 16:59:13.222057 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:13.222028    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f31e5d5-9f66-4b20-ae0b-f932fcc63102-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"0f31e5d5-9f66-4b20-ae0b-f932fcc63102\" (UID: \"0f31e5d5-9f66-4b20-ae0b-f932fcc63102\") "
May 26 16:59:13.222171 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:13.222072    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vjbpg\" (UniqueName: \"kubernetes.io/projected/0f31e5d5-9f66-4b20-ae0b-f932fcc63102-kube-api-access-vjbpg\") pod \"0f31e5d5-9f66-4b20-ae0b-f932fcc63102\" (UID: \"0f31e5d5-9f66-4b20-ae0b-f932fcc63102\") "
May 26 16:59:13.222171 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:13.222121    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f31e5d5-9f66-4b20-ae0b-f932fcc63102-proxy-tls\") pod \"0f31e5d5-9f66-4b20-ae0b-f932fcc63102\" (UID: \"0f31e5d5-9f66-4b20-ae0b-f932fcc63102\") "
May 26 16:59:13.222282 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:13.222214    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f31e5d5-9f66-4b20-ae0b-f932fcc63102-kserve-provision-location\") pod \"0f31e5d5-9f66-4b20-ae0b-f932fcc63102\" (UID: \"0f31e5d5-9f66-4b20-ae0b-f932fcc63102\") "
May 26 16:59:13.222496 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:13.222471    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f31e5d5-9f66-4b20-ae0b-f932fcc63102-xgboost-v2-mlserver-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "xgboost-v2-mlserver-kube-rbac-proxy-sar-config") pod "0f31e5d5-9f66-4b20-ae0b-f932fcc63102" (UID: "0f31e5d5-9f66-4b20-ae0b-f932fcc63102"). InnerVolumeSpecName "xgboost-v2-mlserver-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:59:13.222568 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:13.222522    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f31e5d5-9f66-4b20-ae0b-f932fcc63102-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0f31e5d5-9f66-4b20-ae0b-f932fcc63102" (UID: "0f31e5d5-9f66-4b20-ae0b-f932fcc63102"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:59:13.224155 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:13.224130    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f31e5d5-9f66-4b20-ae0b-f932fcc63102-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0f31e5d5-9f66-4b20-ae0b-f932fcc63102" (UID: "0f31e5d5-9f66-4b20-ae0b-f932fcc63102"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:59:13.224234 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:13.224189    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f31e5d5-9f66-4b20-ae0b-f932fcc63102-kube-api-access-vjbpg" (OuterVolumeSpecName: "kube-api-access-vjbpg") pod "0f31e5d5-9f66-4b20-ae0b-f932fcc63102" (UID: "0f31e5d5-9f66-4b20-ae0b-f932fcc63102"). InnerVolumeSpecName "kube-api-access-vjbpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:59:13.323916 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:13.323856    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f31e5d5-9f66-4b20-ae0b-f932fcc63102-proxy-tls\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:59:13.323916 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:13.323896    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f31e5d5-9f66-4b20-ae0b-f932fcc63102-kserve-provision-location\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:59:13.323916 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:13.323911    2572 reconciler_common.go:299] "Volume detached for volume \"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f31e5d5-9f66-4b20-ae0b-f932fcc63102-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:59:13.324302 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:13.323930    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjbpg\" (UniqueName: \"kubernetes.io/projected/0f31e5d5-9f66-4b20-ae0b-f932fcc63102-kube-api-access-vjbpg\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 16:59:13.506237 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:13.506203    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-ppp6x"]
May 26 16:59:13.510095 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:13.510064    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-ppp6x"]
May 26 16:59:14.079259 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:14.079227    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f31e5d5-9f66-4b20-ae0b-f932fcc63102" path="/var/lib/kubelet/pods/0f31e5d5-9f66-4b20-ae0b-f932fcc63102/volumes"
May 26 16:59:14.189192 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:14.189158    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-fmnv4" podUID="a91cc1f5-fc7a-4c93-92fd-a6d551a180cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
May 26 16:59:19.193263 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:19.193235    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-fmnv4"
May 26 16:59:19.193880 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:19.193844    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-fmnv4" podUID="a91cc1f5-fc7a-4c93-92fd-a6d551a180cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
May 26 16:59:29.194160 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:29.194112    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-fmnv4" podUID="a91cc1f5-fc7a-4c93-92fd-a6d551a180cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
May 26 16:59:39.193988 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:39.193950    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-fmnv4" podUID="a91cc1f5-fc7a-4c93-92fd-a6d551a180cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
May 26 16:59:49.194037 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:49.193954    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-fmnv4" podUID="a91cc1f5-fc7a-4c93-92fd-a6d551a180cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
May 26 16:59:59.193786 ip-10-0-130-83 kubenswrapper[2572]: I0526 16:59:59.193747    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-fmnv4" podUID="a91cc1f5-fc7a-4c93-92fd-a6d551a180cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
May 26 17:00:09.193696 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:09.193659    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-fmnv4" podUID="a91cc1f5-fc7a-4c93-92fd-a6d551a180cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
May 26 17:00:19.194764 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:19.194728    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-fmnv4"
May 26 17:00:26.472127 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:26.472088    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-fmnv4"]
May 26 17:00:26.472517 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:26.472381    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-fmnv4" podUID="a91cc1f5-fc7a-4c93-92fd-a6d551a180cf" containerName="kserve-container" containerID="cri-o://21653f03bb10faf5e9166bb497ff3a4fc9f9fe2395af4b93529450a4008d7f19" gracePeriod=30
May 26 17:00:26.472517 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:26.472415    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-fmnv4" podUID="a91cc1f5-fc7a-4c93-92fd-a6d551a180cf" containerName="kube-rbac-proxy" containerID="cri-o://8b442c332ca6493705df5c73d69a76f9bd23b539f31a5352d0e74a459de3054b" gracePeriod=30
May 26 17:00:26.570853 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:26.570814    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4"]
May 26 17:00:26.571087 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:26.571075    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f31e5d5-9f66-4b20-ae0b-f932fcc63102" containerName="kserve-container"
May 26 17:00:26.571135 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:26.571088    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f31e5d5-9f66-4b20-ae0b-f932fcc63102" containerName="kserve-container"
May 26 17:00:26.571135 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:26.571099    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f31e5d5-9f66-4b20-ae0b-f932fcc63102" containerName="storage-initializer"
May 26 17:00:26.571135 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:26.571104    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f31e5d5-9f66-4b20-ae0b-f932fcc63102" containerName="storage-initializer"
May 26 17:00:26.571135 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:26.571114    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f31e5d5-9f66-4b20-ae0b-f932fcc63102" containerName="kube-rbac-proxy"
May 26 17:00:26.571135 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:26.571120    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f31e5d5-9f66-4b20-ae0b-f932fcc63102" containerName="kube-rbac-proxy"
May 26 17:00:26.571331 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:26.571160    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="0f31e5d5-9f66-4b20-ae0b-f932fcc63102" containerName="kube-rbac-proxy"
May 26 17:00:26.571331 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:26.571169    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="0f31e5d5-9f66-4b20-ae0b-f932fcc63102" containerName="kserve-container"
May 26 17:00:26.573045 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:26.573028    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4"
May 26 17:00:26.574973 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:26.574952    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
May 26 17:00:26.575268 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:26.575252    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
May 26 17:00:26.583067 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:26.583045    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4"]
May 26 17:00:26.651019 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:26.650985    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb522444-10c8-4ea9-8987-42007540e211-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4\" (UID: \"eb522444-10c8-4ea9-8987-42007540e211\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4"
May 26 17:00:26.651185 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:26.651032    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb522444-10c8-4ea9-8987-42007540e211-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4\" (UID: \"eb522444-10c8-4ea9-8987-42007540e211\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4"
May 26 17:00:26.651185 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:26.651067    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj59l\" (UniqueName: \"kubernetes.io/projected/eb522444-10c8-4ea9-8987-42007540e211-kube-api-access-dj59l\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4\" (UID: \"eb522444-10c8-4ea9-8987-42007540e211\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4"
May 26 17:00:26.651185 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:26.651096    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb522444-10c8-4ea9-8987-42007540e211-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4\" (UID: \"eb522444-10c8-4ea9-8987-42007540e211\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4"
May 26 17:00:26.751716 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:26.751627    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb522444-10c8-4ea9-8987-42007540e211-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4\" (UID: \"eb522444-10c8-4ea9-8987-42007540e211\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4"
May 26 17:00:26.751716 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:26.751680    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb522444-10c8-4ea9-8987-42007540e211-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4\" (UID: \"eb522444-10c8-4ea9-8987-42007540e211\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4"
May 26 17:00:26.751716 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:26.751706    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dj59l\" (UniqueName: \"kubernetes.io/projected/eb522444-10c8-4ea9-8987-42007540e211-kube-api-access-dj59l\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4\" (UID: \"eb522444-10c8-4ea9-8987-42007540e211\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4"
May 26 17:00:26.752026 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:26.751733    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb522444-10c8-4ea9-8987-42007540e211-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4\" (UID: \"eb522444-10c8-4ea9-8987-42007540e211\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4"
May 26 17:00:26.752026 ip-10-0-130-83 kubenswrapper[2572]: E0526 17:00:26.751858    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-serving-cert: secret "isvc-xgboost-v2-runtime-predictor-serving-cert" not found
May 26 17:00:26.752026 ip-10-0-130-83 kubenswrapper[2572]: E0526 17:00:26.751926    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb522444-10c8-4ea9-8987-42007540e211-proxy-tls podName:eb522444-10c8-4ea9-8987-42007540e211 nodeName:}" failed. No retries permitted until 2026-05-26 17:00:27.251910746 +0000 UTC m=+3543.727638074 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/eb522444-10c8-4ea9-8987-42007540e211-proxy-tls") pod "isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4" (UID: "eb522444-10c8-4ea9-8987-42007540e211") : secret "isvc-xgboost-v2-runtime-predictor-serving-cert" not found
May 26 17:00:26.752203 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:26.752064    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb522444-10c8-4ea9-8987-42007540e211-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4\" (UID: \"eb522444-10c8-4ea9-8987-42007540e211\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4"
May 26 17:00:26.752268 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:26.752247    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb522444-10c8-4ea9-8987-42007540e211-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4\" (UID: \"eb522444-10c8-4ea9-8987-42007540e211\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4"
May 26 17:00:26.760831 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:26.760808    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj59l\" (UniqueName: \"kubernetes.io/projected/eb522444-10c8-4ea9-8987-42007540e211-kube-api-access-dj59l\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4\" (UID: \"eb522444-10c8-4ea9-8987-42007540e211\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4"
May 26 17:00:27.255311 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:27.255262    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb522444-10c8-4ea9-8987-42007540e211-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4\" (UID: \"eb522444-10c8-4ea9-8987-42007540e211\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4"
May 26 17:00:27.257741 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:27.257720    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb522444-10c8-4ea9-8987-42007540e211-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4\" (UID: \"eb522444-10c8-4ea9-8987-42007540e211\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4"
May 26 17:00:27.395243 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:27.395209    2572 generic.go:358] "Generic (PLEG): container finished" podID="a91cc1f5-fc7a-4c93-92fd-a6d551a180cf" containerID="8b442c332ca6493705df5c73d69a76f9bd23b539f31a5352d0e74a459de3054b" exitCode=2
May 26 17:00:27.395417 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:27.395256    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-fmnv4" event={"ID":"a91cc1f5-fc7a-4c93-92fd-a6d551a180cf","Type":"ContainerDied","Data":"8b442c332ca6493705df5c73d69a76f9bd23b539f31a5352d0e74a459de3054b"}
May 26 17:00:27.483989 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:27.483948    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4"
May 26 17:00:27.642356 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:27.642321    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4"]
May 26 17:00:27.645776 ip-10-0-130-83 kubenswrapper[2572]: W0526 17:00:27.645748    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb522444_10c8_4ea9_8987_42007540e211.slice/crio-92337e4a05200045ec675251d27883074e6884c16c97639870739322bb7297f7 WatchSource:0}: Error finding container 92337e4a05200045ec675251d27883074e6884c16c97639870739322bb7297f7: Status 404 returned error can't find the container with id 92337e4a05200045ec675251d27883074e6884c16c97639870739322bb7297f7
May 26 17:00:28.399230 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:28.399196    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4" event={"ID":"eb522444-10c8-4ea9-8987-42007540e211","Type":"ContainerStarted","Data":"049568b50ac84e1e9ca51e4ad89e9852e8fdc56856ccf2fce9d000aefdaaa238"}
May 26 17:00:28.399230 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:28.399232    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4" event={"ID":"eb522444-10c8-4ea9-8987-42007540e211","Type":"ContainerStarted","Data":"92337e4a05200045ec675251d27883074e6884c16c97639870739322bb7297f7"}
May 26 17:00:29.189543 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:29.189498    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-fmnv4" podUID="a91cc1f5-fc7a-4c93-92fd-a6d551a180cf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.58:8643/healthz\": dial tcp 10.132.0.58:8643: connect: connection refused"
May 26 17:00:29.193703 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:29.193681    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-fmnv4" podUID="a91cc1f5-fc7a-4c93-92fd-a6d551a180cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
May 26 17:00:30.198392 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:30.198368    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-fmnv4"
May 26 17:00:30.279206 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:30.279164    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lw5xj\" (UniqueName: \"kubernetes.io/projected/a91cc1f5-fc7a-4c93-92fd-a6d551a180cf-kube-api-access-lw5xj\") pod \"a91cc1f5-fc7a-4c93-92fd-a6d551a180cf\" (UID: \"a91cc1f5-fc7a-4c93-92fd-a6d551a180cf\") "
May 26 17:00:30.279206 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:30.279211    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a91cc1f5-fc7a-4c93-92fd-a6d551a180cf-isvc-xgboost-runtime-kube-rbac-proxy-sar-config\") pod \"a91cc1f5-fc7a-4c93-92fd-a6d551a180cf\" (UID: \"a91cc1f5-fc7a-4c93-92fd-a6d551a180cf\") "
May 26 17:00:30.279466 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:30.279259    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a91cc1f5-fc7a-4c93-92fd-a6d551a180cf-kserve-provision-location\") pod \"a91cc1f5-fc7a-4c93-92fd-a6d551a180cf\" (UID: \"a91cc1f5-fc7a-4c93-92fd-a6d551a180cf\") "
May 26 17:00:30.279466 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:30.279307    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a91cc1f5-fc7a-4c93-92fd-a6d551a180cf-proxy-tls\") pod \"a91cc1f5-fc7a-4c93-92fd-a6d551a180cf\" (UID: \"a91cc1f5-fc7a-4c93-92fd-a6d551a180cf\") "
May 26 17:00:30.279705 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:30.279679    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a91cc1f5-fc7a-4c93-92fd-a6d551a180cf-isvc-xgboost-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-runtime-kube-rbac-proxy-sar-config") pod "a91cc1f5-fc7a-4c93-92fd-a6d551a180cf" (UID: "a91cc1f5-fc7a-4c93-92fd-a6d551a180cf"). InnerVolumeSpecName "isvc-xgboost-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 17:00:30.279804 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:30.279702    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a91cc1f5-fc7a-4c93-92fd-a6d551a180cf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a91cc1f5-fc7a-4c93-92fd-a6d551a180cf" (UID: "a91cc1f5-fc7a-4c93-92fd-a6d551a180cf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 17:00:30.281345 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:30.281314    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a91cc1f5-fc7a-4c93-92fd-a6d551a180cf-kube-api-access-lw5xj" (OuterVolumeSpecName: "kube-api-access-lw5xj") pod "a91cc1f5-fc7a-4c93-92fd-a6d551a180cf" (UID: "a91cc1f5-fc7a-4c93-92fd-a6d551a180cf"). InnerVolumeSpecName "kube-api-access-lw5xj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 17:00:30.281431 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:30.281405    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a91cc1f5-fc7a-4c93-92fd-a6d551a180cf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a91cc1f5-fc7a-4c93-92fd-a6d551a180cf" (UID: "a91cc1f5-fc7a-4c93-92fd-a6d551a180cf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 17:00:30.380587 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:30.380557    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a91cc1f5-fc7a-4c93-92fd-a6d551a180cf-proxy-tls\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 17:00:30.380587 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:30.380587    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lw5xj\" (UniqueName: \"kubernetes.io/projected/a91cc1f5-fc7a-4c93-92fd-a6d551a180cf-kube-api-access-lw5xj\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 17:00:30.380792 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:30.380598    2572 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a91cc1f5-fc7a-4c93-92fd-a6d551a180cf-isvc-xgboost-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 17:00:30.380792 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:30.380627    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a91cc1f5-fc7a-4c93-92fd-a6d551a180cf-kserve-provision-location\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 17:00:30.406454 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:30.406423    2572 generic.go:358] "Generic (PLEG): container finished" podID="a91cc1f5-fc7a-4c93-92fd-a6d551a180cf" containerID="21653f03bb10faf5e9166bb497ff3a4fc9f9fe2395af4b93529450a4008d7f19" exitCode=0
May 26 17:00:30.406621 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:30.406492    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-fmnv4" event={"ID":"a91cc1f5-fc7a-4c93-92fd-a6d551a180cf","Type":"ContainerDied","Data":"21653f03bb10faf5e9166bb497ff3a4fc9f9fe2395af4b93529450a4008d7f19"}
May 26 17:00:30.406621 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:30.406511    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-fmnv4"
May 26 17:00:30.406621 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:30.406522    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-fmnv4" event={"ID":"a91cc1f5-fc7a-4c93-92fd-a6d551a180cf","Type":"ContainerDied","Data":"24d1e79f03efb1803a5f14669d23c2305bbc9fe2d850dfe28dfc9f813f9401c9"}
May 26 17:00:30.406621 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:30.406543    2572 scope.go:117] "RemoveContainer" containerID="8b442c332ca6493705df5c73d69a76f9bd23b539f31a5352d0e74a459de3054b"
May 26 17:00:30.414594 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:30.414559    2572 scope.go:117] "RemoveContainer" containerID="21653f03bb10faf5e9166bb497ff3a4fc9f9fe2395af4b93529450a4008d7f19"
May 26 17:00:30.421954 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:30.421936    2572 scope.go:117] "RemoveContainer" containerID="9ed51ee8e13bcd5552673f90c5c41a2700cbb48c3a20d457e931c52de4ee2d60"
May 26 17:00:30.427680 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:30.427658    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-fmnv4"]
May 26 17:00:30.429115 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:30.429093    2572 scope.go:117] "RemoveContainer" containerID="8b442c332ca6493705df5c73d69a76f9bd23b539f31a5352d0e74a459de3054b"
May 26 17:00:30.429571 ip-10-0-130-83 kubenswrapper[2572]: E0526 17:00:30.429394    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b442c332ca6493705df5c73d69a76f9bd23b539f31a5352d0e74a459de3054b\": container with ID starting with 8b442c332ca6493705df5c73d69a76f9bd23b539f31a5352d0e74a459de3054b not found: ID does not exist" containerID="8b442c332ca6493705df5c73d69a76f9bd23b539f31a5352d0e74a459de3054b"
May 26 17:00:30.429571 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:30.429454    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b442c332ca6493705df5c73d69a76f9bd23b539f31a5352d0e74a459de3054b"} err="failed to get container status \"8b442c332ca6493705df5c73d69a76f9bd23b539f31a5352d0e74a459de3054b\": rpc error: code = NotFound desc = could not find container \"8b442c332ca6493705df5c73d69a76f9bd23b539f31a5352d0e74a459de3054b\": container with ID starting with 8b442c332ca6493705df5c73d69a76f9bd23b539f31a5352d0e74a459de3054b not found: ID does not exist"
May 26 17:00:30.429571 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:30.429476    2572 scope.go:117] "RemoveContainer" containerID="21653f03bb10faf5e9166bb497ff3a4fc9f9fe2395af4b93529450a4008d7f19"
May 26 17:00:30.430052 ip-10-0-130-83 kubenswrapper[2572]: E0526 17:00:30.429888    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21653f03bb10faf5e9166bb497ff3a4fc9f9fe2395af4b93529450a4008d7f19\": container with ID starting with 21653f03bb10faf5e9166bb497ff3a4fc9f9fe2395af4b93529450a4008d7f19 not found: ID does not exist" containerID="21653f03bb10faf5e9166bb497ff3a4fc9f9fe2395af4b93529450a4008d7f19"
May 26 17:00:30.430052 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:30.429922    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21653f03bb10faf5e9166bb497ff3a4fc9f9fe2395af4b93529450a4008d7f19"} err="failed to get container status \"21653f03bb10faf5e9166bb497ff3a4fc9f9fe2395af4b93529450a4008d7f19\": rpc error: code = NotFound desc = could not find container \"21653f03bb10faf5e9166bb497ff3a4fc9f9fe2395af4b93529450a4008d7f19\": container with ID starting with 21653f03bb10faf5e9166bb497ff3a4fc9f9fe2395af4b93529450a4008d7f19 not found: ID does not exist"
May 26 17:00:30.430052 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:30.429946    2572 scope.go:117] "RemoveContainer" containerID="9ed51ee8e13bcd5552673f90c5c41a2700cbb48c3a20d457e931c52de4ee2d60"
May 26 17:00:30.430251 ip-10-0-130-83 kubenswrapper[2572]: E0526 17:00:30.430229    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ed51ee8e13bcd5552673f90c5c41a2700cbb48c3a20d457e931c52de4ee2d60\": container with ID starting with 9ed51ee8e13bcd5552673f90c5c41a2700cbb48c3a20d457e931c52de4ee2d60 not found: ID does not exist" containerID="9ed51ee8e13bcd5552673f90c5c41a2700cbb48c3a20d457e931c52de4ee2d60"
May 26 17:00:30.430324 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:30.430260    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ed51ee8e13bcd5552673f90c5c41a2700cbb48c3a20d457e931c52de4ee2d60"} err="failed to get container status \"9ed51ee8e13bcd5552673f90c5c41a2700cbb48c3a20d457e931c52de4ee2d60\": rpc error: code = NotFound desc = could not find container \"9ed51ee8e13bcd5552673f90c5c41a2700cbb48c3a20d457e931c52de4ee2d60\": container with ID starting with 9ed51ee8e13bcd5552673f90c5c41a2700cbb48c3a20d457e931c52de4ee2d60 not found: ID does not exist"
May 26 17:00:30.431066 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:30.431047    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-779db84d9-fmnv4"]
May 26 17:00:32.079027 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:32.078995    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a91cc1f5-fc7a-4c93-92fd-a6d551a180cf" path="/var/lib/kubelet/pods/a91cc1f5-fc7a-4c93-92fd-a6d551a180cf/volumes"
May 26 17:00:32.414982 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:32.414947    2572 generic.go:358] "Generic (PLEG): container finished" podID="eb522444-10c8-4ea9-8987-42007540e211" containerID="049568b50ac84e1e9ca51e4ad89e9852e8fdc56856ccf2fce9d000aefdaaa238" exitCode=0
May 26 17:00:32.415176 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:32.414984    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4" event={"ID":"eb522444-10c8-4ea9-8987-42007540e211","Type":"ContainerDied","Data":"049568b50ac84e1e9ca51e4ad89e9852e8fdc56856ccf2fce9d000aefdaaa238"}
May 26 17:00:33.419302 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:33.419268    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4" event={"ID":"eb522444-10c8-4ea9-8987-42007540e211","Type":"ContainerStarted","Data":"90d24c1abb2d257cfc92e89ccc2a828830a76ad3efa2109dcaccaadef0e28cf5"}
May 26 17:00:33.419302 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:33.419307    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4" event={"ID":"eb522444-10c8-4ea9-8987-42007540e211","Type":"ContainerStarted","Data":"9accee429330fabf0ebfbca313a36a18efef97a86313a526955388bb9a606603"}
May 26 17:00:33.419756 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:33.419499    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4"
May 26 17:00:33.438940 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:33.438882    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4" podStartSLOduration=7.43886686 podStartE2EDuration="7.43886686s" podCreationTimestamp="2026-05-26 17:00:26 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 17:00:33.437069274 +0000 UTC m=+3549.912796624" watchObservedRunningTime="2026-05-26 17:00:33.43886686 +0000 UTC m=+3549.914594209"
May 26 17:00:34.422367 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:34.422328    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4"
May 26 17:00:40.429859 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:00:40.429828    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4"
May 26 17:01:10.434865 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:10.434818    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4" podUID="eb522444-10c8-4ea9-8987-42007540e211" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
May 26 17:01:20.432917 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:20.432877    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4"
May 26 17:01:26.622204 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:26.622157    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4"]
May 26 17:01:26.622762 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:26.622575    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4" podUID="eb522444-10c8-4ea9-8987-42007540e211" containerName="kserve-container" containerID="cri-o://9accee429330fabf0ebfbca313a36a18efef97a86313a526955388bb9a606603" gracePeriod=30
May 26 17:01:26.622762 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:26.622590    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4" podUID="eb522444-10c8-4ea9-8987-42007540e211" containerName="kube-rbac-proxy" containerID="cri-o://90d24c1abb2d257cfc92e89ccc2a828830a76ad3efa2109dcaccaadef0e28cf5" gracePeriod=30
May 26 17:01:26.758960 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:26.758925    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-78tfs"]
May 26 17:01:26.759200 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:26.759185    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a91cc1f5-fc7a-4c93-92fd-a6d551a180cf" containerName="storage-initializer"
May 26 17:01:26.759258 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:26.759202    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a91cc1f5-fc7a-4c93-92fd-a6d551a180cf" containerName="storage-initializer"
May 26 17:01:26.759258 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:26.759217    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a91cc1f5-fc7a-4c93-92fd-a6d551a180cf" containerName="kserve-container"
May 26 17:01:26.759258 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:26.759223    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a91cc1f5-fc7a-4c93-92fd-a6d551a180cf" containerName="kserve-container"
May 26 17:01:26.759258 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:26.759239    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a91cc1f5-fc7a-4c93-92fd-a6d551a180cf" containerName="kube-rbac-proxy"
May 26 17:01:26.759258 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:26.759245    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a91cc1f5-fc7a-4c93-92fd-a6d551a180cf" containerName="kube-rbac-proxy"
May 26 17:01:26.759407 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:26.759287    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="a91cc1f5-fc7a-4c93-92fd-a6d551a180cf" containerName="kserve-container"
May 26 17:01:26.759407 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:26.759296    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="a91cc1f5-fc7a-4c93-92fd-a6d551a180cf" containerName="kube-rbac-proxy"
May 26 17:01:26.762189 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:26.762172    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-78tfs"
May 26 17:01:26.764876 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:26.764852    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-predictor-serving-cert\""
May 26 17:01:26.765374 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:26.765353    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
May 26 17:01:26.779358 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:26.779333    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-78tfs"]
May 26 17:01:26.908399 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:26.908301    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/261b83dd-94ac-429f-94f2-9c5cdf39469b-proxy-tls\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-78tfs\" (UID: \"261b83dd-94ac-429f-94f2-9c5cdf39469b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-78tfs"
May 26 17:01:26.908399 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:26.908353    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjl8n\" (UniqueName: \"kubernetes.io/projected/261b83dd-94ac-429f-94f2-9c5cdf39469b-kube-api-access-gjl8n\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-78tfs\" (UID: \"261b83dd-94ac-429f-94f2-9c5cdf39469b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-78tfs"
May 26 17:01:26.908680 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:26.908439    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/261b83dd-94ac-429f-94f2-9c5cdf39469b-isvc-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-78tfs\" (UID: \"261b83dd-94ac-429f-94f2-9c5cdf39469b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-78tfs"
May 26 17:01:26.908680 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:26.908516    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/261b83dd-94ac-429f-94f2-9c5cdf39469b-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-78tfs\" (UID: \"261b83dd-94ac-429f-94f2-9c5cdf39469b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-78tfs"
May 26 17:01:27.009848 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:27.009810    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/261b83dd-94ac-429f-94f2-9c5cdf39469b-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-78tfs\" (UID: \"261b83dd-94ac-429f-94f2-9c5cdf39469b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-78tfs"
May 26 17:01:27.010051 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:27.009866    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/261b83dd-94ac-429f-94f2-9c5cdf39469b-proxy-tls\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-78tfs\" (UID: \"261b83dd-94ac-429f-94f2-9c5cdf39469b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-78tfs"
May 26 17:01:27.010051 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:27.009897    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gjl8n\" (UniqueName: \"kubernetes.io/projected/261b83dd-94ac-429f-94f2-9c5cdf39469b-kube-api-access-gjl8n\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-78tfs\" (UID: \"261b83dd-94ac-429f-94f2-9c5cdf39469b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-78tfs"
May 26 17:01:27.010051 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:27.009928    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/261b83dd-94ac-429f-94f2-9c5cdf39469b-isvc-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-78tfs\" (UID: \"261b83dd-94ac-429f-94f2-9c5cdf39469b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-78tfs"
May 26 17:01:27.010310 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:27.010290    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/261b83dd-94ac-429f-94f2-9c5cdf39469b-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-78tfs\" (UID: \"261b83dd-94ac-429f-94f2-9c5cdf39469b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-78tfs"
May 26 17:01:27.010557 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:27.010540    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/261b83dd-94ac-429f-94f2-9c5cdf39469b-isvc-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-78tfs\" (UID: \"261b83dd-94ac-429f-94f2-9c5cdf39469b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-78tfs"
May 26 17:01:27.012339 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:27.012321    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/261b83dd-94ac-429f-94f2-9c5cdf39469b-proxy-tls\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-78tfs\" (UID: \"261b83dd-94ac-429f-94f2-9c5cdf39469b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-78tfs"
May 26 17:01:27.018851 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:27.018826    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjl8n\" (UniqueName: \"kubernetes.io/projected/261b83dd-94ac-429f-94f2-9c5cdf39469b-kube-api-access-gjl8n\") pod \"isvc-xgboost-v2-predictor-6fcdd6977c-78tfs\" (UID: \"261b83dd-94ac-429f-94f2-9c5cdf39469b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-78tfs"
May 26 17:01:27.072370 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:27.072333    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-78tfs"
May 26 17:01:27.199506 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:27.199437    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-78tfs"]
May 26 17:01:27.202209 ip-10-0-130-83 kubenswrapper[2572]: W0526 17:01:27.202178    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod261b83dd_94ac_429f_94f2_9c5cdf39469b.slice/crio-a6f57a7303f73201ff0e4b2915febc4aea18f48d874e8f897856316ef81a360d WatchSource:0}: Error finding container a6f57a7303f73201ff0e4b2915febc4aea18f48d874e8f897856316ef81a360d: Status 404 returned error can't find the container with id a6f57a7303f73201ff0e4b2915febc4aea18f48d874e8f897856316ef81a360d
May 26 17:01:27.204035 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:27.204019    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 17:01:27.568252 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:27.568160    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-78tfs" event={"ID":"261b83dd-94ac-429f-94f2-9c5cdf39469b","Type":"ContainerStarted","Data":"645ab74d21fc6cdd46d5e293d1d01e8f8a6298c479522ea1f8340212918d6913"}
May 26 17:01:27.568252 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:27.568200    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-78tfs" event={"ID":"261b83dd-94ac-429f-94f2-9c5cdf39469b","Type":"ContainerStarted","Data":"a6f57a7303f73201ff0e4b2915febc4aea18f48d874e8f897856316ef81a360d"}
May 26 17:01:27.570217 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:27.570188    2572 generic.go:358] "Generic (PLEG): container finished" podID="eb522444-10c8-4ea9-8987-42007540e211" containerID="90d24c1abb2d257cfc92e89ccc2a828830a76ad3efa2109dcaccaadef0e28cf5" exitCode=2
May 26 17:01:27.570348 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:27.570265    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4" event={"ID":"eb522444-10c8-4ea9-8987-42007540e211","Type":"ContainerDied","Data":"90d24c1abb2d257cfc92e89ccc2a828830a76ad3efa2109dcaccaadef0e28cf5"}
May 26 17:01:30.424895 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:30.424856    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4" podUID="eb522444-10c8-4ea9-8987-42007540e211" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.59:8643/healthz\": dial tcp 10.132.0.59:8643: connect: connection refused"
May 26 17:01:30.430877 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:30.430838    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4" podUID="eb522444-10c8-4ea9-8987-42007540e211" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.59:8080/v2/models/isvc-xgboost-v2-runtime/ready\": dial tcp 10.132.0.59:8080: connect: connection refused"
May 26 17:01:31.581752 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:31.581713    2572 generic.go:358] "Generic (PLEG): container finished" podID="261b83dd-94ac-429f-94f2-9c5cdf39469b" containerID="645ab74d21fc6cdd46d5e293d1d01e8f8a6298c479522ea1f8340212918d6913" exitCode=0
May 26 17:01:31.582250 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:31.581764    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-78tfs" event={"ID":"261b83dd-94ac-429f-94f2-9c5cdf39469b","Type":"ContainerDied","Data":"645ab74d21fc6cdd46d5e293d1d01e8f8a6298c479522ea1f8340212918d6913"}
May 26 17:01:32.586888 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:32.586851    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-78tfs" event={"ID":"261b83dd-94ac-429f-94f2-9c5cdf39469b","Type":"ContainerStarted","Data":"d765ebccc9ae3e7932ec746abd564565e18628bb5e40a0e41ee439ee09973830"}
May 26 17:01:32.587283 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:32.586902    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-78tfs" event={"ID":"261b83dd-94ac-429f-94f2-9c5cdf39469b","Type":"ContainerStarted","Data":"b353339c90fe0ec38b365447b2881e8969f258a879393d9d421aa253bfe9a6e6"}
May 26 17:01:32.587283 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:32.587098    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-78tfs"
May 26 17:01:32.605968 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:32.605921    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-78tfs" podStartSLOduration=6.605906145 podStartE2EDuration="6.605906145s" podCreationTimestamp="2026-05-26 17:01:26 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 17:01:32.604480881 +0000 UTC m=+3609.080208232" watchObservedRunningTime="2026-05-26 17:01:32.605906145 +0000 UTC m=+3609.081633495"
May 26 17:01:33.589907 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:33.589868    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-78tfs"
May 26 17:01:33.591028 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:33.590992    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-78tfs" podUID="261b83dd-94ac-429f-94f2-9c5cdf39469b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
May 26 17:01:34.360579 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:34.360553    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4"
May 26 17:01:34.467314 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:34.467221    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dj59l\" (UniqueName: \"kubernetes.io/projected/eb522444-10c8-4ea9-8987-42007540e211-kube-api-access-dj59l\") pod \"eb522444-10c8-4ea9-8987-42007540e211\" (UID: \"eb522444-10c8-4ea9-8987-42007540e211\") "
May 26 17:01:34.467314 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:34.467274    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb522444-10c8-4ea9-8987-42007540e211-proxy-tls\") pod \"eb522444-10c8-4ea9-8987-42007540e211\" (UID: \"eb522444-10c8-4ea9-8987-42007540e211\") "
May 26 17:01:34.467518 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:34.467325    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb522444-10c8-4ea9-8987-42007540e211-kserve-provision-location\") pod \"eb522444-10c8-4ea9-8987-42007540e211\" (UID: \"eb522444-10c8-4ea9-8987-42007540e211\") "
May 26 17:01:34.467518 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:34.467350    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb522444-10c8-4ea9-8987-42007540e211-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\") pod \"eb522444-10c8-4ea9-8987-42007540e211\" (UID: \"eb522444-10c8-4ea9-8987-42007540e211\") "
May 26 17:01:34.467672 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:34.467649    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb522444-10c8-4ea9-8987-42007540e211-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "eb522444-10c8-4ea9-8987-42007540e211" (UID: "eb522444-10c8-4ea9-8987-42007540e211"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 17:01:34.467737 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:34.467720    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb522444-10c8-4ea9-8987-42007540e211-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config") pod "eb522444-10c8-4ea9-8987-42007540e211" (UID: "eb522444-10c8-4ea9-8987-42007540e211"). InnerVolumeSpecName "isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 17:01:34.469356 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:34.469334    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb522444-10c8-4ea9-8987-42007540e211-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "eb522444-10c8-4ea9-8987-42007540e211" (UID: "eb522444-10c8-4ea9-8987-42007540e211"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 17:01:34.469443 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:34.469359    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb522444-10c8-4ea9-8987-42007540e211-kube-api-access-dj59l" (OuterVolumeSpecName: "kube-api-access-dj59l") pod "eb522444-10c8-4ea9-8987-42007540e211" (UID: "eb522444-10c8-4ea9-8987-42007540e211"). InnerVolumeSpecName "kube-api-access-dj59l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 17:01:34.567837 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:34.567802    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dj59l\" (UniqueName: \"kubernetes.io/projected/eb522444-10c8-4ea9-8987-42007540e211-kube-api-access-dj59l\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 17:01:34.567837 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:34.567829    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb522444-10c8-4ea9-8987-42007540e211-proxy-tls\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 17:01:34.567837 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:34.567840    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb522444-10c8-4ea9-8987-42007540e211-kserve-provision-location\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 17:01:34.568079 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:34.567852    2572 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb522444-10c8-4ea9-8987-42007540e211-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 17:01:34.593383 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:34.593347    2572 generic.go:358] "Generic (PLEG): container finished" podID="eb522444-10c8-4ea9-8987-42007540e211" containerID="9accee429330fabf0ebfbca313a36a18efef97a86313a526955388bb9a606603" exitCode=0
May 26 17:01:34.593761 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:34.593438    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4"
May 26 17:01:34.593761 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:34.593434    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4" event={"ID":"eb522444-10c8-4ea9-8987-42007540e211","Type":"ContainerDied","Data":"9accee429330fabf0ebfbca313a36a18efef97a86313a526955388bb9a606603"}
May 26 17:01:34.593761 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:34.593546    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4" event={"ID":"eb522444-10c8-4ea9-8987-42007540e211","Type":"ContainerDied","Data":"92337e4a05200045ec675251d27883074e6884c16c97639870739322bb7297f7"}
May 26 17:01:34.593761 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:34.593563    2572 scope.go:117] "RemoveContainer" containerID="90d24c1abb2d257cfc92e89ccc2a828830a76ad3efa2109dcaccaadef0e28cf5"
May 26 17:01:34.594209 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:34.594184    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-78tfs" podUID="261b83dd-94ac-429f-94f2-9c5cdf39469b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
May 26 17:01:34.601772 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:34.601754    2572 scope.go:117] "RemoveContainer" containerID="9accee429330fabf0ebfbca313a36a18efef97a86313a526955388bb9a606603"
May 26 17:01:34.608908 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:34.608888    2572 scope.go:117] "RemoveContainer" containerID="049568b50ac84e1e9ca51e4ad89e9852e8fdc56856ccf2fce9d000aefdaaa238"
May 26 17:01:34.614464 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:34.614431    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4"]
May 26 17:01:34.616687 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:34.616663    2572 scope.go:117] "RemoveContainer" containerID="90d24c1abb2d257cfc92e89ccc2a828830a76ad3efa2109dcaccaadef0e28cf5"
May 26 17:01:34.616965 ip-10-0-130-83 kubenswrapper[2572]: E0526 17:01:34.616928    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90d24c1abb2d257cfc92e89ccc2a828830a76ad3efa2109dcaccaadef0e28cf5\": container with ID starting with 90d24c1abb2d257cfc92e89ccc2a828830a76ad3efa2109dcaccaadef0e28cf5 not found: ID does not exist" containerID="90d24c1abb2d257cfc92e89ccc2a828830a76ad3efa2109dcaccaadef0e28cf5"
May 26 17:01:34.617101 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:34.617003    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90d24c1abb2d257cfc92e89ccc2a828830a76ad3efa2109dcaccaadef0e28cf5"} err="failed to get container status \"90d24c1abb2d257cfc92e89ccc2a828830a76ad3efa2109dcaccaadef0e28cf5\": rpc error: code = NotFound desc = could not find container \"90d24c1abb2d257cfc92e89ccc2a828830a76ad3efa2109dcaccaadef0e28cf5\": container with ID starting with 90d24c1abb2d257cfc92e89ccc2a828830a76ad3efa2109dcaccaadef0e28cf5 not found: ID does not exist"
May 26 17:01:34.617101 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:34.617035    2572 scope.go:117] "RemoveContainer" containerID="9accee429330fabf0ebfbca313a36a18efef97a86313a526955388bb9a606603"
May 26 17:01:34.617366 ip-10-0-130-83 kubenswrapper[2572]: E0526 17:01:34.617343    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9accee429330fabf0ebfbca313a36a18efef97a86313a526955388bb9a606603\": container with ID starting with 9accee429330fabf0ebfbca313a36a18efef97a86313a526955388bb9a606603 not found: ID does not exist" containerID="9accee429330fabf0ebfbca313a36a18efef97a86313a526955388bb9a606603"
May 26 17:01:34.617435 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:34.617372    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9accee429330fabf0ebfbca313a36a18efef97a86313a526955388bb9a606603"} err="failed to get container status \"9accee429330fabf0ebfbca313a36a18efef97a86313a526955388bb9a606603\": rpc error: code = NotFound desc = could not find container \"9accee429330fabf0ebfbca313a36a18efef97a86313a526955388bb9a606603\": container with ID starting with 9accee429330fabf0ebfbca313a36a18efef97a86313a526955388bb9a606603 not found: ID does not exist"
May 26 17:01:34.617435 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:34.617392    2572 scope.go:117] "RemoveContainer" containerID="049568b50ac84e1e9ca51e4ad89e9852e8fdc56856ccf2fce9d000aefdaaa238"
May 26 17:01:34.617861 ip-10-0-130-83 kubenswrapper[2572]: E0526 17:01:34.617824    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"049568b50ac84e1e9ca51e4ad89e9852e8fdc56856ccf2fce9d000aefdaaa238\": container with ID starting with 049568b50ac84e1e9ca51e4ad89e9852e8fdc56856ccf2fce9d000aefdaaa238 not found: ID does not exist" containerID="049568b50ac84e1e9ca51e4ad89e9852e8fdc56856ccf2fce9d000aefdaaa238"
May 26 17:01:34.618023 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:34.618003    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"049568b50ac84e1e9ca51e4ad89e9852e8fdc56856ccf2fce9d000aefdaaa238"} err="failed to get container status \"049568b50ac84e1e9ca51e4ad89e9852e8fdc56856ccf2fce9d000aefdaaa238\": rpc error: code = NotFound desc = could not find container \"049568b50ac84e1e9ca51e4ad89e9852e8fdc56856ccf2fce9d000aefdaaa238\": container with ID starting with 049568b50ac84e1e9ca51e4ad89e9852e8fdc56856ccf2fce9d000aefdaaa238 not found: ID does not exist"
May 26 17:01:34.618914 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:34.618895    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mpsj4"]
May 26 17:01:36.079510 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:36.079473    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb522444-10c8-4ea9-8987-42007540e211" path="/var/lib/kubelet/pods/eb522444-10c8-4ea9-8987-42007540e211/volumes"
May 26 17:01:39.599410 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:39.599382    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-78tfs"
May 26 17:01:39.600038 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:39.600009    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-78tfs" podUID="261b83dd-94ac-429f-94f2-9c5cdf39469b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
May 26 17:01:49.600258 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:49.600219    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-78tfs" podUID="261b83dd-94ac-429f-94f2-9c5cdf39469b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
May 26 17:01:59.600474 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:01:59.600431    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-78tfs" podUID="261b83dd-94ac-429f-94f2-9c5cdf39469b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
May 26 17:02:09.600756 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:09.600709    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-78tfs" podUID="261b83dd-94ac-429f-94f2-9c5cdf39469b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
May 26 17:02:19.600181 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:19.600138    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-78tfs" podUID="261b83dd-94ac-429f-94f2-9c5cdf39469b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
May 26 17:02:29.600659 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:29.600545    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-78tfs" podUID="261b83dd-94ac-429f-94f2-9c5cdf39469b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
May 26 17:02:39.600748 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:39.600714    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-78tfs"
May 26 17:02:46.782999 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:46.782896    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-78tfs"]
May 26 17:02:46.783419 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:46.783320    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-78tfs" podUID="261b83dd-94ac-429f-94f2-9c5cdf39469b" containerName="kserve-container" containerID="cri-o://b353339c90fe0ec38b365447b2881e8969f258a879393d9d421aa253bfe9a6e6" gracePeriod=30
May 26 17:02:46.783419 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:46.783372    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-78tfs" podUID="261b83dd-94ac-429f-94f2-9c5cdf39469b" containerName="kube-rbac-proxy" containerID="cri-o://d765ebccc9ae3e7932ec746abd564565e18628bb5e40a0e41ee439ee09973830" gracePeriod=30
May 26 17:02:46.875109 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:46.875070    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5"]
May 26 17:02:46.875363 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:46.875351    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb522444-10c8-4ea9-8987-42007540e211" containerName="kube-rbac-proxy"
May 26 17:02:46.875418 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:46.875365    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb522444-10c8-4ea9-8987-42007540e211" containerName="kube-rbac-proxy"
May 26 17:02:46.875418 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:46.875375    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb522444-10c8-4ea9-8987-42007540e211" containerName="storage-initializer"
May 26 17:02:46.875418 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:46.875383    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb522444-10c8-4ea9-8987-42007540e211" containerName="storage-initializer"
May 26 17:02:46.875418 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:46.875395    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb522444-10c8-4ea9-8987-42007540e211" containerName="kserve-container"
May 26 17:02:46.875418 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:46.875402    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb522444-10c8-4ea9-8987-42007540e211" containerName="kserve-container"
May 26 17:02:46.875579 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:46.875443    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="eb522444-10c8-4ea9-8987-42007540e211" containerName="kserve-container"
May 26 17:02:46.875579 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:46.875451    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="eb522444-10c8-4ea9-8987-42007540e211" containerName="kube-rbac-proxy"
May 26 17:02:46.880752 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:46.880256    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5"
May 26 17:02:46.883142 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:46.883114    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-predictor-serving-cert\""
May 26 17:02:46.883298 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:46.883139    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
May 26 17:02:46.883298 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:46.883181    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
May 26 17:02:46.890749 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:46.890720    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5"]
May 26 17:02:47.009575 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:47.009536    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065-proxy-tls\") pod \"isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5\" (UID: \"a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5"
May 26 17:02:47.009815 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:47.009681    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5\" (UID: \"a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5"
May 26 17:02:47.009815 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:47.009748    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065-isvc-sklearn-s3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5\" (UID: \"a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5"
May 26 17:02:47.009815 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:47.009775    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbkc5\" (UniqueName: \"kubernetes.io/projected/a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065-kube-api-access-cbkc5\") pod \"isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5\" (UID: \"a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5"
May 26 17:02:47.111112 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:47.111011    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065-isvc-sklearn-s3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5\" (UID: \"a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5"
May 26 17:02:47.111112 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:47.111056    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cbkc5\" (UniqueName: \"kubernetes.io/projected/a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065-kube-api-access-cbkc5\") pod \"isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5\" (UID: \"a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5"
May 26 17:02:47.111350 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:47.111134    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065-proxy-tls\") pod \"isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5\" (UID: \"a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5"
May 26 17:02:47.111350 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:47.111160    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5\" (UID: \"a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5"
May 26 17:02:47.111573 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:47.111550    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5\" (UID: \"a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5"
May 26 17:02:47.111724 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:47.111705    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065-isvc-sklearn-s3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5\" (UID: \"a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5"
May 26 17:02:47.113422 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:47.113404    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065-proxy-tls\") pod \"isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5\" (UID: \"a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5"
May 26 17:02:47.121014 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:47.120989    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbkc5\" (UniqueName: \"kubernetes.io/projected/a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065-kube-api-access-cbkc5\") pod \"isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5\" (UID: \"a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5"
May 26 17:02:47.191820 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:47.191780    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5"
May 26 17:02:47.325453 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:47.325388    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5"]
May 26 17:02:47.327678 ip-10-0-130-83 kubenswrapper[2572]: W0526 17:02:47.327645    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4f3a7f4_78e9_4db0_a2c1_d7edbf22d065.slice/crio-23467885bf715436619066641662ea5719f6f66a0fb328021a172a1f98337b20 WatchSource:0}: Error finding container 23467885bf715436619066641662ea5719f6f66a0fb328021a172a1f98337b20: Status 404 returned error can't find the container with id 23467885bf715436619066641662ea5719f6f66a0fb328021a172a1f98337b20
May 26 17:02:47.790039 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:47.789943    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5" event={"ID":"a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065","Type":"ContainerStarted","Data":"ab1ec8440bafb4a1dae8eb50417e06db98bb1e86bfaecfbdbee013af84a7b8c4"}
May 26 17:02:47.790039 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:47.789992    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5" event={"ID":"a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065","Type":"ContainerStarted","Data":"23467885bf715436619066641662ea5719f6f66a0fb328021a172a1f98337b20"}
May 26 17:02:47.792080 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:47.792051    2572 generic.go:358] "Generic (PLEG): container finished" podID="261b83dd-94ac-429f-94f2-9c5cdf39469b" containerID="d765ebccc9ae3e7932ec746abd564565e18628bb5e40a0e41ee439ee09973830" exitCode=2
May 26 17:02:47.792191 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:47.792101    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-78tfs" event={"ID":"261b83dd-94ac-429f-94f2-9c5cdf39469b","Type":"ContainerDied","Data":"d765ebccc9ae3e7932ec746abd564565e18628bb5e40a0e41ee439ee09973830"}
May 26 17:02:48.796371 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:48.796335    2572 generic.go:358] "Generic (PLEG): container finished" podID="a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065" containerID="ab1ec8440bafb4a1dae8eb50417e06db98bb1e86bfaecfbdbee013af84a7b8c4" exitCode=0
May 26 17:02:48.796868 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:48.796418    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5" event={"ID":"a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065","Type":"ContainerDied","Data":"ab1ec8440bafb4a1dae8eb50417e06db98bb1e86bfaecfbdbee013af84a7b8c4"}
May 26 17:02:49.594717 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:49.594677    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-78tfs" podUID="261b83dd-94ac-429f-94f2-9c5cdf39469b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.60:8643/healthz\": dial tcp 10.132.0.60:8643: connect: connection refused"
May 26 17:02:49.600393 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:49.600355    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-78tfs" podUID="261b83dd-94ac-429f-94f2-9c5cdf39469b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
May 26 17:02:49.801066 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:49.801030    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5" event={"ID":"a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065","Type":"ContainerStarted","Data":"a664d95afbd415121a52a5312ba6462675316dbb98914c96ab1a03338460106b"}
May 26 17:02:49.801066 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:49.801072    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5" event={"ID":"a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065","Type":"ContainerStarted","Data":"2124dcce8ff411c737fc60c71389f97144fb7006e0e3e3ea7072e6d5d153fe7d"}
May 26 17:02:49.801473 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:49.801162    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5"
May 26 17:02:49.821990 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:49.821935    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5" podStartSLOduration=3.821920801 podStartE2EDuration="3.821920801s" podCreationTimestamp="2026-05-26 17:02:46 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 17:02:49.820018979 +0000 UTC m=+3686.295746326" watchObservedRunningTime="2026-05-26 17:02:49.821920801 +0000 UTC m=+3686.297648145"
May 26 17:02:50.625815 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:50.625792    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-78tfs"
May 26 17:02:50.737502 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:50.737414    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjl8n\" (UniqueName: \"kubernetes.io/projected/261b83dd-94ac-429f-94f2-9c5cdf39469b-kube-api-access-gjl8n\") pod \"261b83dd-94ac-429f-94f2-9c5cdf39469b\" (UID: \"261b83dd-94ac-429f-94f2-9c5cdf39469b\") "
May 26 17:02:50.737502 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:50.737466    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/261b83dd-94ac-429f-94f2-9c5cdf39469b-proxy-tls\") pod \"261b83dd-94ac-429f-94f2-9c5cdf39469b\" (UID: \"261b83dd-94ac-429f-94f2-9c5cdf39469b\") "
May 26 17:02:50.737786 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:50.737551    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/261b83dd-94ac-429f-94f2-9c5cdf39469b-isvc-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"261b83dd-94ac-429f-94f2-9c5cdf39469b\" (UID: \"261b83dd-94ac-429f-94f2-9c5cdf39469b\") "
May 26 17:02:50.737786 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:50.737580    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/261b83dd-94ac-429f-94f2-9c5cdf39469b-kserve-provision-location\") pod \"261b83dd-94ac-429f-94f2-9c5cdf39469b\" (UID: \"261b83dd-94ac-429f-94f2-9c5cdf39469b\") "
May 26 17:02:50.737936 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:50.737905    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/261b83dd-94ac-429f-94f2-9c5cdf39469b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "261b83dd-94ac-429f-94f2-9c5cdf39469b" (UID: "261b83dd-94ac-429f-94f2-9c5cdf39469b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 17:02:50.737991 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:50.737925    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/261b83dd-94ac-429f-94f2-9c5cdf39469b-isvc-xgboost-v2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-v2-kube-rbac-proxy-sar-config") pod "261b83dd-94ac-429f-94f2-9c5cdf39469b" (UID: "261b83dd-94ac-429f-94f2-9c5cdf39469b"). InnerVolumeSpecName "isvc-xgboost-v2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 17:02:50.739620 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:50.739587    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/261b83dd-94ac-429f-94f2-9c5cdf39469b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "261b83dd-94ac-429f-94f2-9c5cdf39469b" (UID: "261b83dd-94ac-429f-94f2-9c5cdf39469b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 17:02:50.739732 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:50.739705    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/261b83dd-94ac-429f-94f2-9c5cdf39469b-kube-api-access-gjl8n" (OuterVolumeSpecName: "kube-api-access-gjl8n") pod "261b83dd-94ac-429f-94f2-9c5cdf39469b" (UID: "261b83dd-94ac-429f-94f2-9c5cdf39469b"). InnerVolumeSpecName "kube-api-access-gjl8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 17:02:50.805688 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:50.805653    2572 generic.go:358] "Generic (PLEG): container finished" podID="261b83dd-94ac-429f-94f2-9c5cdf39469b" containerID="b353339c90fe0ec38b365447b2881e8969f258a879393d9d421aa253bfe9a6e6" exitCode=0
May 26 17:02:50.806097 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:50.805735    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-78tfs"
May 26 17:02:50.806097 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:50.805738    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-78tfs" event={"ID":"261b83dd-94ac-429f-94f2-9c5cdf39469b","Type":"ContainerDied","Data":"b353339c90fe0ec38b365447b2881e8969f258a879393d9d421aa253bfe9a6e6"}
May 26 17:02:50.806097 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:50.805776    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-78tfs" event={"ID":"261b83dd-94ac-429f-94f2-9c5cdf39469b","Type":"ContainerDied","Data":"a6f57a7303f73201ff0e4b2915febc4aea18f48d874e8f897856316ef81a360d"}
May 26 17:02:50.806097 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:50.805792    2572 scope.go:117] "RemoveContainer" containerID="d765ebccc9ae3e7932ec746abd564565e18628bb5e40a0e41ee439ee09973830"
May 26 17:02:50.806322 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:50.806303    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5"
May 26 17:02:50.807958 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:50.807932    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5" podUID="a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
May 26 17:02:50.814756 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:50.814735    2572 scope.go:117] "RemoveContainer" containerID="b353339c90fe0ec38b365447b2881e8969f258a879393d9d421aa253bfe9a6e6"
May 26 17:02:50.821519 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:50.821502    2572 scope.go:117] "RemoveContainer" containerID="645ab74d21fc6cdd46d5e293d1d01e8f8a6298c479522ea1f8340212918d6913"
May 26 17:02:50.828361 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:50.828338    2572 scope.go:117] "RemoveContainer" containerID="d765ebccc9ae3e7932ec746abd564565e18628bb5e40a0e41ee439ee09973830"
May 26 17:02:50.828696 ip-10-0-130-83 kubenswrapper[2572]: E0526 17:02:50.828664    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d765ebccc9ae3e7932ec746abd564565e18628bb5e40a0e41ee439ee09973830\": container with ID starting with d765ebccc9ae3e7932ec746abd564565e18628bb5e40a0e41ee439ee09973830 not found: ID does not exist" containerID="d765ebccc9ae3e7932ec746abd564565e18628bb5e40a0e41ee439ee09973830"
May 26 17:02:50.828785 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:50.828705    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d765ebccc9ae3e7932ec746abd564565e18628bb5e40a0e41ee439ee09973830"} err="failed to get container status \"d765ebccc9ae3e7932ec746abd564565e18628bb5e40a0e41ee439ee09973830\": rpc error: code = NotFound desc = could not find container \"d765ebccc9ae3e7932ec746abd564565e18628bb5e40a0e41ee439ee09973830\": container with ID starting with d765ebccc9ae3e7932ec746abd564565e18628bb5e40a0e41ee439ee09973830 not found: ID does not exist"
May 26 17:02:50.828785 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:50.828725    2572 scope.go:117] "RemoveContainer" containerID="b353339c90fe0ec38b365447b2881e8969f258a879393d9d421aa253bfe9a6e6"
May 26 17:02:50.828997 ip-10-0-130-83 kubenswrapper[2572]: E0526 17:02:50.828976    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b353339c90fe0ec38b365447b2881e8969f258a879393d9d421aa253bfe9a6e6\": container with ID starting with b353339c90fe0ec38b365447b2881e8969f258a879393d9d421aa253bfe9a6e6 not found: ID does not exist" containerID="b353339c90fe0ec38b365447b2881e8969f258a879393d9d421aa253bfe9a6e6"
May 26 17:02:50.829046 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:50.829005    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-78tfs"]
May 26 17:02:50.829046 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:50.829006    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b353339c90fe0ec38b365447b2881e8969f258a879393d9d421aa253bfe9a6e6"} err="failed to get container status \"b353339c90fe0ec38b365447b2881e8969f258a879393d9d421aa253bfe9a6e6\": rpc error: code = NotFound desc = could not find container \"b353339c90fe0ec38b365447b2881e8969f258a879393d9d421aa253bfe9a6e6\": container with ID starting with b353339c90fe0ec38b365447b2881e8969f258a879393d9d421aa253bfe9a6e6 not found: ID does not exist"
May 26 17:02:50.829046 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:50.829029    2572 scope.go:117] "RemoveContainer" containerID="645ab74d21fc6cdd46d5e293d1d01e8f8a6298c479522ea1f8340212918d6913"
May 26 17:02:50.829281 ip-10-0-130-83 kubenswrapper[2572]: E0526 17:02:50.829259    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"645ab74d21fc6cdd46d5e293d1d01e8f8a6298c479522ea1f8340212918d6913\": container with ID starting with 645ab74d21fc6cdd46d5e293d1d01e8f8a6298c479522ea1f8340212918d6913 not found: ID does not exist" containerID="645ab74d21fc6cdd46d5e293d1d01e8f8a6298c479522ea1f8340212918d6913"
May 26 17:02:50.829362 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:50.829289    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"645ab74d21fc6cdd46d5e293d1d01e8f8a6298c479522ea1f8340212918d6913"} err="failed to get container status \"645ab74d21fc6cdd46d5e293d1d01e8f8a6298c479522ea1f8340212918d6913\": rpc error: code = NotFound desc = could not find container \"645ab74d21fc6cdd46d5e293d1d01e8f8a6298c479522ea1f8340212918d6913\": container with ID starting with 645ab74d21fc6cdd46d5e293d1d01e8f8a6298c479522ea1f8340212918d6913 not found: ID does not exist"
May 26 17:02:50.833232 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:50.833209    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-6fcdd6977c-78tfs"]
May 26 17:02:50.838968 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:50.838948    2572 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/261b83dd-94ac-429f-94f2-9c5cdf39469b-isvc-xgboost-v2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 17:02:50.839037 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:50.838972    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/261b83dd-94ac-429f-94f2-9c5cdf39469b-kserve-provision-location\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 17:02:50.839037 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:50.838985    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjl8n\" (UniqueName: \"kubernetes.io/projected/261b83dd-94ac-429f-94f2-9c5cdf39469b-kube-api-access-gjl8n\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 17:02:50.839037 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:50.838999    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/261b83dd-94ac-429f-94f2-9c5cdf39469b-proxy-tls\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 17:02:51.808655 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:51.808595    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5" podUID="a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
May 26 17:02:52.079182 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:52.079101    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="261b83dd-94ac-429f-94f2-9c5cdf39469b" path="/var/lib/kubelet/pods/261b83dd-94ac-429f-94f2-9c5cdf39469b/volumes"
May 26 17:02:56.812995 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:56.812960    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5"
May 26 17:02:56.813551 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:02:56.813523    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5" podUID="a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
May 26 17:03:06.813652 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:03:06.813590    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5" podUID="a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
May 26 17:03:16.813533 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:03:16.813495    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5" podUID="a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
May 26 17:03:26.814032 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:03:26.813994    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5" podUID="a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
May 26 17:03:36.813459 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:03:36.813415    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5" podUID="a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
May 26 17:03:46.813511 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:03:46.813469    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5" podUID="a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
May 26 17:03:56.814759 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:03:56.814725    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5"
May 26 17:04:06.965447 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:06.965411    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5"]
May 26 17:04:06.965878 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:06.965821    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5" podUID="a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065" containerName="kserve-container" containerID="cri-o://2124dcce8ff411c737fc60c71389f97144fb7006e0e3e3ea7072e6d5d153fe7d" gracePeriod=30
May 26 17:04:06.965983 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:06.965896    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5" podUID="a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065" containerName="kube-rbac-proxy" containerID="cri-o://a664d95afbd415121a52a5312ba6462675316dbb98914c96ab1a03338460106b" gracePeriod=30
May 26 17:04:07.061374 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:07.061336    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm"]
May 26 17:04:07.061716 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:07.061699    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="261b83dd-94ac-429f-94f2-9c5cdf39469b" containerName="kube-rbac-proxy"
May 26 17:04:07.061801 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:07.061718    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="261b83dd-94ac-429f-94f2-9c5cdf39469b" containerName="kube-rbac-proxy"
May 26 17:04:07.061801 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:07.061733    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="261b83dd-94ac-429f-94f2-9c5cdf39469b" containerName="storage-initializer"
May 26 17:04:07.061801 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:07.061741    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="261b83dd-94ac-429f-94f2-9c5cdf39469b" containerName="storage-initializer"
May 26 17:04:07.061801 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:07.061759    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="261b83dd-94ac-429f-94f2-9c5cdf39469b" containerName="kserve-container"
May 26 17:04:07.061801 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:07.061768    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="261b83dd-94ac-429f-94f2-9c5cdf39469b" containerName="kserve-container"
May 26 17:04:07.062085 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:07.061848    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="261b83dd-94ac-429f-94f2-9c5cdf39469b" containerName="kserve-container"
May 26 17:04:07.062085 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:07.061862    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="261b83dd-94ac-429f-94f2-9c5cdf39469b" containerName="kube-rbac-proxy"
May 26 17:04:07.064193 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:07.064171    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm"
May 26 17:04:07.066358 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:07.066335    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
May 26 17:04:07.066885 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:07.066858    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
May 26 17:04:07.067041 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:07.066874    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
May 26 17:04:07.077563 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:07.077525    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm"]
May 26 17:04:07.117932 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:07.117879    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm\" (UID: \"ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm"
May 26 17:04:07.117932 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:07.117938    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckr67\" (UniqueName: \"kubernetes.io/projected/ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63-kube-api-access-ckr67\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm\" (UID: \"ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm"
May 26 17:04:07.118189 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:07.118016    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm\" (UID: \"ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm"
May 26 17:04:07.118189 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:07.118063    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm\" (UID: \"ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm"
May 26 17:04:07.118189 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:07.118145    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm\" (UID: \"ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm"
May 26 17:04:07.219303 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:07.219195    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm\" (UID: \"ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm"
May 26 17:04:07.219303 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:07.219284    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm\" (UID: \"ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm"
May 26 17:04:07.219575 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:07.219325    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ckr67\" (UniqueName: \"kubernetes.io/projected/ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63-kube-api-access-ckr67\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm\" (UID: \"ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm"
May 26 17:04:07.219575 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:07.219359    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm\" (UID: \"ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm"
May 26 17:04:07.219575 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:07.219387    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm\" (UID: \"ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm"
May 26 17:04:07.219804 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:07.219771    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm\" (UID: \"ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm"
May 26 17:04:07.219892 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:07.219871    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm\" (UID: \"ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm"
May 26 17:04:07.219933 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:07.219924    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm\" (UID: \"ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm"
May 26 17:04:07.221833 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:07.221815    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm\" (UID: \"ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm"
May 26 17:04:07.228208 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:07.228183    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckr67\" (UniqueName: \"kubernetes.io/projected/ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63-kube-api-access-ckr67\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm\" (UID: \"ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm"
May 26 17:04:07.382325 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:07.382286    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm"
May 26 17:04:07.510754 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:07.510668    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm"]
May 26 17:04:07.513645 ip-10-0-130-83 kubenswrapper[2572]: W0526 17:04:07.513612    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce6fc6d0_e5a5_4880_b2f7_b24f5fe7bb63.slice/crio-ef3a217f3f7a267aa43a43da0f54aff1117f68900abc413ce08e41bb4890fed1 WatchSource:0}: Error finding container ef3a217f3f7a267aa43a43da0f54aff1117f68900abc413ce08e41bb4890fed1: Status 404 returned error can't find the container with id ef3a217f3f7a267aa43a43da0f54aff1117f68900abc413ce08e41bb4890fed1
May 26 17:04:08.015738 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:08.015709    2572 generic.go:358] "Generic (PLEG): container finished" podID="a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065" containerID="a664d95afbd415121a52a5312ba6462675316dbb98914c96ab1a03338460106b" exitCode=2
May 26 17:04:08.016151 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:08.015785    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5" event={"ID":"a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065","Type":"ContainerDied","Data":"a664d95afbd415121a52a5312ba6462675316dbb98914c96ab1a03338460106b"}
May 26 17:04:08.017123 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:08.017099    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm" event={"ID":"ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63","Type":"ContainerStarted","Data":"7e0d246a6ab494cbe956507777c62a9c740968a1ea347e7a0e3adffc29cf5861"}
May 26 17:04:08.017123 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:08.017127    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm" event={"ID":"ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63","Type":"ContainerStarted","Data":"ef3a217f3f7a267aa43a43da0f54aff1117f68900abc413ce08e41bb4890fed1"}
May 26 17:04:09.021558 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:09.021529    2572 generic.go:358] "Generic (PLEG): container finished" podID="ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63" containerID="7e0d246a6ab494cbe956507777c62a9c740968a1ea347e7a0e3adffc29cf5861" exitCode=0
May 26 17:04:09.021960 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:09.021598    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm" event={"ID":"ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63","Type":"ContainerDied","Data":"7e0d246a6ab494cbe956507777c62a9c740968a1ea347e7a0e3adffc29cf5861"}
May 26 17:04:10.025884 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:10.025850    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm" event={"ID":"ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63","Type":"ContainerStarted","Data":"8830a4b54fe65cbab3b2d09d0887ab11ce4fb1ce80225ac4ad04e24f21ae8532"}
May 26 17:04:10.025884 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:10.025892    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm" event={"ID":"ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63","Type":"ContainerStarted","Data":"19dd4dd5f247ea6e9c0f4fb398ea98a916d18f1995e7c969b5b4dba2545ed4ff"}
May 26 17:04:10.026322 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:10.025989    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm"
May 26 17:04:10.048573 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:10.048522    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm" podStartSLOduration=3.048507952 podStartE2EDuration="3.048507952s" podCreationTimestamp="2026-05-26 17:04:07 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 17:04:10.047041328 +0000 UTC m=+3766.522768679" watchObservedRunningTime="2026-05-26 17:04:10.048507952 +0000 UTC m=+3766.524235301"
May 26 17:04:11.028925 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:11.028888    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm"
May 26 17:04:11.030082 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:11.030056    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm" podUID="ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
May 26 17:04:11.603306 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:11.603282    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5"
May 26 17:04:11.658885 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:11.658855    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065-kserve-provision-location\") pod \"a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065\" (UID: \"a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065\") "
May 26 17:04:11.658885 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:11.658890    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065-isvc-sklearn-s3-kube-rbac-proxy-sar-config\") pod \"a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065\" (UID: \"a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065\") "
May 26 17:04:11.659159 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:11.658911    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065-proxy-tls\") pod \"a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065\" (UID: \"a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065\") "
May 26 17:04:11.659159 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:11.658981    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cbkc5\" (UniqueName: \"kubernetes.io/projected/a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065-kube-api-access-cbkc5\") pod \"a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065\" (UID: \"a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065\") "
May 26 17:04:11.659278 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:11.659209    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065" (UID: "a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 17:04:11.659331 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:11.659285    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065-isvc-sklearn-s3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-kube-rbac-proxy-sar-config") pod "a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065" (UID: "a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065"). InnerVolumeSpecName "isvc-sklearn-s3-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 17:04:11.661002 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:11.660981    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065" (UID: "a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 17:04:11.661124 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:11.661102    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065-kube-api-access-cbkc5" (OuterVolumeSpecName: "kube-api-access-cbkc5") pod "a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065" (UID: "a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065"). InnerVolumeSpecName "kube-api-access-cbkc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 17:04:11.760279 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:11.760187    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbkc5\" (UniqueName: \"kubernetes.io/projected/a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065-kube-api-access-cbkc5\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 17:04:11.760279 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:11.760224    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065-kserve-provision-location\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 17:04:11.760279 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:11.760241    2572 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065-isvc-sklearn-s3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 17:04:11.760279 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:11.760256    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065-proxy-tls\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 17:04:12.032463 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:12.032366    2572 generic.go:358] "Generic (PLEG): container finished" podID="a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065" containerID="2124dcce8ff411c737fc60c71389f97144fb7006e0e3e3ea7072e6d5d153fe7d" exitCode=0
May 26 17:04:12.032463 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:12.032441    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5"
May 26 17:04:12.032463 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:12.032440    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5" event={"ID":"a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065","Type":"ContainerDied","Data":"2124dcce8ff411c737fc60c71389f97144fb7006e0e3e3ea7072e6d5d153fe7d"}
May 26 17:04:12.033080 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:12.032483    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5" event={"ID":"a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065","Type":"ContainerDied","Data":"23467885bf715436619066641662ea5719f6f66a0fb328021a172a1f98337b20"}
May 26 17:04:12.033080 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:12.032504    2572 scope.go:117] "RemoveContainer" containerID="a664d95afbd415121a52a5312ba6462675316dbb98914c96ab1a03338460106b"
May 26 17:04:12.033241 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:12.033220    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm" podUID="ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
May 26 17:04:12.046192 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:12.046171    2572 scope.go:117] "RemoveContainer" containerID="2124dcce8ff411c737fc60c71389f97144fb7006e0e3e3ea7072e6d5d153fe7d"
May 26 17:04:12.053258 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:12.053236    2572 scope.go:117] "RemoveContainer" containerID="ab1ec8440bafb4a1dae8eb50417e06db98bb1e86bfaecfbdbee013af84a7b8c4"
May 26 17:04:12.056872 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:12.056849    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5"]
May 26 17:04:12.060461 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:12.060421    2572 scope.go:117] "RemoveContainer" containerID="a664d95afbd415121a52a5312ba6462675316dbb98914c96ab1a03338460106b"
May 26 17:04:12.061098 ip-10-0-130-83 kubenswrapper[2572]: E0526 17:04:12.061056    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a664d95afbd415121a52a5312ba6462675316dbb98914c96ab1a03338460106b\": container with ID starting with a664d95afbd415121a52a5312ba6462675316dbb98914c96ab1a03338460106b not found: ID does not exist" containerID="a664d95afbd415121a52a5312ba6462675316dbb98914c96ab1a03338460106b"
May 26 17:04:12.061216 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:12.061098    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a664d95afbd415121a52a5312ba6462675316dbb98914c96ab1a03338460106b"} err="failed to get container status \"a664d95afbd415121a52a5312ba6462675316dbb98914c96ab1a03338460106b\": rpc error: code = NotFound desc = could not find container \"a664d95afbd415121a52a5312ba6462675316dbb98914c96ab1a03338460106b\": container with ID starting with a664d95afbd415121a52a5312ba6462675316dbb98914c96ab1a03338460106b not found: ID does not exist"
May 26 17:04:12.061216 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:12.061144    2572 scope.go:117] "RemoveContainer" containerID="2124dcce8ff411c737fc60c71389f97144fb7006e0e3e3ea7072e6d5d153fe7d"
May 26 17:04:12.061478 ip-10-0-130-83 kubenswrapper[2572]: E0526 17:04:12.061450    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2124dcce8ff411c737fc60c71389f97144fb7006e0e3e3ea7072e6d5d153fe7d\": container with ID starting with 2124dcce8ff411c737fc60c71389f97144fb7006e0e3e3ea7072e6d5d153fe7d not found: ID does not exist" containerID="2124dcce8ff411c737fc60c71389f97144fb7006e0e3e3ea7072e6d5d153fe7d"
May 26 17:04:12.061551 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:12.061489    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2124dcce8ff411c737fc60c71389f97144fb7006e0e3e3ea7072e6d5d153fe7d"} err="failed to get container status \"2124dcce8ff411c737fc60c71389f97144fb7006e0e3e3ea7072e6d5d153fe7d\": rpc error: code = NotFound desc = could not find container \"2124dcce8ff411c737fc60c71389f97144fb7006e0e3e3ea7072e6d5d153fe7d\": container with ID starting with 2124dcce8ff411c737fc60c71389f97144fb7006e0e3e3ea7072e6d5d153fe7d not found: ID does not exist"
May 26 17:04:12.061551 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:12.061514    2572 scope.go:117] "RemoveContainer" containerID="ab1ec8440bafb4a1dae8eb50417e06db98bb1e86bfaecfbdbee013af84a7b8c4"
May 26 17:04:12.061844 ip-10-0-130-83 kubenswrapper[2572]: E0526 17:04:12.061820    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab1ec8440bafb4a1dae8eb50417e06db98bb1e86bfaecfbdbee013af84a7b8c4\": container with ID starting with ab1ec8440bafb4a1dae8eb50417e06db98bb1e86bfaecfbdbee013af84a7b8c4 not found: ID does not exist" containerID="ab1ec8440bafb4a1dae8eb50417e06db98bb1e86bfaecfbdbee013af84a7b8c4"
May 26 17:04:12.061951 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:12.061849    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab1ec8440bafb4a1dae8eb50417e06db98bb1e86bfaecfbdbee013af84a7b8c4"} err="failed to get container status \"ab1ec8440bafb4a1dae8eb50417e06db98bb1e86bfaecfbdbee013af84a7b8c4\": rpc error: code = NotFound desc = could not find container \"ab1ec8440bafb4a1dae8eb50417e06db98bb1e86bfaecfbdbee013af84a7b8c4\": container with ID starting with ab1ec8440bafb4a1dae8eb50417e06db98bb1e86bfaecfbdbee013af84a7b8c4 not found: ID does not exist"
May 26 17:04:12.062389 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:12.062371    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-89bbdf6fb-rwhd5"]
May 26 17:04:12.078800 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:12.078777    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065" path="/var/lib/kubelet/pods/a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065/volumes"
May 26 17:04:17.038333 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:17.038261    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm"
May 26 17:04:17.038936 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:17.038911    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm" podUID="ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
May 26 17:04:27.039823 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:27.039780    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm" podUID="ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
May 26 17:04:37.039423 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:37.039386    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm" podUID="ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
May 26 17:04:47.039873 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:47.039824    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm" podUID="ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
May 26 17:04:57.039132 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:04:57.039093    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm" podUID="ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
May 26 17:05:07.039562 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:07.039513    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm" podUID="ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
May 26 17:05:17.039571 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:17.039542    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm"
May 26 17:05:27.087966 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:27.087929    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm"]
May 26 17:05:27.088390 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:27.088241    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm" podUID="ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63" containerName="kserve-container" containerID="cri-o://19dd4dd5f247ea6e9c0f4fb398ea98a916d18f1995e7c969b5b4dba2545ed4ff" gracePeriod=30
May 26 17:05:27.088481 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:27.088421    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm" podUID="ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63" containerName="kube-rbac-proxy" containerID="cri-o://8830a4b54fe65cbab3b2d09d0887ab11ce4fb1ce80225ac4ad04e24f21ae8532" gracePeriod=30
May 26 17:05:27.238967 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:27.238936    2572 generic.go:358] "Generic (PLEG): container finished" podID="ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63" containerID="8830a4b54fe65cbab3b2d09d0887ab11ce4fb1ce80225ac4ad04e24f21ae8532" exitCode=2
May 26 17:05:27.239138 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:27.239008    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm" event={"ID":"ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63","Type":"ContainerDied","Data":"8830a4b54fe65cbab3b2d09d0887ab11ce4fb1ce80225ac4ad04e24f21ae8532"}
May 26 17:05:28.179697 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:28.179661    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db"]
May 26 17:05:28.180077 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:28.179931    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065" containerName="storage-initializer"
May 26 17:05:28.180077 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:28.179941    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065" containerName="storage-initializer"
May 26 17:05:28.180077 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:28.179951    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065" containerName="kube-rbac-proxy"
May 26 17:05:28.180077 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:28.179956    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065" containerName="kube-rbac-proxy"
May 26 17:05:28.180077 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:28.179970    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065" containerName="kserve-container"
May 26 17:05:28.180077 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:28.179977    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065" containerName="kserve-container"
May 26 17:05:28.180077 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:28.180018    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065" containerName="kserve-container"
May 26 17:05:28.180077 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:28.180026    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="a4f3a7f4-78e9-4db0-a2c1-d7edbf22d065" containerName="kube-rbac-proxy"
May 26 17:05:28.182115 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:28.182097    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db"
May 26 17:05:28.184416 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:28.184395    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
May 26 17:05:28.184679 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:28.184663    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
May 26 17:05:28.193845 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:28.193823    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db"]
May 26 17:05:28.228716 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:28.228687    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/850d0ee0-b3bf-4a8f-853b-a88807a481a7-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db\" (UID: \"850d0ee0-b3bf-4a8f-853b-a88807a481a7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db"
May 26 17:05:28.228884 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:28.228726    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmjfz\" (UniqueName: \"kubernetes.io/projected/850d0ee0-b3bf-4a8f-853b-a88807a481a7-kube-api-access-nmjfz\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db\" (UID: \"850d0ee0-b3bf-4a8f-853b-a88807a481a7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db"
May 26 17:05:28.228884 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:28.228771    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/850d0ee0-b3bf-4a8f-853b-a88807a481a7-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db\" (UID: \"850d0ee0-b3bf-4a8f-853b-a88807a481a7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db"
May 26 17:05:28.228884 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:28.228806    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/850d0ee0-b3bf-4a8f-853b-a88807a481a7-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db\" (UID: \"850d0ee0-b3bf-4a8f-853b-a88807a481a7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db"
May 26 17:05:28.329442 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:28.329395    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/850d0ee0-b3bf-4a8f-853b-a88807a481a7-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db\" (UID: \"850d0ee0-b3bf-4a8f-853b-a88807a481a7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db"
May 26 17:05:28.329442 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:28.329451    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nmjfz\" (UniqueName: \"kubernetes.io/projected/850d0ee0-b3bf-4a8f-853b-a88807a481a7-kube-api-access-nmjfz\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db\" (UID: \"850d0ee0-b3bf-4a8f-853b-a88807a481a7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db"
May 26 17:05:28.329757 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:28.329659    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/850d0ee0-b3bf-4a8f-853b-a88807a481a7-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db\" (UID: \"850d0ee0-b3bf-4a8f-853b-a88807a481a7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db"
May 26 17:05:28.329757 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:28.329726    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/850d0ee0-b3bf-4a8f-853b-a88807a481a7-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db\" (UID: \"850d0ee0-b3bf-4a8f-853b-a88807a481a7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db"
May 26 17:05:28.330065 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:28.330040    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/850d0ee0-b3bf-4a8f-853b-a88807a481a7-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db\" (UID: \"850d0ee0-b3bf-4a8f-853b-a88807a481a7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db"
May 26 17:05:28.330335 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:28.330304    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/850d0ee0-b3bf-4a8f-853b-a88807a481a7-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db\" (UID: \"850d0ee0-b3bf-4a8f-853b-a88807a481a7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db"
May 26 17:05:28.331908 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:28.331887    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/850d0ee0-b3bf-4a8f-853b-a88807a481a7-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db\" (UID: \"850d0ee0-b3bf-4a8f-853b-a88807a481a7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db"
May 26 17:05:28.338699 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:28.338677    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmjfz\" (UniqueName: \"kubernetes.io/projected/850d0ee0-b3bf-4a8f-853b-a88807a481a7-kube-api-access-nmjfz\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db\" (UID: \"850d0ee0-b3bf-4a8f-853b-a88807a481a7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db"
May 26 17:05:28.493347 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:28.493257    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db"
May 26 17:05:28.619557 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:28.619494    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db"]
May 26 17:05:28.624075 ip-10-0-130-83 kubenswrapper[2572]: W0526 17:05:28.624047    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod850d0ee0_b3bf_4a8f_853b_a88807a481a7.slice/crio-72ec4b343d56d8fcdfbcac67b27a6f29b5df37d656269eaac58c2104c245e329 WatchSource:0}: Error finding container 72ec4b343d56d8fcdfbcac67b27a6f29b5df37d656269eaac58c2104c245e329: Status 404 returned error can't find the container with id 72ec4b343d56d8fcdfbcac67b27a6f29b5df37d656269eaac58c2104c245e329
May 26 17:05:29.248926 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:29.248889    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db" event={"ID":"850d0ee0-b3bf-4a8f-853b-a88807a481a7","Type":"ContainerStarted","Data":"fabd7b27106a63f2e8ddba1fd77171f22d7bc83907a6901b557ad41c17d0a159"}
May 26 17:05:29.248926 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:29.248930    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db" event={"ID":"850d0ee0-b3bf-4a8f-853b-a88807a481a7","Type":"ContainerStarted","Data":"72ec4b343d56d8fcdfbcac67b27a6f29b5df37d656269eaac58c2104c245e329"}
May 26 17:05:31.666133 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:31.666108    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm"
May 26 17:05:31.756127 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:31.756036    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63-cabundle-cert\") pod \"ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63\" (UID: \"ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63\") "
May 26 17:05:31.756127 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:31.756103    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ckr67\" (UniqueName: \"kubernetes.io/projected/ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63-kube-api-access-ckr67\") pod \"ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63\" (UID: \"ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63\") "
May 26 17:05:31.756127 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:31.756131    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63-kserve-provision-location\") pod \"ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63\" (UID: \"ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63\") "
May 26 17:05:31.756419 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:31.756289    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\") pod \"ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63\" (UID: \"ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63\") "
May 26 17:05:31.756419 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:31.756341    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63-proxy-tls\") pod \"ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63\" (UID: \"ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63\") "
May 26 17:05:31.756533 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:31.756432    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63" (UID: "ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 17:05:31.756634 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:31.756484    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63" (UID: "ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 17:05:31.756708 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:31.756654    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config") pod "ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63" (UID: "ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63"). InnerVolumeSpecName "isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 17:05:31.756708 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:31.756693    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63-kserve-provision-location\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 17:05:31.756799 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:31.756715    2572 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63-cabundle-cert\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 17:05:31.758283 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:31.758259    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63-kube-api-access-ckr67" (OuterVolumeSpecName: "kube-api-access-ckr67") pod "ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63" (UID: "ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63"). InnerVolumeSpecName "kube-api-access-ckr67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 17:05:31.758402 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:31.758299    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63" (UID: "ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 17:05:31.857918 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:31.857881    2572 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 17:05:31.857918 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:31.857909    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63-proxy-tls\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 17:05:31.857918 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:31.857920    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckr67\" (UniqueName: \"kubernetes.io/projected/ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63-kube-api-access-ckr67\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 17:05:32.259720 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:32.259687    2572 generic.go:358] "Generic (PLEG): container finished" podID="ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63" containerID="19dd4dd5f247ea6e9c0f4fb398ea98a916d18f1995e7c969b5b4dba2545ed4ff" exitCode=0
May 26 17:05:32.259873 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:32.259741    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm" event={"ID":"ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63","Type":"ContainerDied","Data":"19dd4dd5f247ea6e9c0f4fb398ea98a916d18f1995e7c969b5b4dba2545ed4ff"}
May 26 17:05:32.259873 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:32.259767    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm"
May 26 17:05:32.259873 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:32.259779    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm" event={"ID":"ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63","Type":"ContainerDied","Data":"ef3a217f3f7a267aa43a43da0f54aff1117f68900abc413ce08e41bb4890fed1"}
May 26 17:05:32.259873 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:32.259796    2572 scope.go:117] "RemoveContainer" containerID="8830a4b54fe65cbab3b2d09d0887ab11ce4fb1ce80225ac4ad04e24f21ae8532"
May 26 17:05:32.267515 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:32.267494    2572 scope.go:117] "RemoveContainer" containerID="19dd4dd5f247ea6e9c0f4fb398ea98a916d18f1995e7c969b5b4dba2545ed4ff"
May 26 17:05:32.274370 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:32.274352    2572 scope.go:117] "RemoveContainer" containerID="7e0d246a6ab494cbe956507777c62a9c740968a1ea347e7a0e3adffc29cf5861"
May 26 17:05:32.277532 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:32.277508    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm"]
May 26 17:05:32.280794 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:32.280771    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-648d97f77b-2kvfm"]
May 26 17:05:32.282505 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:32.282491    2572 scope.go:117] "RemoveContainer" containerID="8830a4b54fe65cbab3b2d09d0887ab11ce4fb1ce80225ac4ad04e24f21ae8532"
May 26 17:05:32.282763 ip-10-0-130-83 kubenswrapper[2572]: E0526 17:05:32.282742    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8830a4b54fe65cbab3b2d09d0887ab11ce4fb1ce80225ac4ad04e24f21ae8532\": container with ID starting with 8830a4b54fe65cbab3b2d09d0887ab11ce4fb1ce80225ac4ad04e24f21ae8532 not found: ID does not exist" containerID="8830a4b54fe65cbab3b2d09d0887ab11ce4fb1ce80225ac4ad04e24f21ae8532"
May 26 17:05:32.282810 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:32.282771    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8830a4b54fe65cbab3b2d09d0887ab11ce4fb1ce80225ac4ad04e24f21ae8532"} err="failed to get container status \"8830a4b54fe65cbab3b2d09d0887ab11ce4fb1ce80225ac4ad04e24f21ae8532\": rpc error: code = NotFound desc = could not find container \"8830a4b54fe65cbab3b2d09d0887ab11ce4fb1ce80225ac4ad04e24f21ae8532\": container with ID starting with 8830a4b54fe65cbab3b2d09d0887ab11ce4fb1ce80225ac4ad04e24f21ae8532 not found: ID does not exist"
May 26 17:05:32.282810 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:32.282790    2572 scope.go:117] "RemoveContainer" containerID="19dd4dd5f247ea6e9c0f4fb398ea98a916d18f1995e7c969b5b4dba2545ed4ff"
May 26 17:05:32.283026 ip-10-0-130-83 kubenswrapper[2572]: E0526 17:05:32.283010    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19dd4dd5f247ea6e9c0f4fb398ea98a916d18f1995e7c969b5b4dba2545ed4ff\": container with ID starting with 19dd4dd5f247ea6e9c0f4fb398ea98a916d18f1995e7c969b5b4dba2545ed4ff not found: ID does not exist" containerID="19dd4dd5f247ea6e9c0f4fb398ea98a916d18f1995e7c969b5b4dba2545ed4ff"
May 26 17:05:32.283071 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:32.283031    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19dd4dd5f247ea6e9c0f4fb398ea98a916d18f1995e7c969b5b4dba2545ed4ff"} err="failed to get container status \"19dd4dd5f247ea6e9c0f4fb398ea98a916d18f1995e7c969b5b4dba2545ed4ff\": rpc error: code = NotFound desc = could not find container \"19dd4dd5f247ea6e9c0f4fb398ea98a916d18f1995e7c969b5b4dba2545ed4ff\": container with ID starting with 19dd4dd5f247ea6e9c0f4fb398ea98a916d18f1995e7c969b5b4dba2545ed4ff not found: ID does not exist"
May 26 17:05:32.283071 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:32.283043    2572 scope.go:117] "RemoveContainer" containerID="7e0d246a6ab494cbe956507777c62a9c740968a1ea347e7a0e3adffc29cf5861"
May 26 17:05:32.283301 ip-10-0-130-83 kubenswrapper[2572]: E0526 17:05:32.283284    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e0d246a6ab494cbe956507777c62a9c740968a1ea347e7a0e3adffc29cf5861\": container with ID starting with 7e0d246a6ab494cbe956507777c62a9c740968a1ea347e7a0e3adffc29cf5861 not found: ID does not exist" containerID="7e0d246a6ab494cbe956507777c62a9c740968a1ea347e7a0e3adffc29cf5861"
May 26 17:05:32.283354 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:32.283309    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e0d246a6ab494cbe956507777c62a9c740968a1ea347e7a0e3adffc29cf5861"} err="failed to get container status \"7e0d246a6ab494cbe956507777c62a9c740968a1ea347e7a0e3adffc29cf5861\": rpc error: code = NotFound desc = could not find container \"7e0d246a6ab494cbe956507777c62a9c740968a1ea347e7a0e3adffc29cf5861\": container with ID starting with 7e0d246a6ab494cbe956507777c62a9c740968a1ea347e7a0e3adffc29cf5861 not found: ID does not exist"
May 26 17:05:34.079157 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:34.079126    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63" path="/var/lib/kubelet/pods/ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63/volumes"
May 26 17:05:34.267261 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:34.267232    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db_850d0ee0-b3bf-4a8f-853b-a88807a481a7/storage-initializer/0.log"
May 26 17:05:34.267428 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:34.267271    2572 generic.go:358] "Generic (PLEG): container finished" podID="850d0ee0-b3bf-4a8f-853b-a88807a481a7" containerID="fabd7b27106a63f2e8ddba1fd77171f22d7bc83907a6901b557ad41c17d0a159" exitCode=1
May 26 17:05:34.267428 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:34.267351    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db" event={"ID":"850d0ee0-b3bf-4a8f-853b-a88807a481a7","Type":"ContainerDied","Data":"fabd7b27106a63f2e8ddba1fd77171f22d7bc83907a6901b557ad41c17d0a159"}
May 26 17:05:35.271482 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:35.271453    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db_850d0ee0-b3bf-4a8f-853b-a88807a481a7/storage-initializer/0.log"
May 26 17:05:35.271887 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:35.271553    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db" event={"ID":"850d0ee0-b3bf-4a8f-853b-a88807a481a7","Type":"ContainerStarted","Data":"6eaba5c849ecc8da53936667cc380401dace749a767595b55e9be7219733ffda"}
May 26 17:05:37.277793 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:37.277716    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db_850d0ee0-b3bf-4a8f-853b-a88807a481a7/storage-initializer/1.log"
May 26 17:05:37.278155 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:37.278046    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db_850d0ee0-b3bf-4a8f-853b-a88807a481a7/storage-initializer/0.log"
May 26 17:05:37.278155 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:37.278076    2572 generic.go:358] "Generic (PLEG): container finished" podID="850d0ee0-b3bf-4a8f-853b-a88807a481a7" containerID="6eaba5c849ecc8da53936667cc380401dace749a767595b55e9be7219733ffda" exitCode=1
May 26 17:05:37.278155 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:37.278132    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db" event={"ID":"850d0ee0-b3bf-4a8f-853b-a88807a481a7","Type":"ContainerDied","Data":"6eaba5c849ecc8da53936667cc380401dace749a767595b55e9be7219733ffda"}
May 26 17:05:37.278274 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:37.278163    2572 scope.go:117] "RemoveContainer" containerID="fabd7b27106a63f2e8ddba1fd77171f22d7bc83907a6901b557ad41c17d0a159"
May 26 17:05:37.278511 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:37.278493    2572 scope.go:117] "RemoveContainer" containerID="fabd7b27106a63f2e8ddba1fd77171f22d7bc83907a6901b557ad41c17d0a159"
May 26 17:05:37.287408 ip-10-0-130-83 kubenswrapper[2572]: E0526 17:05:37.287380    2572 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db_kserve-ci-e2e-test_850d0ee0-b3bf-4a8f-853b-a88807a481a7_0 in pod sandbox 72ec4b343d56d8fcdfbcac67b27a6f29b5df37d656269eaac58c2104c245e329 from index: no such id: 'fabd7b27106a63f2e8ddba1fd77171f22d7bc83907a6901b557ad41c17d0a159'" containerID="fabd7b27106a63f2e8ddba1fd77171f22d7bc83907a6901b557ad41c17d0a159"
May 26 17:05:37.287484 ip-10-0-130-83 kubenswrapper[2572]: E0526 17:05:37.287424    2572 kuberuntime_container.go:951] "Unhandled Error" err="failed to remove pod init container \"storage-initializer\": rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db_kserve-ci-e2e-test_850d0ee0-b3bf-4a8f-853b-a88807a481a7_0 in pod sandbox 72ec4b343d56d8fcdfbcac67b27a6f29b5df37d656269eaac58c2104c245e329 from index: no such id: 'fabd7b27106a63f2e8ddba1fd77171f22d7bc83907a6901b557ad41c17d0a159'; Skipping pod \"isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db_kserve-ci-e2e-test(850d0ee0-b3bf-4a8f-853b-a88807a481a7)\"" logger="UnhandledError"
May 26 17:05:37.288764 ip-10-0-130-83 kubenswrapper[2572]: E0526 17:05:37.288744    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db_kserve-ci-e2e-test(850d0ee0-b3bf-4a8f-853b-a88807a481a7)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db" podUID="850d0ee0-b3bf-4a8f-853b-a88807a481a7"
May 26 17:05:38.155533 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:38.155493    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db"]
May 26 17:05:38.282408 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:38.282377    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db_850d0ee0-b3bf-4a8f-853b-a88807a481a7/storage-initializer/1.log"
May 26 17:05:38.408298 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:38.408242    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db_850d0ee0-b3bf-4a8f-853b-a88807a481a7/storage-initializer/1.log"
May 26 17:05:38.408412 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:38.408303    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db"
May 26 17:05:38.509939 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:38.509901    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nmjfz\" (UniqueName: \"kubernetes.io/projected/850d0ee0-b3bf-4a8f-853b-a88807a481a7-kube-api-access-nmjfz\") pod \"850d0ee0-b3bf-4a8f-853b-a88807a481a7\" (UID: \"850d0ee0-b3bf-4a8f-853b-a88807a481a7\") "
May 26 17:05:38.510208 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:38.509980    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/850d0ee0-b3bf-4a8f-853b-a88807a481a7-kserve-provision-location\") pod \"850d0ee0-b3bf-4a8f-853b-a88807a481a7\" (UID: \"850d0ee0-b3bf-4a8f-853b-a88807a481a7\") "
May 26 17:05:38.510208 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:38.510010    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/850d0ee0-b3bf-4a8f-853b-a88807a481a7-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\") pod \"850d0ee0-b3bf-4a8f-853b-a88807a481a7\" (UID: \"850d0ee0-b3bf-4a8f-853b-a88807a481a7\") "
May 26 17:05:38.510208 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:38.510046    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/850d0ee0-b3bf-4a8f-853b-a88807a481a7-proxy-tls\") pod \"850d0ee0-b3bf-4a8f-853b-a88807a481a7\" (UID: \"850d0ee0-b3bf-4a8f-853b-a88807a481a7\") "
May 26 17:05:38.510377 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:38.510297    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/850d0ee0-b3bf-4a8f-853b-a88807a481a7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "850d0ee0-b3bf-4a8f-853b-a88807a481a7" (UID: "850d0ee0-b3bf-4a8f-853b-a88807a481a7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 17:05:38.510416 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:38.510375    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/850d0ee0-b3bf-4a8f-853b-a88807a481a7-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config") pod "850d0ee0-b3bf-4a8f-853b-a88807a481a7" (UID: "850d0ee0-b3bf-4a8f-853b-a88807a481a7"). InnerVolumeSpecName "isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 17:05:38.512039 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:38.512016    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/850d0ee0-b3bf-4a8f-853b-a88807a481a7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "850d0ee0-b3bf-4a8f-853b-a88807a481a7" (UID: "850d0ee0-b3bf-4a8f-853b-a88807a481a7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 17:05:38.512123 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:38.512064    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/850d0ee0-b3bf-4a8f-853b-a88807a481a7-kube-api-access-nmjfz" (OuterVolumeSpecName: "kube-api-access-nmjfz") pod "850d0ee0-b3bf-4a8f-853b-a88807a481a7" (UID: "850d0ee0-b3bf-4a8f-853b-a88807a481a7"). InnerVolumeSpecName "kube-api-access-nmjfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 17:05:38.611410 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:38.611371    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/850d0ee0-b3bf-4a8f-853b-a88807a481a7-kserve-provision-location\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 17:05:38.611410 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:38.611403    2572 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/850d0ee0-b3bf-4a8f-853b-a88807a481a7-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 17:05:38.611410 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:38.611414    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/850d0ee0-b3bf-4a8f-853b-a88807a481a7-proxy-tls\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 17:05:38.611692 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:38.611424    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmjfz\" (UniqueName: \"kubernetes.io/projected/850d0ee0-b3bf-4a8f-853b-a88807a481a7-kube-api-access-nmjfz\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 17:05:39.253063 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:39.253025    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq"]
May 26 17:05:39.253303 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:39.253291    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63" containerName="kube-rbac-proxy"
May 26 17:05:39.253350 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:39.253305    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63" containerName="kube-rbac-proxy"
May 26 17:05:39.253350 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:39.253314    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="850d0ee0-b3bf-4a8f-853b-a88807a481a7" containerName="storage-initializer"
May 26 17:05:39.253350 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:39.253320    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="850d0ee0-b3bf-4a8f-853b-a88807a481a7" containerName="storage-initializer"
May 26 17:05:39.253350 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:39.253333    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="850d0ee0-b3bf-4a8f-853b-a88807a481a7" containerName="storage-initializer"
May 26 17:05:39.253350 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:39.253342    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="850d0ee0-b3bf-4a8f-853b-a88807a481a7" containerName="storage-initializer"
May 26 17:05:39.253350 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:39.253350    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63" containerName="kserve-container"
May 26 17:05:39.253662 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:39.253356    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63" containerName="kserve-container"
May 26 17:05:39.253662 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:39.253367    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63" containerName="storage-initializer"
May 26 17:05:39.253662 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:39.253373    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63" containerName="storage-initializer"
May 26 17:05:39.253662 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:39.253422    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63" containerName="kserve-container"
May 26 17:05:39.253662 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:39.253431    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="850d0ee0-b3bf-4a8f-853b-a88807a481a7" containerName="storage-initializer"
May 26 17:05:39.253662 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:39.253437    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="ce6fc6d0-e5a5-4880-b2f7-b24f5fe7bb63" containerName="kube-rbac-proxy"
May 26 17:05:39.253662 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:39.253442    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="850d0ee0-b3bf-4a8f-853b-a88807a481a7" containerName="storage-initializer"
May 26 17:05:39.255775 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:39.255756    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq"
May 26 17:05:39.258143 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:39.258119    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
May 26 17:05:39.258267 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:39.258213    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
May 26 17:05:39.258349 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:39.258339    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
May 26 17:05:39.269907 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:39.269880    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq"]
May 26 17:05:39.286477 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:39.286445    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db_850d0ee0-b3bf-4a8f-853b-a88807a481a7/storage-initializer/1.log"
May 26 17:05:39.286918 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:39.286566    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db" event={"ID":"850d0ee0-b3bf-4a8f-853b-a88807a481a7","Type":"ContainerDied","Data":"72ec4b343d56d8fcdfbcac67b27a6f29b5df37d656269eaac58c2104c245e329"}
May 26 17:05:39.286918 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:39.286622    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db"
May 26 17:05:39.286918 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:39.286632    2572 scope.go:117] "RemoveContainer" containerID="6eaba5c849ecc8da53936667cc380401dace749a767595b55e9be7219733ffda"
May 26 17:05:39.316645 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:39.316589    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eea6a3b4-1e1d-49d2-91e5-4e57a23457ef-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq\" (UID: \"eea6a3b4-1e1d-49d2-91e5-4e57a23457ef\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq"
May 26 17:05:39.316819 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:39.316700    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eea6a3b4-1e1d-49d2-91e5-4e57a23457ef-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq\" (UID: \"eea6a3b4-1e1d-49d2-91e5-4e57a23457ef\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq"
May 26 17:05:39.316819 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:39.316774    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klw2l\" (UniqueName: \"kubernetes.io/projected/eea6a3b4-1e1d-49d2-91e5-4e57a23457ef-kube-api-access-klw2l\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq\" (UID: \"eea6a3b4-1e1d-49d2-91e5-4e57a23457ef\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq"
May 26 17:05:39.316940 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:39.316861    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eea6a3b4-1e1d-49d2-91e5-4e57a23457ef-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq\" (UID: \"eea6a3b4-1e1d-49d2-91e5-4e57a23457ef\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq"
May 26 17:05:39.316940 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:39.316910    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eea6a3b4-1e1d-49d2-91e5-4e57a23457ef-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq\" (UID: \"eea6a3b4-1e1d-49d2-91e5-4e57a23457ef\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq"
May 26 17:05:39.321935 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:39.321912    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db"]
May 26 17:05:39.325904 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:39.325879    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7fdcf885c4-xz6db"]
May 26 17:05:39.417900 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:39.417861    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eea6a3b4-1e1d-49d2-91e5-4e57a23457ef-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq\" (UID: \"eea6a3b4-1e1d-49d2-91e5-4e57a23457ef\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq"
May 26 17:05:39.418104 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:39.417915    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-klw2l\" (UniqueName: \"kubernetes.io/projected/eea6a3b4-1e1d-49d2-91e5-4e57a23457ef-kube-api-access-klw2l\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq\" (UID: \"eea6a3b4-1e1d-49d2-91e5-4e57a23457ef\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq"
May 26 17:05:39.418104 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:39.417941    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eea6a3b4-1e1d-49d2-91e5-4e57a23457ef-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq\" (UID: \"eea6a3b4-1e1d-49d2-91e5-4e57a23457ef\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq"
May 26 17:05:39.418104 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:39.417958    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eea6a3b4-1e1d-49d2-91e5-4e57a23457ef-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq\" (UID: \"eea6a3b4-1e1d-49d2-91e5-4e57a23457ef\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq"
May 26 17:05:39.418104 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:39.417993    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eea6a3b4-1e1d-49d2-91e5-4e57a23457ef-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq\" (UID: \"eea6a3b4-1e1d-49d2-91e5-4e57a23457ef\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq"
May 26 17:05:39.418446 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:39.418423    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eea6a3b4-1e1d-49d2-91e5-4e57a23457ef-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq\" (UID: \"eea6a3b4-1e1d-49d2-91e5-4e57a23457ef\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq"
May 26 17:05:39.418573 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:39.418555    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eea6a3b4-1e1d-49d2-91e5-4e57a23457ef-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq\" (UID: \"eea6a3b4-1e1d-49d2-91e5-4e57a23457ef\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq"
May 26 17:05:39.418649 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:39.418564    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eea6a3b4-1e1d-49d2-91e5-4e57a23457ef-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq\" (UID: \"eea6a3b4-1e1d-49d2-91e5-4e57a23457ef\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq"
May 26 17:05:39.420433 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:39.420407    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eea6a3b4-1e1d-49d2-91e5-4e57a23457ef-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq\" (UID: \"eea6a3b4-1e1d-49d2-91e5-4e57a23457ef\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq"
May 26 17:05:39.426976 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:39.426956    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klw2l\" (UniqueName: \"kubernetes.io/projected/eea6a3b4-1e1d-49d2-91e5-4e57a23457ef-kube-api-access-klw2l\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq\" (UID: \"eea6a3b4-1e1d-49d2-91e5-4e57a23457ef\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq"
May 26 17:05:39.565978 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:39.565891    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq"
May 26 17:05:39.693060 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:39.693025    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq"]
May 26 17:05:39.695825 ip-10-0-130-83 kubenswrapper[2572]: W0526 17:05:39.695789    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeea6a3b4_1e1d_49d2_91e5_4e57a23457ef.slice/crio-9f7d2b05d4f4684c5f1351dba5af0746515263300a2f28bc3f7ffd75e96b1294 WatchSource:0}: Error finding container 9f7d2b05d4f4684c5f1351dba5af0746515263300a2f28bc3f7ffd75e96b1294: Status 404 returned error can't find the container with id 9f7d2b05d4f4684c5f1351dba5af0746515263300a2f28bc3f7ffd75e96b1294
May 26 17:05:40.079129 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:40.079090    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="850d0ee0-b3bf-4a8f-853b-a88807a481a7" path="/var/lib/kubelet/pods/850d0ee0-b3bf-4a8f-853b-a88807a481a7/volumes"
May 26 17:05:40.297399 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:40.297357    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq" event={"ID":"eea6a3b4-1e1d-49d2-91e5-4e57a23457ef","Type":"ContainerStarted","Data":"40d884d3f66bfa290c6258bda263061ba5b606ae1cf2687bc7c5273511a344bd"}
May 26 17:05:40.297399 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:40.297401    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq" event={"ID":"eea6a3b4-1e1d-49d2-91e5-4e57a23457ef","Type":"ContainerStarted","Data":"9f7d2b05d4f4684c5f1351dba5af0746515263300a2f28bc3f7ffd75e96b1294"}
May 26 17:05:41.301556 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:41.301512    2572 generic.go:358] "Generic (PLEG): container finished" podID="eea6a3b4-1e1d-49d2-91e5-4e57a23457ef" containerID="40d884d3f66bfa290c6258bda263061ba5b606ae1cf2687bc7c5273511a344bd" exitCode=0
May 26 17:05:41.302038 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:41.301643    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq" event={"ID":"eea6a3b4-1e1d-49d2-91e5-4e57a23457ef","Type":"ContainerDied","Data":"40d884d3f66bfa290c6258bda263061ba5b606ae1cf2687bc7c5273511a344bd"}
May 26 17:05:42.306319 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:42.306281    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq" event={"ID":"eea6a3b4-1e1d-49d2-91e5-4e57a23457ef","Type":"ContainerStarted","Data":"ff5ebfc5d4359ddb15ae48009791bf080f40b7a4a27358ef613cd20e03800e72"}
May 26 17:05:42.306725 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:42.306326    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq" event={"ID":"eea6a3b4-1e1d-49d2-91e5-4e57a23457ef","Type":"ContainerStarted","Data":"f3d48950f267b997dca9614be0a379c9bf0043d7de6866e3dc846c611fca163b"}
May 26 17:05:42.306725 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:42.306430    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq"
May 26 17:05:42.326174 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:42.326123    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq" podStartSLOduration=3.326107194 podStartE2EDuration="3.326107194s" podCreationTimestamp="2026-05-26 17:05:39 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 17:05:42.32434151 +0000 UTC m=+3858.800068860" watchObservedRunningTime="2026-05-26 17:05:42.326107194 +0000 UTC m=+3858.801834912"
May 26 17:05:43.308758 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:43.308721    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq"
May 26 17:05:43.310094 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:43.310070    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq" podUID="eea6a3b4-1e1d-49d2-91e5-4e57a23457ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
May 26 17:05:44.311940 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:44.311899    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq" podUID="eea6a3b4-1e1d-49d2-91e5-4e57a23457ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
May 26 17:05:49.316201 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:49.316169    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq"
May 26 17:05:49.316844 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:49.316812    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq" podUID="eea6a3b4-1e1d-49d2-91e5-4e57a23457ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
May 26 17:05:59.317521 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:05:59.317485    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq" podUID="eea6a3b4-1e1d-49d2-91e5-4e57a23457ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
May 26 17:06:09.317224 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:06:09.317183    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq" podUID="eea6a3b4-1e1d-49d2-91e5-4e57a23457ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
May 26 17:06:19.316677 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:06:19.316638    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq" podUID="eea6a3b4-1e1d-49d2-91e5-4e57a23457ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
May 26 17:06:29.316690 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:06:29.316652    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq" podUID="eea6a3b4-1e1d-49d2-91e5-4e57a23457ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
May 26 17:06:39.317121 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:06:39.317071    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq" podUID="eea6a3b4-1e1d-49d2-91e5-4e57a23457ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
May 26 17:06:49.317305 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:06:49.317275    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq"
May 26 17:06:59.265758 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:06:59.265720    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq"]
May 26 17:06:59.266211 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:06:59.266091    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq" podUID="eea6a3b4-1e1d-49d2-91e5-4e57a23457ef" containerName="kserve-container" containerID="cri-o://f3d48950f267b997dca9614be0a379c9bf0043d7de6866e3dc846c611fca163b" gracePeriod=30
May 26 17:06:59.266352 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:06:59.266206    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq" podUID="eea6a3b4-1e1d-49d2-91e5-4e57a23457ef" containerName="kube-rbac-proxy" containerID="cri-o://ff5ebfc5d4359ddb15ae48009791bf080f40b7a4a27358ef613cd20e03800e72" gracePeriod=30
May 26 17:06:59.313056 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:06:59.313015    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq" podUID="eea6a3b4-1e1d-49d2-91e5-4e57a23457ef" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.64:8643/healthz\": dial tcp 10.132.0.64:8643: connect: connection refused"
May 26 17:06:59.519295 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:06:59.519205    2572 generic.go:358] "Generic (PLEG): container finished" podID="eea6a3b4-1e1d-49d2-91e5-4e57a23457ef" containerID="ff5ebfc5d4359ddb15ae48009791bf080f40b7a4a27358ef613cd20e03800e72" exitCode=2
May 26 17:06:59.519295 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:06:59.519251    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq" event={"ID":"eea6a3b4-1e1d-49d2-91e5-4e57a23457ef","Type":"ContainerDied","Data":"ff5ebfc5d4359ddb15ae48009791bf080f40b7a4a27358ef613cd20e03800e72"}
May 26 17:07:00.348935 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:00.348896    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-4d284"]
May 26 17:07:00.351308 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:00.351286    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-4d284"
May 26 17:07:00.353427 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:00.353397    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
May 26 17:07:00.353619 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:00.353423    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
May 26 17:07:00.362910 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:00.362884    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-4d284"]
May 26 17:07:00.433728 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:00.433688    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c857f87-c8a8-4572-aaf1-5caa120ba67a-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-4d284\" (UID: \"3c857f87-c8a8-4572-aaf1-5caa120ba67a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-4d284"
May 26 17:07:00.433925 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:00.433751    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h75cv\" (UniqueName: \"kubernetes.io/projected/3c857f87-c8a8-4572-aaf1-5caa120ba67a-kube-api-access-h75cv\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-4d284\" (UID: \"3c857f87-c8a8-4572-aaf1-5caa120ba67a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-4d284"
May 26 17:07:00.433925 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:00.433860    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c857f87-c8a8-4572-aaf1-5caa120ba67a-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-4d284\" (UID: \"3c857f87-c8a8-4572-aaf1-5caa120ba67a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-4d284"
May 26 17:07:00.433925 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:00.433907    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c857f87-c8a8-4572-aaf1-5caa120ba67a-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-4d284\" (UID: \"3c857f87-c8a8-4572-aaf1-5caa120ba67a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-4d284"
May 26 17:07:00.534861 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:00.534825    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-h75cv\" (UniqueName: \"kubernetes.io/projected/3c857f87-c8a8-4572-aaf1-5caa120ba67a-kube-api-access-h75cv\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-4d284\" (UID: \"3c857f87-c8a8-4572-aaf1-5caa120ba67a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-4d284"
May 26 17:07:00.535045 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:00.534881    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c857f87-c8a8-4572-aaf1-5caa120ba67a-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-4d284\" (UID: \"3c857f87-c8a8-4572-aaf1-5caa120ba67a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-4d284"
May 26 17:07:00.535045 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:00.534911    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c857f87-c8a8-4572-aaf1-5caa120ba67a-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-4d284\" (UID: \"3c857f87-c8a8-4572-aaf1-5caa120ba67a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-4d284"
May 26 17:07:00.535045 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:00.534975    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c857f87-c8a8-4572-aaf1-5caa120ba67a-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-4d284\" (UID: \"3c857f87-c8a8-4572-aaf1-5caa120ba67a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-4d284"
May 26 17:07:00.535435 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:00.535412    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c857f87-c8a8-4572-aaf1-5caa120ba67a-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-4d284\" (UID: \"3c857f87-c8a8-4572-aaf1-5caa120ba67a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-4d284"
May 26 17:07:00.535744 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:00.535716    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c857f87-c8a8-4572-aaf1-5caa120ba67a-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-4d284\" (UID: \"3c857f87-c8a8-4572-aaf1-5caa120ba67a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-4d284"
May 26 17:07:00.537481 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:00.537463    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c857f87-c8a8-4572-aaf1-5caa120ba67a-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-4d284\" (UID: \"3c857f87-c8a8-4572-aaf1-5caa120ba67a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-4d284"
May 26 17:07:00.543075 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:00.543052    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h75cv\" (UniqueName: \"kubernetes.io/projected/3c857f87-c8a8-4572-aaf1-5caa120ba67a-kube-api-access-h75cv\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-4d284\" (UID: \"3c857f87-c8a8-4572-aaf1-5caa120ba67a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-4d284"
May 26 17:07:00.662660 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:00.662597    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-4d284"
May 26 17:07:00.792753 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:00.792716    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-4d284"]
May 26 17:07:00.796597 ip-10-0-130-83 kubenswrapper[2572]: W0526 17:07:00.796564    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c857f87_c8a8_4572_aaf1_5caa120ba67a.slice/crio-afd63bd794f2597b9d69144df353cb473199e35a54415e640a968c9b8e5a2aad WatchSource:0}: Error finding container afd63bd794f2597b9d69144df353cb473199e35a54415e640a968c9b8e5a2aad: Status 404 returned error can't find the container with id afd63bd794f2597b9d69144df353cb473199e35a54415e640a968c9b8e5a2aad
May 26 17:07:00.798579 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:00.798563    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 17:07:01.526368 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:01.526327    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-4d284" event={"ID":"3c857f87-c8a8-4572-aaf1-5caa120ba67a","Type":"ContainerStarted","Data":"071362843f28aab964ab233e73614eced1afb68b00fc417503b8290c566f4007"}
May 26 17:07:01.526368 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:01.526367    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-4d284" event={"ID":"3c857f87-c8a8-4572-aaf1-5caa120ba67a","Type":"ContainerStarted","Data":"afd63bd794f2597b9d69144df353cb473199e35a54415e640a968c9b8e5a2aad"}
May 26 17:07:03.908415 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:03.908388    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq"
May 26 17:07:03.962314 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:03.962277    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eea6a3b4-1e1d-49d2-91e5-4e57a23457ef-kserve-provision-location\") pod \"eea6a3b4-1e1d-49d2-91e5-4e57a23457ef\" (UID: \"eea6a3b4-1e1d-49d2-91e5-4e57a23457ef\") "
May 26 17:07:03.962513 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:03.962327    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eea6a3b4-1e1d-49d2-91e5-4e57a23457ef-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\") pod \"eea6a3b4-1e1d-49d2-91e5-4e57a23457ef\" (UID: \"eea6a3b4-1e1d-49d2-91e5-4e57a23457ef\") "
May 26 17:07:03.962513 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:03.962369    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-klw2l\" (UniqueName: \"kubernetes.io/projected/eea6a3b4-1e1d-49d2-91e5-4e57a23457ef-kube-api-access-klw2l\") pod \"eea6a3b4-1e1d-49d2-91e5-4e57a23457ef\" (UID: \"eea6a3b4-1e1d-49d2-91e5-4e57a23457ef\") "
May 26 17:07:03.962671 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:03.962547    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eea6a3b4-1e1d-49d2-91e5-4e57a23457ef-proxy-tls\") pod \"eea6a3b4-1e1d-49d2-91e5-4e57a23457ef\" (UID: \"eea6a3b4-1e1d-49d2-91e5-4e57a23457ef\") "
May 26 17:07:03.962671 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:03.962588    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eea6a3b4-1e1d-49d2-91e5-4e57a23457ef-cabundle-cert\") pod \"eea6a3b4-1e1d-49d2-91e5-4e57a23457ef\" (UID: \"eea6a3b4-1e1d-49d2-91e5-4e57a23457ef\") "
May 26 17:07:03.962789 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:03.962724    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eea6a3b4-1e1d-49d2-91e5-4e57a23457ef-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "eea6a3b4-1e1d-49d2-91e5-4e57a23457ef" (UID: "eea6a3b4-1e1d-49d2-91e5-4e57a23457ef"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 17:07:03.962895 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:03.962870    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eea6a3b4-1e1d-49d2-91e5-4e57a23457ef-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config") pod "eea6a3b4-1e1d-49d2-91e5-4e57a23457ef" (UID: "eea6a3b4-1e1d-49d2-91e5-4e57a23457ef"). InnerVolumeSpecName "isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 17:07:03.962955 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:03.962935    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eea6a3b4-1e1d-49d2-91e5-4e57a23457ef-kserve-provision-location\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 17:07:03.963009 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:03.962947    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eea6a3b4-1e1d-49d2-91e5-4e57a23457ef-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "eea6a3b4-1e1d-49d2-91e5-4e57a23457ef" (UID: "eea6a3b4-1e1d-49d2-91e5-4e57a23457ef"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 17:07:03.964581 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:03.964558    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eea6a3b4-1e1d-49d2-91e5-4e57a23457ef-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "eea6a3b4-1e1d-49d2-91e5-4e57a23457ef" (UID: "eea6a3b4-1e1d-49d2-91e5-4e57a23457ef"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 17:07:03.964764 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:03.964745    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eea6a3b4-1e1d-49d2-91e5-4e57a23457ef-kube-api-access-klw2l" (OuterVolumeSpecName: "kube-api-access-klw2l") pod "eea6a3b4-1e1d-49d2-91e5-4e57a23457ef" (UID: "eea6a3b4-1e1d-49d2-91e5-4e57a23457ef"). InnerVolumeSpecName "kube-api-access-klw2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 17:07:04.063672 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:04.063546    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eea6a3b4-1e1d-49d2-91e5-4e57a23457ef-proxy-tls\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 17:07:04.063672 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:04.063580    2572 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eea6a3b4-1e1d-49d2-91e5-4e57a23457ef-cabundle-cert\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 17:07:04.063672 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:04.063590    2572 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eea6a3b4-1e1d-49d2-91e5-4e57a23457ef-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 17:07:04.063672 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:04.063621    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klw2l\" (UniqueName: \"kubernetes.io/projected/eea6a3b4-1e1d-49d2-91e5-4e57a23457ef-kube-api-access-klw2l\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 17:07:04.536716 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:04.536680    2572 generic.go:358] "Generic (PLEG): container finished" podID="eea6a3b4-1e1d-49d2-91e5-4e57a23457ef" containerID="f3d48950f267b997dca9614be0a379c9bf0043d7de6866e3dc846c611fca163b" exitCode=0
May 26 17:07:04.536892 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:04.536757    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq"
May 26 17:07:04.536892 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:04.536756    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq" event={"ID":"eea6a3b4-1e1d-49d2-91e5-4e57a23457ef","Type":"ContainerDied","Data":"f3d48950f267b997dca9614be0a379c9bf0043d7de6866e3dc846c611fca163b"}
May 26 17:07:04.536892 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:04.536799    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq" event={"ID":"eea6a3b4-1e1d-49d2-91e5-4e57a23457ef","Type":"ContainerDied","Data":"9f7d2b05d4f4684c5f1351dba5af0746515263300a2f28bc3f7ffd75e96b1294"}
May 26 17:07:04.536892 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:04.536815    2572 scope.go:117] "RemoveContainer" containerID="ff5ebfc5d4359ddb15ae48009791bf080f40b7a4a27358ef613cd20e03800e72"
May 26 17:07:04.544695 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:04.544675    2572 scope.go:117] "RemoveContainer" containerID="f3d48950f267b997dca9614be0a379c9bf0043d7de6866e3dc846c611fca163b"
May 26 17:07:04.551465 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:04.551439    2572 scope.go:117] "RemoveContainer" containerID="40d884d3f66bfa290c6258bda263061ba5b606ae1cf2687bc7c5273511a344bd"
May 26 17:07:04.553967 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:04.553937    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq"]
May 26 17:07:04.557534 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:04.557509    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5dfc47f4fb-lh2tq"]
May 26 17:07:04.562037 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:04.562017    2572 scope.go:117] "RemoveContainer" containerID="ff5ebfc5d4359ddb15ae48009791bf080f40b7a4a27358ef613cd20e03800e72"
May 26 17:07:04.562330 ip-10-0-130-83 kubenswrapper[2572]: E0526 17:07:04.562311    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff5ebfc5d4359ddb15ae48009791bf080f40b7a4a27358ef613cd20e03800e72\": container with ID starting with ff5ebfc5d4359ddb15ae48009791bf080f40b7a4a27358ef613cd20e03800e72 not found: ID does not exist" containerID="ff5ebfc5d4359ddb15ae48009791bf080f40b7a4a27358ef613cd20e03800e72"
May 26 17:07:04.562380 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:04.562340    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff5ebfc5d4359ddb15ae48009791bf080f40b7a4a27358ef613cd20e03800e72"} err="failed to get container status \"ff5ebfc5d4359ddb15ae48009791bf080f40b7a4a27358ef613cd20e03800e72\": rpc error: code = NotFound desc = could not find container \"ff5ebfc5d4359ddb15ae48009791bf080f40b7a4a27358ef613cd20e03800e72\": container with ID starting with ff5ebfc5d4359ddb15ae48009791bf080f40b7a4a27358ef613cd20e03800e72 not found: ID does not exist"
May 26 17:07:04.562380 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:04.562360    2572 scope.go:117] "RemoveContainer" containerID="f3d48950f267b997dca9614be0a379c9bf0043d7de6866e3dc846c611fca163b"
May 26 17:07:04.562638 ip-10-0-130-83 kubenswrapper[2572]: E0526 17:07:04.562618    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3d48950f267b997dca9614be0a379c9bf0043d7de6866e3dc846c611fca163b\": container with ID starting with f3d48950f267b997dca9614be0a379c9bf0043d7de6866e3dc846c611fca163b not found: ID does not exist" containerID="f3d48950f267b997dca9614be0a379c9bf0043d7de6866e3dc846c611fca163b"
May 26 17:07:04.562707 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:04.562647    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3d48950f267b997dca9614be0a379c9bf0043d7de6866e3dc846c611fca163b"} err="failed to get container status \"f3d48950f267b997dca9614be0a379c9bf0043d7de6866e3dc846c611fca163b\": rpc error: code = NotFound desc = could not find container \"f3d48950f267b997dca9614be0a379c9bf0043d7de6866e3dc846c611fca163b\": container with ID starting with f3d48950f267b997dca9614be0a379c9bf0043d7de6866e3dc846c611fca163b not found: ID does not exist"
May 26 17:07:04.562707 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:04.562664    2572 scope.go:117] "RemoveContainer" containerID="40d884d3f66bfa290c6258bda263061ba5b606ae1cf2687bc7c5273511a344bd"
May 26 17:07:04.562867 ip-10-0-130-83 kubenswrapper[2572]: E0526 17:07:04.562852    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40d884d3f66bfa290c6258bda263061ba5b606ae1cf2687bc7c5273511a344bd\": container with ID starting with 40d884d3f66bfa290c6258bda263061ba5b606ae1cf2687bc7c5273511a344bd not found: ID does not exist" containerID="40d884d3f66bfa290c6258bda263061ba5b606ae1cf2687bc7c5273511a344bd"
May 26 17:07:04.562900 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:04.562872    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40d884d3f66bfa290c6258bda263061ba5b606ae1cf2687bc7c5273511a344bd"} err="failed to get container status \"40d884d3f66bfa290c6258bda263061ba5b606ae1cf2687bc7c5273511a344bd\": rpc error: code = NotFound desc = could not find container \"40d884d3f66bfa290c6258bda263061ba5b606ae1cf2687bc7c5273511a344bd\": container with ID starting with 40d884d3f66bfa290c6258bda263061ba5b606ae1cf2687bc7c5273511a344bd not found: ID does not exist"
May 26 17:07:05.541151 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:05.541121    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-4d284_3c857f87-c8a8-4572-aaf1-5caa120ba67a/storage-initializer/0.log"
May 26 17:07:05.541546 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:05.541161    2572 generic.go:358] "Generic (PLEG): container finished" podID="3c857f87-c8a8-4572-aaf1-5caa120ba67a" containerID="071362843f28aab964ab233e73614eced1afb68b00fc417503b8290c566f4007" exitCode=1
May 26 17:07:05.541546 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:05.541206    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-4d284" event={"ID":"3c857f87-c8a8-4572-aaf1-5caa120ba67a","Type":"ContainerDied","Data":"071362843f28aab964ab233e73614eced1afb68b00fc417503b8290c566f4007"}
May 26 17:07:06.081179 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:06.081147    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eea6a3b4-1e1d-49d2-91e5-4e57a23457ef" path="/var/lib/kubelet/pods/eea6a3b4-1e1d-49d2-91e5-4e57a23457ef/volumes"
May 26 17:07:06.545369 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:06.545335    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-4d284_3c857f87-c8a8-4572-aaf1-5caa120ba67a/storage-initializer/0.log"
May 26 17:07:06.545875 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:06.545448    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-4d284" event={"ID":"3c857f87-c8a8-4572-aaf1-5caa120ba67a","Type":"ContainerStarted","Data":"e40e733db2bf62aca78ae059d0926065bf6ce746435b5691236e63adb606f4ce"}
May 26 17:07:10.353950 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:10.353919    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-4d284"]
May 26 17:07:10.354343 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:10.354174    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-4d284" podUID="3c857f87-c8a8-4572-aaf1-5caa120ba67a" containerName="storage-initializer" containerID="cri-o://e40e733db2bf62aca78ae059d0926065bf6ce746435b5691236e63adb606f4ce" gracePeriod=30
May 26 17:07:10.485064 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:10.485039    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-4d284_3c857f87-c8a8-4572-aaf1-5caa120ba67a/storage-initializer/1.log"
May 26 17:07:10.485400 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:10.485383    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-4d284_3c857f87-c8a8-4572-aaf1-5caa120ba67a/storage-initializer/0.log"
May 26 17:07:10.485460 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:10.485450    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-4d284"
May 26 17:07:10.557554 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:10.557528    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-4d284_3c857f87-c8a8-4572-aaf1-5caa120ba67a/storage-initializer/1.log"
May 26 17:07:10.557911 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:10.557896    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-4d284_3c857f87-c8a8-4572-aaf1-5caa120ba67a/storage-initializer/0.log"
May 26 17:07:10.557982 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:10.557932    2572 generic.go:358] "Generic (PLEG): container finished" podID="3c857f87-c8a8-4572-aaf1-5caa120ba67a" containerID="e40e733db2bf62aca78ae059d0926065bf6ce746435b5691236e63adb606f4ce" exitCode=1
May 26 17:07:10.558048 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:10.558016    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-4d284" event={"ID":"3c857f87-c8a8-4572-aaf1-5caa120ba67a","Type":"ContainerDied","Data":"e40e733db2bf62aca78ae059d0926065bf6ce746435b5691236e63adb606f4ce"}
May 26 17:07:10.558106 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:10.558041    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-4d284"
May 26 17:07:10.558106 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:10.558054    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-4d284" event={"ID":"3c857f87-c8a8-4572-aaf1-5caa120ba67a","Type":"ContainerDied","Data":"afd63bd794f2597b9d69144df353cb473199e35a54415e640a968c9b8e5a2aad"}
May 26 17:07:10.558106 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:10.558070    2572 scope.go:117] "RemoveContainer" containerID="e40e733db2bf62aca78ae059d0926065bf6ce746435b5691236e63adb606f4ce"
May 26 17:07:10.565494 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:10.565477    2572 scope.go:117] "RemoveContainer" containerID="071362843f28aab964ab233e73614eced1afb68b00fc417503b8290c566f4007"
May 26 17:07:10.572499 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:10.572471    2572 scope.go:117] "RemoveContainer" containerID="e40e733db2bf62aca78ae059d0926065bf6ce746435b5691236e63adb606f4ce"
May 26 17:07:10.572838 ip-10-0-130-83 kubenswrapper[2572]: E0526 17:07:10.572815    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e40e733db2bf62aca78ae059d0926065bf6ce746435b5691236e63adb606f4ce\": container with ID starting with e40e733db2bf62aca78ae059d0926065bf6ce746435b5691236e63adb606f4ce not found: ID does not exist" containerID="e40e733db2bf62aca78ae059d0926065bf6ce746435b5691236e63adb606f4ce"
May 26 17:07:10.572919 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:10.572844    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e40e733db2bf62aca78ae059d0926065bf6ce746435b5691236e63adb606f4ce"} err="failed to get container status \"e40e733db2bf62aca78ae059d0926065bf6ce746435b5691236e63adb606f4ce\": rpc error: code = NotFound desc = could not find container \"e40e733db2bf62aca78ae059d0926065bf6ce746435b5691236e63adb606f4ce\": container with ID starting with e40e733db2bf62aca78ae059d0926065bf6ce746435b5691236e63adb606f4ce not found: ID does not exist"
May 26 17:07:10.572919 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:10.572863    2572 scope.go:117] "RemoveContainer" containerID="071362843f28aab964ab233e73614eced1afb68b00fc417503b8290c566f4007"
May 26 17:07:10.573112 ip-10-0-130-83 kubenswrapper[2572]: E0526 17:07:10.573091    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"071362843f28aab964ab233e73614eced1afb68b00fc417503b8290c566f4007\": container with ID starting with 071362843f28aab964ab233e73614eced1afb68b00fc417503b8290c566f4007 not found: ID does not exist" containerID="071362843f28aab964ab233e73614eced1afb68b00fc417503b8290c566f4007"
May 26 17:07:10.573149 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:10.573119    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"071362843f28aab964ab233e73614eced1afb68b00fc417503b8290c566f4007"} err="failed to get container status \"071362843f28aab964ab233e73614eced1afb68b00fc417503b8290c566f4007\": rpc error: code = NotFound desc = could not find container \"071362843f28aab964ab233e73614eced1afb68b00fc417503b8290c566f4007\": container with ID starting with 071362843f28aab964ab233e73614eced1afb68b00fc417503b8290c566f4007 not found: ID does not exist"
May 26 17:07:10.617704 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:10.617674    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c857f87-c8a8-4572-aaf1-5caa120ba67a-proxy-tls\") pod \"3c857f87-c8a8-4572-aaf1-5caa120ba67a\" (UID: \"3c857f87-c8a8-4572-aaf1-5caa120ba67a\") "
May 26 17:07:10.617863 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:10.617727    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c857f87-c8a8-4572-aaf1-5caa120ba67a-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\") pod \"3c857f87-c8a8-4572-aaf1-5caa120ba67a\" (UID: \"3c857f87-c8a8-4572-aaf1-5caa120ba67a\") "
May 26 17:07:10.617863 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:10.617759    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c857f87-c8a8-4572-aaf1-5caa120ba67a-kserve-provision-location\") pod \"3c857f87-c8a8-4572-aaf1-5caa120ba67a\" (UID: \"3c857f87-c8a8-4572-aaf1-5caa120ba67a\") "
May 26 17:07:10.617863 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:10.617801    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h75cv\" (UniqueName: \"kubernetes.io/projected/3c857f87-c8a8-4572-aaf1-5caa120ba67a-kube-api-access-h75cv\") pod \"3c857f87-c8a8-4572-aaf1-5caa120ba67a\" (UID: \"3c857f87-c8a8-4572-aaf1-5caa120ba67a\") "
May 26 17:07:10.618088 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:10.618064    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c857f87-c8a8-4572-aaf1-5caa120ba67a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3c857f87-c8a8-4572-aaf1-5caa120ba67a" (UID: "3c857f87-c8a8-4572-aaf1-5caa120ba67a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 17:07:10.618141 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:10.618124    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c857f87-c8a8-4572-aaf1-5caa120ba67a-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config") pod "3c857f87-c8a8-4572-aaf1-5caa120ba67a" (UID: "3c857f87-c8a8-4572-aaf1-5caa120ba67a"). InnerVolumeSpecName "isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 17:07:10.619846 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:10.619819    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c857f87-c8a8-4572-aaf1-5caa120ba67a-kube-api-access-h75cv" (OuterVolumeSpecName: "kube-api-access-h75cv") pod "3c857f87-c8a8-4572-aaf1-5caa120ba67a" (UID: "3c857f87-c8a8-4572-aaf1-5caa120ba67a"). InnerVolumeSpecName "kube-api-access-h75cv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 17:07:10.619951 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:10.619853    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c857f87-c8a8-4572-aaf1-5caa120ba67a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3c857f87-c8a8-4572-aaf1-5caa120ba67a" (UID: "3c857f87-c8a8-4572-aaf1-5caa120ba67a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 17:07:10.719269 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:10.719233    2572 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c857f87-c8a8-4572-aaf1-5caa120ba67a-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 17:07:10.719269 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:10.719262    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c857f87-c8a8-4572-aaf1-5caa120ba67a-kserve-provision-location\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 17:07:10.719269 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:10.719273    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h75cv\" (UniqueName: \"kubernetes.io/projected/3c857f87-c8a8-4572-aaf1-5caa120ba67a-kube-api-access-h75cv\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 17:07:10.719502 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:10.719282    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c857f87-c8a8-4572-aaf1-5caa120ba67a-proxy-tls\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 17:07:10.895442 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:10.895405    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-4d284"]
May 26 17:07:10.898465 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:10.898435    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5c666c557d-4d284"]
May 26 17:07:11.423725 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:11.423687    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v"]
May 26 17:07:11.424075 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:11.423948    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eea6a3b4-1e1d-49d2-91e5-4e57a23457ef" containerName="storage-initializer"
May 26 17:07:11.424075 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:11.423959    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="eea6a3b4-1e1d-49d2-91e5-4e57a23457ef" containerName="storage-initializer"
May 26 17:07:11.424075 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:11.423969    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eea6a3b4-1e1d-49d2-91e5-4e57a23457ef" containerName="kserve-container"
May 26 17:07:11.424075 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:11.423975    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="eea6a3b4-1e1d-49d2-91e5-4e57a23457ef" containerName="kserve-container"
May 26 17:07:11.424075 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:11.423982    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c857f87-c8a8-4572-aaf1-5caa120ba67a" containerName="storage-initializer"
May 26 17:07:11.424075 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:11.423987    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c857f87-c8a8-4572-aaf1-5caa120ba67a" containerName="storage-initializer"
May 26 17:07:11.424075 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:11.423994    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c857f87-c8a8-4572-aaf1-5caa120ba67a" containerName="storage-initializer"
May 26 17:07:11.424075 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:11.423999    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c857f87-c8a8-4572-aaf1-5caa120ba67a" containerName="storage-initializer"
May 26 17:07:11.424075 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:11.424008    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eea6a3b4-1e1d-49d2-91e5-4e57a23457ef" containerName="kube-rbac-proxy"
May 26 17:07:11.424075 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:11.424013    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="eea6a3b4-1e1d-49d2-91e5-4e57a23457ef" containerName="kube-rbac-proxy"
May 26 17:07:11.424075 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:11.424054    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="3c857f87-c8a8-4572-aaf1-5caa120ba67a" containerName="storage-initializer"
May 26 17:07:11.424075 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:11.424062    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="eea6a3b4-1e1d-49d2-91e5-4e57a23457ef" containerName="kserve-container"
May 26 17:07:11.424075 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:11.424070    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="eea6a3b4-1e1d-49d2-91e5-4e57a23457ef" containerName="kube-rbac-proxy"
May 26 17:07:11.424471 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:11.424147    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="3c857f87-c8a8-4572-aaf1-5caa120ba67a" containerName="storage-initializer"
May 26 17:07:11.426855 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:11.426834    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v"
May 26 17:07:11.429135 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:11.429086    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-c9tjh\""
May 26 17:07:11.429439 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:11.429412    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
May 26 17:07:11.429539 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:11.429461    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 26 17:07:11.429539 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:11.429475    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 26 17:07:11.429734 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:11.429719    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
May 26 17:07:11.429830 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:11.429814    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
May 26 17:07:11.429976 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:11.429961    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
May 26 17:07:11.438580 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:11.438558    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v"]
May 26 17:07:11.526598 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:11.526561    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffqlw\" (UniqueName: \"kubernetes.io/projected/62617635-8da9-4fbe-8fd0-cc83a778bb85-kube-api-access-ffqlw\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v\" (UID: \"62617635-8da9-4fbe-8fd0-cc83a778bb85\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v"
May 26 17:07:11.526598 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:11.526631    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62617635-8da9-4fbe-8fd0-cc83a778bb85-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v\" (UID: \"62617635-8da9-4fbe-8fd0-cc83a778bb85\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v"
May 26 17:07:11.526874 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:11.526691    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62617635-8da9-4fbe-8fd0-cc83a778bb85-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v\" (UID: \"62617635-8da9-4fbe-8fd0-cc83a778bb85\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v"
May 26 17:07:11.526874 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:11.526750    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62617635-8da9-4fbe-8fd0-cc83a778bb85-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v\" (UID: \"62617635-8da9-4fbe-8fd0-cc83a778bb85\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v"
May 26 17:07:11.526874 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:11.526792    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62617635-8da9-4fbe-8fd0-cc83a778bb85-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v\" (UID: \"62617635-8da9-4fbe-8fd0-cc83a778bb85\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v"
May 26 17:07:11.627584 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:11.627542    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ffqlw\" (UniqueName: \"kubernetes.io/projected/62617635-8da9-4fbe-8fd0-cc83a778bb85-kube-api-access-ffqlw\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v\" (UID: \"62617635-8da9-4fbe-8fd0-cc83a778bb85\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v"
May 26 17:07:11.627584 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:11.627588    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62617635-8da9-4fbe-8fd0-cc83a778bb85-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v\" (UID: \"62617635-8da9-4fbe-8fd0-cc83a778bb85\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v"
May 26 17:07:11.627884 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:11.627649    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62617635-8da9-4fbe-8fd0-cc83a778bb85-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v\" (UID: \"62617635-8da9-4fbe-8fd0-cc83a778bb85\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v"
May 26 17:07:11.627884 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:11.627671    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62617635-8da9-4fbe-8fd0-cc83a778bb85-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v\" (UID: \"62617635-8da9-4fbe-8fd0-cc83a778bb85\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v"
May 26 17:07:11.627884 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:11.627697    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62617635-8da9-4fbe-8fd0-cc83a778bb85-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v\" (UID: \"62617635-8da9-4fbe-8fd0-cc83a778bb85\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v"
May 26 17:07:11.627884 ip-10-0-130-83 kubenswrapper[2572]: E0526 17:07:11.627800    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
May 26 17:07:11.627884 ip-10-0-130-83 kubenswrapper[2572]: E0526 17:07:11.627874    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62617635-8da9-4fbe-8fd0-cc83a778bb85-proxy-tls podName:62617635-8da9-4fbe-8fd0-cc83a778bb85 nodeName:}" failed. No retries permitted until 2026-05-26 17:07:12.127851371 +0000 UTC m=+3948.603578705 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/62617635-8da9-4fbe-8fd0-cc83a778bb85-proxy-tls") pod "isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v" (UID: "62617635-8da9-4fbe-8fd0-cc83a778bb85") : secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
May 26 17:07:11.628142 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:11.628110    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62617635-8da9-4fbe-8fd0-cc83a778bb85-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v\" (UID: \"62617635-8da9-4fbe-8fd0-cc83a778bb85\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v"
May 26 17:07:11.628371 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:11.628351    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62617635-8da9-4fbe-8fd0-cc83a778bb85-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v\" (UID: \"62617635-8da9-4fbe-8fd0-cc83a778bb85\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v"
May 26 17:07:11.628427 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:11.628409    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62617635-8da9-4fbe-8fd0-cc83a778bb85-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v\" (UID: \"62617635-8da9-4fbe-8fd0-cc83a778bb85\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v"
May 26 17:07:11.636274 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:11.636242    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffqlw\" (UniqueName: \"kubernetes.io/projected/62617635-8da9-4fbe-8fd0-cc83a778bb85-kube-api-access-ffqlw\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v\" (UID: \"62617635-8da9-4fbe-8fd0-cc83a778bb85\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v"
May 26 17:07:12.078846 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:12.078809    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c857f87-c8a8-4572-aaf1-5caa120ba67a" path="/var/lib/kubelet/pods/3c857f87-c8a8-4572-aaf1-5caa120ba67a/volumes"
May 26 17:07:12.131643 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:12.131586    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62617635-8da9-4fbe-8fd0-cc83a778bb85-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v\" (UID: \"62617635-8da9-4fbe-8fd0-cc83a778bb85\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v"
May 26 17:07:12.134136 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:12.134105    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62617635-8da9-4fbe-8fd0-cc83a778bb85-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v\" (UID: \"62617635-8da9-4fbe-8fd0-cc83a778bb85\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v"
May 26 17:07:12.337259 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:12.337165    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v"
May 26 17:07:12.465635 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:12.465587    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v"]
May 26 17:07:12.468748 ip-10-0-130-83 kubenswrapper[2572]: W0526 17:07:12.468719    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62617635_8da9_4fbe_8fd0_cc83a778bb85.slice/crio-e08f5fab31b5c5e8c1932857e2d9dc9eb187a5748c9dc486ce0c6b170b211e5d WatchSource:0}: Error finding container e08f5fab31b5c5e8c1932857e2d9dc9eb187a5748c9dc486ce0c6b170b211e5d: Status 404 returned error can't find the container with id e08f5fab31b5c5e8c1932857e2d9dc9eb187a5748c9dc486ce0c6b170b211e5d
May 26 17:07:12.564932 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:12.564897    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v" event={"ID":"62617635-8da9-4fbe-8fd0-cc83a778bb85","Type":"ContainerStarted","Data":"0785b2696e103c39e88a1d79d3bbff69a6f46abb0395216c38e3f085b1c71f01"}
May 26 17:07:12.564932 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:12.564932    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v" event={"ID":"62617635-8da9-4fbe-8fd0-cc83a778bb85","Type":"ContainerStarted","Data":"e08f5fab31b5c5e8c1932857e2d9dc9eb187a5748c9dc486ce0c6b170b211e5d"}
May 26 17:07:13.568595 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:13.568562    2572 generic.go:358] "Generic (PLEG): container finished" podID="62617635-8da9-4fbe-8fd0-cc83a778bb85" containerID="0785b2696e103c39e88a1d79d3bbff69a6f46abb0395216c38e3f085b1c71f01" exitCode=0
May 26 17:07:13.568978 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:13.568655    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v" event={"ID":"62617635-8da9-4fbe-8fd0-cc83a778bb85","Type":"ContainerDied","Data":"0785b2696e103c39e88a1d79d3bbff69a6f46abb0395216c38e3f085b1c71f01"}
May 26 17:07:14.573388 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:14.573353    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v" event={"ID":"62617635-8da9-4fbe-8fd0-cc83a778bb85","Type":"ContainerStarted","Data":"ff2995c107860d6961d5013370b11dc6d7466370b9fee8d6dd3de84e79eb31e7"}
May 26 17:07:14.573388 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:14.573390    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v" event={"ID":"62617635-8da9-4fbe-8fd0-cc83a778bb85","Type":"ContainerStarted","Data":"9ac6e7657b8b434820144d1e41639bd008b2162df0fcc89dd6d8ea4954a6a336"}
May 26 17:07:14.573864 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:14.573475    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v"
May 26 17:07:14.593568 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:14.593507    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v" podStartSLOduration=3.593489416 podStartE2EDuration="3.593489416s" podCreationTimestamp="2026-05-26 17:07:11 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 17:07:14.592885192 +0000 UTC m=+3951.068612543" watchObservedRunningTime="2026-05-26 17:07:14.593489416 +0000 UTC m=+3951.069216766"
May 26 17:07:15.576949 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:15.576919    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v"
May 26 17:07:15.578260 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:15.578233    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v" podUID="62617635-8da9-4fbe-8fd0-cc83a778bb85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
May 26 17:07:16.579463 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:16.579423    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v" podUID="62617635-8da9-4fbe-8fd0-cc83a778bb85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
May 26 17:07:21.583934 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:21.583904    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v"
May 26 17:07:21.584453 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:21.584423    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v" podUID="62617635-8da9-4fbe-8fd0-cc83a778bb85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
May 26 17:07:31.585179 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:31.585136    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v" podUID="62617635-8da9-4fbe-8fd0-cc83a778bb85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
May 26 17:07:41.584349 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:41.584307    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v" podUID="62617635-8da9-4fbe-8fd0-cc83a778bb85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
May 26 17:07:51.584951 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:07:51.584907    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v" podUID="62617635-8da9-4fbe-8fd0-cc83a778bb85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
May 26 17:08:01.584754 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:01.584717    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v" podUID="62617635-8da9-4fbe-8fd0-cc83a778bb85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
May 26 17:08:11.585227 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:11.585186    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v" podUID="62617635-8da9-4fbe-8fd0-cc83a778bb85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
May 26 17:08:21.584810 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:21.584775    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v"
May 26 17:08:31.441734 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:31.441698    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v"]
May 26 17:08:31.444161 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:31.442023    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v" podUID="62617635-8da9-4fbe-8fd0-cc83a778bb85" containerName="kserve-container" containerID="cri-o://9ac6e7657b8b434820144d1e41639bd008b2162df0fcc89dd6d8ea4954a6a336" gracePeriod=30
May 26 17:08:31.444161 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:31.442075    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v" podUID="62617635-8da9-4fbe-8fd0-cc83a778bb85" containerName="kube-rbac-proxy" containerID="cri-o://ff2995c107860d6961d5013370b11dc6d7466370b9fee8d6dd3de84e79eb31e7" gracePeriod=30
May 26 17:08:31.580663 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:31.580596    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v" podUID="62617635-8da9-4fbe-8fd0-cc83a778bb85" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.66:8643/healthz\": dial tcp 10.132.0.66:8643: connect: connection refused"
May 26 17:08:31.584896 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:31.584871    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v" podUID="62617635-8da9-4fbe-8fd0-cc83a778bb85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.66:8080: connect: connection refused"
May 26 17:08:31.786942 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:31.786854    2572 generic.go:358] "Generic (PLEG): container finished" podID="62617635-8da9-4fbe-8fd0-cc83a778bb85" containerID="ff2995c107860d6961d5013370b11dc6d7466370b9fee8d6dd3de84e79eb31e7" exitCode=2
May 26 17:08:31.786942 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:31.786928    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v" event={"ID":"62617635-8da9-4fbe-8fd0-cc83a778bb85","Type":"ContainerDied","Data":"ff2995c107860d6961d5013370b11dc6d7466370b9fee8d6dd3de84e79eb31e7"}
May 26 17:08:32.501036 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:32.500996    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8"]
May 26 17:08:32.504551 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:32.504528    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8"
May 26 17:08:32.506633 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:32.506595    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
May 26 17:08:32.506752 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:32.506655    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
May 26 17:08:32.514755 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:32.514727    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8"]
May 26 17:08:32.600972 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:32.600935    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/060cbf2e-497d-405a-8a05-c129289d9385-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8\" (UID: \"060cbf2e-497d-405a-8a05-c129289d9385\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8"
May 26 17:08:32.601255 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:32.600982    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/060cbf2e-497d-405a-8a05-c129289d9385-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8\" (UID: \"060cbf2e-497d-405a-8a05-c129289d9385\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8"
May 26 17:08:32.601255 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:32.601041    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/060cbf2e-497d-405a-8a05-c129289d9385-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8\" (UID: \"060cbf2e-497d-405a-8a05-c129289d9385\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8"
May 26 17:08:32.601255 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:32.601088    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55q9l\" (UniqueName: \"kubernetes.io/projected/060cbf2e-497d-405a-8a05-c129289d9385-kube-api-access-55q9l\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8\" (UID: \"060cbf2e-497d-405a-8a05-c129289d9385\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8"
May 26 17:08:32.702470 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:32.702431    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/060cbf2e-497d-405a-8a05-c129289d9385-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8\" (UID: \"060cbf2e-497d-405a-8a05-c129289d9385\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8"
May 26 17:08:32.702666 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:32.702478    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/060cbf2e-497d-405a-8a05-c129289d9385-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8\" (UID: \"060cbf2e-497d-405a-8a05-c129289d9385\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8"
May 26 17:08:32.702666 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:32.702514    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/060cbf2e-497d-405a-8a05-c129289d9385-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8\" (UID: \"060cbf2e-497d-405a-8a05-c129289d9385\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8"
May 26 17:08:32.702666 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:32.702534    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-55q9l\" (UniqueName: \"kubernetes.io/projected/060cbf2e-497d-405a-8a05-c129289d9385-kube-api-access-55q9l\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8\" (UID: \"060cbf2e-497d-405a-8a05-c129289d9385\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8"
May 26 17:08:32.702953 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:32.702927    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/060cbf2e-497d-405a-8a05-c129289d9385-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8\" (UID: \"060cbf2e-497d-405a-8a05-c129289d9385\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8"
May 26 17:08:32.703195 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:32.703178    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/060cbf2e-497d-405a-8a05-c129289d9385-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8\" (UID: \"060cbf2e-497d-405a-8a05-c129289d9385\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8"
May 26 17:08:32.704907 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:32.704877    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/060cbf2e-497d-405a-8a05-c129289d9385-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8\" (UID: \"060cbf2e-497d-405a-8a05-c129289d9385\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8"
May 26 17:08:32.711909 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:32.711883    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55q9l\" (UniqueName: \"kubernetes.io/projected/060cbf2e-497d-405a-8a05-c129289d9385-kube-api-access-55q9l\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8\" (UID: \"060cbf2e-497d-405a-8a05-c129289d9385\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8"
May 26 17:08:32.815859 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:32.815772    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8"
May 26 17:08:32.942061 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:32.942014    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8"]
May 26 17:08:32.945437 ip-10-0-130-83 kubenswrapper[2572]: W0526 17:08:32.945406    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod060cbf2e_497d_405a_8a05_c129289d9385.slice/crio-dc3cc47f49461072f1dc1e3c16e3f25562cd7f625e1048eab16dca02c91ba121 WatchSource:0}: Error finding container dc3cc47f49461072f1dc1e3c16e3f25562cd7f625e1048eab16dca02c91ba121: Status 404 returned error can't find the container with id dc3cc47f49461072f1dc1e3c16e3f25562cd7f625e1048eab16dca02c91ba121
May 26 17:08:33.793185 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:33.793146    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8" event={"ID":"060cbf2e-497d-405a-8a05-c129289d9385","Type":"ContainerStarted","Data":"c3b0df37ff2b2d517b262a247e8feadf9b42573f62bdda0f34ba7d5133e869a4"}
May 26 17:08:33.793185 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:33.793185    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8" event={"ID":"060cbf2e-497d-405a-8a05-c129289d9385","Type":"ContainerStarted","Data":"dc3cc47f49461072f1dc1e3c16e3f25562cd7f625e1048eab16dca02c91ba121"}
May 26 17:08:35.799888 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:35.799860    2572 generic.go:358] "Generic (PLEG): container finished" podID="62617635-8da9-4fbe-8fd0-cc83a778bb85" containerID="9ac6e7657b8b434820144d1e41639bd008b2162df0fcc89dd6d8ea4954a6a336" exitCode=0
May 26 17:08:35.799888 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:35.799901    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v" event={"ID":"62617635-8da9-4fbe-8fd0-cc83a778bb85","Type":"ContainerDied","Data":"9ac6e7657b8b434820144d1e41639bd008b2162df0fcc89dd6d8ea4954a6a336"}
May 26 17:08:35.886094 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:35.886070    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v"
May 26 17:08:35.926180 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:35.926141    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62617635-8da9-4fbe-8fd0-cc83a778bb85-cabundle-cert\") pod \"62617635-8da9-4fbe-8fd0-cc83a778bb85\" (UID: \"62617635-8da9-4fbe-8fd0-cc83a778bb85\") "
May 26 17:08:35.926370 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:35.926231    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62617635-8da9-4fbe-8fd0-cc83a778bb85-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\") pod \"62617635-8da9-4fbe-8fd0-cc83a778bb85\" (UID: \"62617635-8da9-4fbe-8fd0-cc83a778bb85\") "
May 26 17:08:35.926370 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:35.926282    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62617635-8da9-4fbe-8fd0-cc83a778bb85-kserve-provision-location\") pod \"62617635-8da9-4fbe-8fd0-cc83a778bb85\" (UID: \"62617635-8da9-4fbe-8fd0-cc83a778bb85\") "
May 26 17:08:35.926370 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:35.926308    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62617635-8da9-4fbe-8fd0-cc83a778bb85-proxy-tls\") pod \"62617635-8da9-4fbe-8fd0-cc83a778bb85\" (UID: \"62617635-8da9-4fbe-8fd0-cc83a778bb85\") "
May 26 17:08:35.926370 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:35.926338    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ffqlw\" (UniqueName: \"kubernetes.io/projected/62617635-8da9-4fbe-8fd0-cc83a778bb85-kube-api-access-ffqlw\") pod \"62617635-8da9-4fbe-8fd0-cc83a778bb85\" (UID: \"62617635-8da9-4fbe-8fd0-cc83a778bb85\") "
May 26 17:08:35.926634 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:35.926567    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62617635-8da9-4fbe-8fd0-cc83a778bb85-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "62617635-8da9-4fbe-8fd0-cc83a778bb85" (UID: "62617635-8da9-4fbe-8fd0-cc83a778bb85"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 17:08:35.926709 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:35.926628    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62617635-8da9-4fbe-8fd0-cc83a778bb85-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "62617635-8da9-4fbe-8fd0-cc83a778bb85" (UID: "62617635-8da9-4fbe-8fd0-cc83a778bb85"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 17:08:35.926709 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:35.926685    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62617635-8da9-4fbe-8fd0-cc83a778bb85-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config") pod "62617635-8da9-4fbe-8fd0-cc83a778bb85" (UID: "62617635-8da9-4fbe-8fd0-cc83a778bb85"). InnerVolumeSpecName "isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 17:08:35.928440 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:35.928412    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62617635-8da9-4fbe-8fd0-cc83a778bb85-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "62617635-8da9-4fbe-8fd0-cc83a778bb85" (UID: "62617635-8da9-4fbe-8fd0-cc83a778bb85"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 17:08:35.928550 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:35.928455    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62617635-8da9-4fbe-8fd0-cc83a778bb85-kube-api-access-ffqlw" (OuterVolumeSpecName: "kube-api-access-ffqlw") pod "62617635-8da9-4fbe-8fd0-cc83a778bb85" (UID: "62617635-8da9-4fbe-8fd0-cc83a778bb85"). InnerVolumeSpecName "kube-api-access-ffqlw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 17:08:36.027297 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:36.027206    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62617635-8da9-4fbe-8fd0-cc83a778bb85-kserve-provision-location\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 17:08:36.027297 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:36.027239    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62617635-8da9-4fbe-8fd0-cc83a778bb85-proxy-tls\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 17:08:36.027297 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:36.027250    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffqlw\" (UniqueName: \"kubernetes.io/projected/62617635-8da9-4fbe-8fd0-cc83a778bb85-kube-api-access-ffqlw\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 17:08:36.027297 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:36.027259    2572 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62617635-8da9-4fbe-8fd0-cc83a778bb85-cabundle-cert\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 17:08:36.027297 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:36.027269    2572 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62617635-8da9-4fbe-8fd0-cc83a778bb85-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 17:08:36.804713 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:36.804676    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v" event={"ID":"62617635-8da9-4fbe-8fd0-cc83a778bb85","Type":"ContainerDied","Data":"e08f5fab31b5c5e8c1932857e2d9dc9eb187a5748c9dc486ce0c6b170b211e5d"}
May 26 17:08:36.804713 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:36.804713    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v"
May 26 17:08:36.805174 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:36.804732    2572 scope.go:117] "RemoveContainer" containerID="ff2995c107860d6961d5013370b11dc6d7466370b9fee8d6dd3de84e79eb31e7"
May 26 17:08:36.812398 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:36.812376    2572 scope.go:117] "RemoveContainer" containerID="9ac6e7657b8b434820144d1e41639bd008b2162df0fcc89dd6d8ea4954a6a336"
May 26 17:08:36.819011 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:36.818992    2572 scope.go:117] "RemoveContainer" containerID="0785b2696e103c39e88a1d79d3bbff69a6f46abb0395216c38e3f085b1c71f01"
May 26 17:08:36.821078 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:36.821049    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v"]
May 26 17:08:36.827825 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:36.827795    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-754974974c-bbb4v"]
May 26 17:08:37.809783 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:37.809753    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8_060cbf2e-497d-405a-8a05-c129289d9385/storage-initializer/0.log"
May 26 17:08:37.810189 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:37.809790    2572 generic.go:358] "Generic (PLEG): container finished" podID="060cbf2e-497d-405a-8a05-c129289d9385" containerID="c3b0df37ff2b2d517b262a247e8feadf9b42573f62bdda0f34ba7d5133e869a4" exitCode=1
May 26 17:08:37.810189 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:37.809871    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8" event={"ID":"060cbf2e-497d-405a-8a05-c129289d9385","Type":"ContainerDied","Data":"c3b0df37ff2b2d517b262a247e8feadf9b42573f62bdda0f34ba7d5133e869a4"}
May 26 17:08:38.079831 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:38.079797    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62617635-8da9-4fbe-8fd0-cc83a778bb85" path="/var/lib/kubelet/pods/62617635-8da9-4fbe-8fd0-cc83a778bb85/volumes"
May 26 17:08:38.816882 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:38.816850    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8_060cbf2e-497d-405a-8a05-c129289d9385/storage-initializer/0.log"
May 26 17:08:38.817348 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:38.816918    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8" event={"ID":"060cbf2e-497d-405a-8a05-c129289d9385","Type":"ContainerStarted","Data":"c59f737822ff3a2d0783a6a843d7281c7ea93d5f0b2d7c45656dad92b7dce0b6"}
May 26 17:08:41.765812 ip-10-0-130-83 kubenswrapper[2572]: E0526 17:08:41.765779    2572 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod060cbf2e_497d_405a_8a05_c129289d9385.slice/crio-c59f737822ff3a2d0783a6a843d7281c7ea93d5f0b2d7c45656dad92b7dce0b6.scope\": RecentStats: unable to find data in memory cache]"
May 26 17:08:41.826450 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:41.826422    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8_060cbf2e-497d-405a-8a05-c129289d9385/storage-initializer/1.log"
May 26 17:08:41.826817 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:41.826798    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8_060cbf2e-497d-405a-8a05-c129289d9385/storage-initializer/0.log"
May 26 17:08:41.826878 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:41.826837    2572 generic.go:358] "Generic (PLEG): container finished" podID="060cbf2e-497d-405a-8a05-c129289d9385" containerID="c59f737822ff3a2d0783a6a843d7281c7ea93d5f0b2d7c45656dad92b7dce0b6" exitCode=1
May 26 17:08:41.826936 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:41.826919    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8" event={"ID":"060cbf2e-497d-405a-8a05-c129289d9385","Type":"ContainerDied","Data":"c59f737822ff3a2d0783a6a843d7281c7ea93d5f0b2d7c45656dad92b7dce0b6"}
May 26 17:08:41.826975 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:41.826959    2572 scope.go:117] "RemoveContainer" containerID="c3b0df37ff2b2d517b262a247e8feadf9b42573f62bdda0f34ba7d5133e869a4"
May 26 17:08:41.827372 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:41.827355    2572 scope.go:117] "RemoveContainer" containerID="c3b0df37ff2b2d517b262a247e8feadf9b42573f62bdda0f34ba7d5133e869a4"
May 26 17:08:41.837114 ip-10-0-130-83 kubenswrapper[2572]: E0526 17:08:41.837084    2572 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8_kserve-ci-e2e-test_060cbf2e-497d-405a-8a05-c129289d9385_0 in pod sandbox dc3cc47f49461072f1dc1e3c16e3f25562cd7f625e1048eab16dca02c91ba121 from index: no such id: 'c3b0df37ff2b2d517b262a247e8feadf9b42573f62bdda0f34ba7d5133e869a4'" containerID="c3b0df37ff2b2d517b262a247e8feadf9b42573f62bdda0f34ba7d5133e869a4"
May 26 17:08:41.837183 ip-10-0-130-83 kubenswrapper[2572]: E0526 17:08:41.837135    2572 kuberuntime_container.go:951] "Unhandled Error" err="failed to remove pod init container \"storage-initializer\": rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8_kserve-ci-e2e-test_060cbf2e-497d-405a-8a05-c129289d9385_0 in pod sandbox dc3cc47f49461072f1dc1e3c16e3f25562cd7f625e1048eab16dca02c91ba121 from index: no such id: 'c3b0df37ff2b2d517b262a247e8feadf9b42573f62bdda0f34ba7d5133e869a4'; Skipping pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8_kserve-ci-e2e-test(060cbf2e-497d-405a-8a05-c129289d9385)\"" logger="UnhandledError"
May 26 17:08:41.838473 ip-10-0-130-83 kubenswrapper[2572]: E0526 17:08:41.838453    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8_kserve-ci-e2e-test(060cbf2e-497d-405a-8a05-c129289d9385)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8" podUID="060cbf2e-497d-405a-8a05-c129289d9385"
May 26 17:08:42.488477 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:42.488445    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8"]
May 26 17:08:42.830882 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:42.830799    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8_060cbf2e-497d-405a-8a05-c129289d9385/storage-initializer/1.log"
May 26 17:08:42.961687 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:42.961666    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8_060cbf2e-497d-405a-8a05-c129289d9385/storage-initializer/1.log"
May 26 17:08:42.961803 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:42.961731    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8"
May 26 17:08:42.980893 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:42.980864    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55q9l\" (UniqueName: \"kubernetes.io/projected/060cbf2e-497d-405a-8a05-c129289d9385-kube-api-access-55q9l\") pod \"060cbf2e-497d-405a-8a05-c129289d9385\" (UID: \"060cbf2e-497d-405a-8a05-c129289d9385\") "
May 26 17:08:42.981051 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:42.980911    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/060cbf2e-497d-405a-8a05-c129289d9385-proxy-tls\") pod \"060cbf2e-497d-405a-8a05-c129289d9385\" (UID: \"060cbf2e-497d-405a-8a05-c129289d9385\") "
May 26 17:08:42.981051 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:42.980973    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/060cbf2e-497d-405a-8a05-c129289d9385-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\") pod \"060cbf2e-497d-405a-8a05-c129289d9385\" (UID: \"060cbf2e-497d-405a-8a05-c129289d9385\") "
May 26 17:08:42.981051 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:42.981011    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/060cbf2e-497d-405a-8a05-c129289d9385-kserve-provision-location\") pod \"060cbf2e-497d-405a-8a05-c129289d9385\" (UID: \"060cbf2e-497d-405a-8a05-c129289d9385\") "
May 26 17:08:42.981412 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:42.981384    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/060cbf2e-497d-405a-8a05-c129289d9385-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "060cbf2e-497d-405a-8a05-c129289d9385" (UID: "060cbf2e-497d-405a-8a05-c129289d9385"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 17:08:42.981509 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:42.981465    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/060cbf2e-497d-405a-8a05-c129289d9385-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config") pod "060cbf2e-497d-405a-8a05-c129289d9385" (UID: "060cbf2e-497d-405a-8a05-c129289d9385"). InnerVolumeSpecName "isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 17:08:42.983122 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:42.983100    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/060cbf2e-497d-405a-8a05-c129289d9385-kube-api-access-55q9l" (OuterVolumeSpecName: "kube-api-access-55q9l") pod "060cbf2e-497d-405a-8a05-c129289d9385" (UID: "060cbf2e-497d-405a-8a05-c129289d9385"). InnerVolumeSpecName "kube-api-access-55q9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 17:08:42.983204 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:42.983124    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/060cbf2e-497d-405a-8a05-c129289d9385-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "060cbf2e-497d-405a-8a05-c129289d9385" (UID: "060cbf2e-497d-405a-8a05-c129289d9385"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 17:08:43.082275 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:43.082184    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/060cbf2e-497d-405a-8a05-c129289d9385-kserve-provision-location\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 17:08:43.082275 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:43.082220    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55q9l\" (UniqueName: \"kubernetes.io/projected/060cbf2e-497d-405a-8a05-c129289d9385-kube-api-access-55q9l\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 17:08:43.082275 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:43.082232    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/060cbf2e-497d-405a-8a05-c129289d9385-proxy-tls\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 17:08:43.082275 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:43.082242    2572 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/060cbf2e-497d-405a-8a05-c129289d9385-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 17:08:43.835223 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:43.835192    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8_060cbf2e-497d-405a-8a05-c129289d9385/storage-initializer/1.log"
May 26 17:08:43.835629 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:43.835267    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8" event={"ID":"060cbf2e-497d-405a-8a05-c129289d9385","Type":"ContainerDied","Data":"dc3cc47f49461072f1dc1e3c16e3f25562cd7f625e1048eab16dca02c91ba121"}
May 26 17:08:43.835629 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:43.835297    2572 scope.go:117] "RemoveContainer" containerID="c59f737822ff3a2d0783a6a843d7281c7ea93d5f0b2d7c45656dad92b7dce0b6"
May 26 17:08:43.835629 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:43.835334    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8"
May 26 17:08:43.870194 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:43.870163    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8"]
May 26 17:08:43.876357 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:43.876324    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc64468b9-w5tx8"]
May 26 17:08:44.079431 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:44.079400    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="060cbf2e-497d-405a-8a05-c129289d9385" path="/var/lib/kubelet/pods/060cbf2e-497d-405a-8a05-c129289d9385/volumes"
May 26 17:08:44.519243 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:44.519205    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jdj48/must-gather-fmttx"]
May 26 17:08:44.519461 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:44.519450    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62617635-8da9-4fbe-8fd0-cc83a778bb85" containerName="kube-rbac-proxy"
May 26 17:08:44.519513 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:44.519463    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="62617635-8da9-4fbe-8fd0-cc83a778bb85" containerName="kube-rbac-proxy"
May 26 17:08:44.519513 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:44.519473    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62617635-8da9-4fbe-8fd0-cc83a778bb85" containerName="kserve-container"
May 26 17:08:44.519513 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:44.519479    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="62617635-8da9-4fbe-8fd0-cc83a778bb85" containerName="kserve-container"
May 26 17:08:44.519513 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:44.519487    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62617635-8da9-4fbe-8fd0-cc83a778bb85" containerName="storage-initializer"
May 26 17:08:44.519513 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:44.519493    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="62617635-8da9-4fbe-8fd0-cc83a778bb85" containerName="storage-initializer"
May 26 17:08:44.519513 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:44.519500    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="060cbf2e-497d-405a-8a05-c129289d9385" containerName="storage-initializer"
May 26 17:08:44.519513 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:44.519505    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="060cbf2e-497d-405a-8a05-c129289d9385" containerName="storage-initializer"
May 26 17:08:44.519778 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:44.519516    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="060cbf2e-497d-405a-8a05-c129289d9385" containerName="storage-initializer"
May 26 17:08:44.519778 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:44.519521    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="060cbf2e-497d-405a-8a05-c129289d9385" containerName="storage-initializer"
May 26 17:08:44.519778 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:44.519565    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="62617635-8da9-4fbe-8fd0-cc83a778bb85" containerName="kserve-container"
May 26 17:08:44.519778 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:44.519574    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="060cbf2e-497d-405a-8a05-c129289d9385" containerName="storage-initializer"
May 26 17:08:44.519778 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:44.519580    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="060cbf2e-497d-405a-8a05-c129289d9385" containerName="storage-initializer"
May 26 17:08:44.519778 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:44.519588    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="62617635-8da9-4fbe-8fd0-cc83a778bb85" containerName="kube-rbac-proxy"
May 26 17:08:44.523810 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:44.523793    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jdj48/must-gather-fmttx"
May 26 17:08:44.526319 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:44.526297    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jdj48\"/\"kube-root-ca.crt\""
May 26 17:08:44.526381 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:44.526297    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jdj48\"/\"openshift-service-ca.crt\""
May 26 17:08:44.531519 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:44.531497    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jdj48/must-gather-fmttx"]
May 26 17:08:44.595486 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:44.595450    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkzm4\" (UniqueName: \"kubernetes.io/projected/cf21b5c4-2d5f-444c-8822-5687a94c469a-kube-api-access-kkzm4\") pod \"must-gather-fmttx\" (UID: \"cf21b5c4-2d5f-444c-8822-5687a94c469a\") " pod="openshift-must-gather-jdj48/must-gather-fmttx"
May 26 17:08:44.595692 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:44.595492    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cf21b5c4-2d5f-444c-8822-5687a94c469a-must-gather-output\") pod \"must-gather-fmttx\" (UID: \"cf21b5c4-2d5f-444c-8822-5687a94c469a\") " pod="openshift-must-gather-jdj48/must-gather-fmttx"
May 26 17:08:44.696542 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:44.696505    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kkzm4\" (UniqueName: \"kubernetes.io/projected/cf21b5c4-2d5f-444c-8822-5687a94c469a-kube-api-access-kkzm4\") pod \"must-gather-fmttx\" (UID: \"cf21b5c4-2d5f-444c-8822-5687a94c469a\") " pod="openshift-must-gather-jdj48/must-gather-fmttx"
May 26 17:08:44.696542 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:44.696546    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cf21b5c4-2d5f-444c-8822-5687a94c469a-must-gather-output\") pod \"must-gather-fmttx\" (UID: \"cf21b5c4-2d5f-444c-8822-5687a94c469a\") " pod="openshift-must-gather-jdj48/must-gather-fmttx"
May 26 17:08:44.696862 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:44.696841    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cf21b5c4-2d5f-444c-8822-5687a94c469a-must-gather-output\") pod \"must-gather-fmttx\" (UID: \"cf21b5c4-2d5f-444c-8822-5687a94c469a\") " pod="openshift-must-gather-jdj48/must-gather-fmttx"
May 26 17:08:44.704692 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:44.704668    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkzm4\" (UniqueName: \"kubernetes.io/projected/cf21b5c4-2d5f-444c-8822-5687a94c469a-kube-api-access-kkzm4\") pod \"must-gather-fmttx\" (UID: \"cf21b5c4-2d5f-444c-8822-5687a94c469a\") " pod="openshift-must-gather-jdj48/must-gather-fmttx"
May 26 17:08:44.833146 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:44.833059    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jdj48/must-gather-fmttx"
May 26 17:08:44.958496 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:44.958468    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jdj48/must-gather-fmttx"]
May 26 17:08:44.961793 ip-10-0-130-83 kubenswrapper[2572]: W0526 17:08:44.961763    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf21b5c4_2d5f_444c_8822_5687a94c469a.slice/crio-e5ea740aebe18598e1434915ca53215349d82cee2d5d2fa7b24a57c10a09010f WatchSource:0}: Error finding container e5ea740aebe18598e1434915ca53215349d82cee2d5d2fa7b24a57c10a09010f: Status 404 returned error can't find the container with id e5ea740aebe18598e1434915ca53215349d82cee2d5d2fa7b24a57c10a09010f
May 26 17:08:45.842855 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:45.842812    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jdj48/must-gather-fmttx" event={"ID":"cf21b5c4-2d5f-444c-8822-5687a94c469a","Type":"ContainerStarted","Data":"e5ea740aebe18598e1434915ca53215349d82cee2d5d2fa7b24a57c10a09010f"}
May 26 17:08:51.863423 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:51.863376    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jdj48/must-gather-fmttx" event={"ID":"cf21b5c4-2d5f-444c-8822-5687a94c469a","Type":"ContainerStarted","Data":"35c381558e559fbf405ff55bed73881fb9152a7ea46f0bafb3d80f725cf44a9f"}
May 26 17:08:51.863423 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:51.863427    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jdj48/must-gather-fmttx" event={"ID":"cf21b5c4-2d5f-444c-8822-5687a94c469a","Type":"ContainerStarted","Data":"7b668db83e4b075533b2a3216079e080cb4cca0e6e00eee719c0de00f91c9f1f"}
May 26 17:08:51.880235 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:08:51.880185    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-jdj48/must-gather-fmttx" podStartSLOduration=1.328917535 podStartE2EDuration="7.88016904s" podCreationTimestamp="2026-05-26 17:08:44 +0000 UTC" firstStartedPulling="2026-05-26 17:08:44.963361602 +0000 UTC m=+4041.439088933" lastFinishedPulling="2026-05-26 17:08:51.514613093 +0000 UTC m=+4047.990340438" observedRunningTime="2026-05-26 17:08:51.878794454 +0000 UTC m=+4048.354521805" watchObservedRunningTime="2026-05-26 17:08:51.88016904 +0000 UTC m=+4048.355896389"
May 26 17:09:16.934793 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:16.934757    2572 generic.go:358] "Generic (PLEG): container finished" podID="cf21b5c4-2d5f-444c-8822-5687a94c469a" containerID="7b668db83e4b075533b2a3216079e080cb4cca0e6e00eee719c0de00f91c9f1f" exitCode=0
May 26 17:09:16.935216 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:16.934801    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jdj48/must-gather-fmttx" event={"ID":"cf21b5c4-2d5f-444c-8822-5687a94c469a","Type":"ContainerDied","Data":"7b668db83e4b075533b2a3216079e080cb4cca0e6e00eee719c0de00f91c9f1f"}
May 26 17:09:16.935216 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:16.935124    2572 scope.go:117] "RemoveContainer" containerID="7b668db83e4b075533b2a3216079e080cb4cca0e6e00eee719c0de00f91c9f1f"
May 26 17:09:17.168564 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:17.168540    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-jdj48_must-gather-fmttx_cf21b5c4-2d5f-444c-8822-5687a94c469a/gather/0.log"
May 26 17:09:20.270070 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:20.270040    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2fdgm_81fbb108-f79a-4018-82a4-0b0e29541320/global-pull-secret-syncer/0.log"
May 26 17:09:20.493680 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:20.493650    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-tbphl_32dd34b2-8588-46f9-b744-7138c2ac7025/konnectivity-agent/0.log"
May 26 17:09:20.560587 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:20.560514    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-83.ec2.internal_79549bd5c5e41ee1a69fee2a0b6c1223/haproxy/0.log"
May 26 17:09:22.678014 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:22.677974    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-jdj48/must-gather-fmttx"]
May 26 17:09:22.678528 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:22.678192    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-must-gather-jdj48/must-gather-fmttx" podUID="cf21b5c4-2d5f-444c-8822-5687a94c469a" containerName="copy" containerID="cri-o://35c381558e559fbf405ff55bed73881fb9152a7ea46f0bafb3d80f725cf44a9f" gracePeriod=2
May 26 17:09:22.681868 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:22.681826    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-jdj48/must-gather-fmttx"]
May 26 17:09:22.907169 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:22.907143    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-jdj48_must-gather-fmttx_cf21b5c4-2d5f-444c-8822-5687a94c469a/copy/0.log"
May 26 17:09:22.907489 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:22.907472    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jdj48/must-gather-fmttx"
May 26 17:09:22.909391 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:22.909364    2572 status_manager.go:895] "Failed to get status for pod" podUID="cf21b5c4-2d5f-444c-8822-5687a94c469a" pod="openshift-must-gather-jdj48/must-gather-fmttx" err="pods \"must-gather-fmttx\" is forbidden: User \"system:node:ip-10-0-130-83.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-jdj48\": no relationship found between node 'ip-10-0-130-83.ec2.internal' and this object"
May 26 17:09:22.954476 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:22.954401    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-jdj48_must-gather-fmttx_cf21b5c4-2d5f-444c-8822-5687a94c469a/copy/0.log"
May 26 17:09:22.954753 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:22.954731    2572 generic.go:358] "Generic (PLEG): container finished" podID="cf21b5c4-2d5f-444c-8822-5687a94c469a" containerID="35c381558e559fbf405ff55bed73881fb9152a7ea46f0bafb3d80f725cf44a9f" exitCode=143
May 26 17:09:22.954822 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:22.954783    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jdj48/must-gather-fmttx"
May 26 17:09:22.954822 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:22.954803    2572 scope.go:117] "RemoveContainer" containerID="35c381558e559fbf405ff55bed73881fb9152a7ea46f0bafb3d80f725cf44a9f"
May 26 17:09:22.956667 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:22.956647    2572 status_manager.go:895] "Failed to get status for pod" podUID="cf21b5c4-2d5f-444c-8822-5687a94c469a" pod="openshift-must-gather-jdj48/must-gather-fmttx" err="pods \"must-gather-fmttx\" is forbidden: User \"system:node:ip-10-0-130-83.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-jdj48\": no relationship found between node 'ip-10-0-130-83.ec2.internal' and this object"
May 26 17:09:22.961784 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:22.961769    2572 scope.go:117] "RemoveContainer" containerID="7b668db83e4b075533b2a3216079e080cb4cca0e6e00eee719c0de00f91c9f1f"
May 26 17:09:22.973373 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:22.973350    2572 scope.go:117] "RemoveContainer" containerID="35c381558e559fbf405ff55bed73881fb9152a7ea46f0bafb3d80f725cf44a9f"
May 26 17:09:22.973640 ip-10-0-130-83 kubenswrapper[2572]: E0526 17:09:22.973619    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35c381558e559fbf405ff55bed73881fb9152a7ea46f0bafb3d80f725cf44a9f\": container with ID starting with 35c381558e559fbf405ff55bed73881fb9152a7ea46f0bafb3d80f725cf44a9f not found: ID does not exist" containerID="35c381558e559fbf405ff55bed73881fb9152a7ea46f0bafb3d80f725cf44a9f"
May 26 17:09:22.973720 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:22.973647    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35c381558e559fbf405ff55bed73881fb9152a7ea46f0bafb3d80f725cf44a9f"} err="failed to get container status \"35c381558e559fbf405ff55bed73881fb9152a7ea46f0bafb3d80f725cf44a9f\": rpc error: code = NotFound desc = could not find container \"35c381558e559fbf405ff55bed73881fb9152a7ea46f0bafb3d80f725cf44a9f\": container with ID starting with 35c381558e559fbf405ff55bed73881fb9152a7ea46f0bafb3d80f725cf44a9f not found: ID does not exist"
May 26 17:09:22.973720 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:22.973666    2572 scope.go:117] "RemoveContainer" containerID="7b668db83e4b075533b2a3216079e080cb4cca0e6e00eee719c0de00f91c9f1f"
May 26 17:09:22.973883 ip-10-0-130-83 kubenswrapper[2572]: E0526 17:09:22.973867    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b668db83e4b075533b2a3216079e080cb4cca0e6e00eee719c0de00f91c9f1f\": container with ID starting with 7b668db83e4b075533b2a3216079e080cb4cca0e6e00eee719c0de00f91c9f1f not found: ID does not exist" containerID="7b668db83e4b075533b2a3216079e080cb4cca0e6e00eee719c0de00f91c9f1f"
May 26 17:09:22.973929 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:22.973886    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b668db83e4b075533b2a3216079e080cb4cca0e6e00eee719c0de00f91c9f1f"} err="failed to get container status \"7b668db83e4b075533b2a3216079e080cb4cca0e6e00eee719c0de00f91c9f1f\": rpc error: code = NotFound desc = could not find container \"7b668db83e4b075533b2a3216079e080cb4cca0e6e00eee719c0de00f91c9f1f\": container with ID starting with 7b668db83e4b075533b2a3216079e080cb4cca0e6e00eee719c0de00f91c9f1f not found: ID does not exist"
May 26 17:09:23.001215 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:23.001186    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cf21b5c4-2d5f-444c-8822-5687a94c469a-must-gather-output\") pod \"cf21b5c4-2d5f-444c-8822-5687a94c469a\" (UID: \"cf21b5c4-2d5f-444c-8822-5687a94c469a\") "
May 26 17:09:23.001374 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:23.001245    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kkzm4\" (UniqueName: \"kubernetes.io/projected/cf21b5c4-2d5f-444c-8822-5687a94c469a-kube-api-access-kkzm4\") pod \"cf21b5c4-2d5f-444c-8822-5687a94c469a\" (UID: \"cf21b5c4-2d5f-444c-8822-5687a94c469a\") "
May 26 17:09:23.002763 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:23.002733    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf21b5c4-2d5f-444c-8822-5687a94c469a-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "cf21b5c4-2d5f-444c-8822-5687a94c469a" (UID: "cf21b5c4-2d5f-444c-8822-5687a94c469a"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 17:09:23.003322 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:23.003302    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf21b5c4-2d5f-444c-8822-5687a94c469a-kube-api-access-kkzm4" (OuterVolumeSpecName: "kube-api-access-kkzm4") pod "cf21b5c4-2d5f-444c-8822-5687a94c469a" (UID: "cf21b5c4-2d5f-444c-8822-5687a94c469a"). InnerVolumeSpecName "kube-api-access-kkzm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 17:09:23.102003 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:23.101951    2572 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cf21b5c4-2d5f-444c-8822-5687a94c469a-must-gather-output\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 17:09:23.102003 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:23.101998    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkzm4\" (UniqueName: \"kubernetes.io/projected/cf21b5c4-2d5f-444c-8822-5687a94c469a-kube-api-access-kkzm4\") on node \"ip-10-0-130-83.ec2.internal\" DevicePath \"\""
May 26 17:09:23.264120 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:23.264091    2572 status_manager.go:895] "Failed to get status for pod" podUID="cf21b5c4-2d5f-444c-8822-5687a94c469a" pod="openshift-must-gather-jdj48/must-gather-fmttx" err="pods \"must-gather-fmttx\" is forbidden: User \"system:node:ip-10-0-130-83.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-jdj48\": no relationship found between node 'ip-10-0-130-83.ec2.internal' and this object"
May 26 17:09:24.079139 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:24.079107    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf21b5c4-2d5f-444c-8822-5687a94c469a" path="/var/lib/kubelet/pods/cf21b5c4-2d5f-444c-8822-5687a94c469a/volumes"
May 26 17:09:24.079913 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:24.079888    2572 status_manager.go:895] "Failed to get status for pod" podUID="cf21b5c4-2d5f-444c-8822-5687a94c469a" pod="openshift-must-gather-jdj48/must-gather-fmttx" err="pods \"must-gather-fmttx\" is forbidden: User \"system:node:ip-10-0-130-83.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-jdj48\": no relationship found between node 'ip-10-0-130-83.ec2.internal' and this object"
May 26 17:09:24.300408 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:24.300369    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hjs7w_b94c5ba6-2dd8-47f7-8cba-723af642e2e3/node-exporter/0.log"
May 26 17:09:24.320463 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:24.320434    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hjs7w_b94c5ba6-2dd8-47f7-8cba-723af642e2e3/kube-rbac-proxy/0.log"
May 26 17:09:24.342526 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:24.342451    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hjs7w_b94c5ba6-2dd8-47f7-8cba-723af642e2e3/init-textfile/0.log"
May 26 17:09:26.113680 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:26.113648    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-d4d8f94f9-bj22q_be441f80-19f8-48c2-b1b2-624b3bddcb27/networking-console-plugin/0.log"
May 26 17:09:27.735845 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:27.735811    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-r4v6g/perf-node-gather-daemonset-f2xwm"]
May 26 17:09:27.736225 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:27.736093    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf21b5c4-2d5f-444c-8822-5687a94c469a" containerName="copy"
May 26 17:09:27.736225 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:27.736104    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf21b5c4-2d5f-444c-8822-5687a94c469a" containerName="copy"
May 26 17:09:27.736225 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:27.736118    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf21b5c4-2d5f-444c-8822-5687a94c469a" containerName="gather"
May 26 17:09:27.736225 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:27.736124    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf21b5c4-2d5f-444c-8822-5687a94c469a" containerName="gather"
May 26 17:09:27.736225 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:27.736169    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="cf21b5c4-2d5f-444c-8822-5687a94c469a" containerName="gather"
May 26 17:09:27.736225 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:27.736177    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="cf21b5c4-2d5f-444c-8822-5687a94c469a" containerName="copy"
May 26 17:09:27.739486 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:27.739461    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r4v6g/perf-node-gather-daemonset-f2xwm"
May 26 17:09:27.741554 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:27.741533    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-r4v6g\"/\"openshift-service-ca.crt\""
May 26 17:09:27.741700 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:27.741540    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-r4v6g\"/\"default-dockercfg-pq547\""
May 26 17:09:27.742152 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:27.742135    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-r4v6g\"/\"kube-root-ca.crt\""
May 26 17:09:27.748296 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:27.748271    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r4v6g/perf-node-gather-daemonset-f2xwm"]
May 26 17:09:27.837304 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:27.837269    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/df420945-3c9d-41c5-bf0a-0c11463418a3-sys\") pod \"perf-node-gather-daemonset-f2xwm\" (UID: \"df420945-3c9d-41c5-bf0a-0c11463418a3\") " pod="openshift-must-gather-r4v6g/perf-node-gather-daemonset-f2xwm"
May 26 17:09:27.837483 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:27.837310    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8k89\" (UniqueName: \"kubernetes.io/projected/df420945-3c9d-41c5-bf0a-0c11463418a3-kube-api-access-x8k89\") pod \"perf-node-gather-daemonset-f2xwm\" (UID: \"df420945-3c9d-41c5-bf0a-0c11463418a3\") " pod="openshift-must-gather-r4v6g/perf-node-gather-daemonset-f2xwm"
May 26 17:09:27.837483 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:27.837332    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/df420945-3c9d-41c5-bf0a-0c11463418a3-podres\") pod \"perf-node-gather-daemonset-f2xwm\" (UID: \"df420945-3c9d-41c5-bf0a-0c11463418a3\") " pod="openshift-must-gather-r4v6g/perf-node-gather-daemonset-f2xwm"
May 26 17:09:27.837483 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:27.837369    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/df420945-3c9d-41c5-bf0a-0c11463418a3-proc\") pod \"perf-node-gather-daemonset-f2xwm\" (UID: \"df420945-3c9d-41c5-bf0a-0c11463418a3\") " pod="openshift-must-gather-r4v6g/perf-node-gather-daemonset-f2xwm"
May 26 17:09:27.837483 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:27.837394    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/df420945-3c9d-41c5-bf0a-0c11463418a3-lib-modules\") pod \"perf-node-gather-daemonset-f2xwm\" (UID: \"df420945-3c9d-41c5-bf0a-0c11463418a3\") " pod="openshift-must-gather-r4v6g/perf-node-gather-daemonset-f2xwm"
May 26 17:09:27.938080 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:27.938043    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-x8k89\" (UniqueName: \"kubernetes.io/projected/df420945-3c9d-41c5-bf0a-0c11463418a3-kube-api-access-x8k89\") pod \"perf-node-gather-daemonset-f2xwm\" (UID: \"df420945-3c9d-41c5-bf0a-0c11463418a3\") " pod="openshift-must-gather-r4v6g/perf-node-gather-daemonset-f2xwm"
May 26 17:09:27.938080 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:27.938091    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/df420945-3c9d-41c5-bf0a-0c11463418a3-podres\") pod \"perf-node-gather-daemonset-f2xwm\" (UID: \"df420945-3c9d-41c5-bf0a-0c11463418a3\") " pod="openshift-must-gather-r4v6g/perf-node-gather-daemonset-f2xwm"
May 26 17:09:27.938355 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:27.938126    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/df420945-3c9d-41c5-bf0a-0c11463418a3-proc\") pod \"perf-node-gather-daemonset-f2xwm\" (UID: \"df420945-3c9d-41c5-bf0a-0c11463418a3\") " pod="openshift-must-gather-r4v6g/perf-node-gather-daemonset-f2xwm"
May 26 17:09:27.938355 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:27.938179    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/df420945-3c9d-41c5-bf0a-0c11463418a3-lib-modules\") pod \"perf-node-gather-daemonset-f2xwm\" (UID: \"df420945-3c9d-41c5-bf0a-0c11463418a3\") " pod="openshift-must-gather-r4v6g/perf-node-gather-daemonset-f2xwm"
May 26 17:09:27.938355 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:27.938208    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/df420945-3c9d-41c5-bf0a-0c11463418a3-proc\") pod \"perf-node-gather-daemonset-f2xwm\" (UID: \"df420945-3c9d-41c5-bf0a-0c11463418a3\") " pod="openshift-must-gather-r4v6g/perf-node-gather-daemonset-f2xwm"
May 26 17:09:27.938355 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:27.938228    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/df420945-3c9d-41c5-bf0a-0c11463418a3-sys\") pod \"perf-node-gather-daemonset-f2xwm\" (UID: \"df420945-3c9d-41c5-bf0a-0c11463418a3\") " pod="openshift-must-gather-r4v6g/perf-node-gather-daemonset-f2xwm"
May 26 17:09:27.938355 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:27.938250    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/df420945-3c9d-41c5-bf0a-0c11463418a3-podres\") pod \"perf-node-gather-daemonset-f2xwm\" (UID: \"df420945-3c9d-41c5-bf0a-0c11463418a3\") " pod="openshift-must-gather-r4v6g/perf-node-gather-daemonset-f2xwm"
May 26 17:09:27.938355 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:27.938298    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/df420945-3c9d-41c5-bf0a-0c11463418a3-sys\") pod \"perf-node-gather-daemonset-f2xwm\" (UID: \"df420945-3c9d-41c5-bf0a-0c11463418a3\") " pod="openshift-must-gather-r4v6g/perf-node-gather-daemonset-f2xwm"
May 26 17:09:27.938355 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:27.938315    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/df420945-3c9d-41c5-bf0a-0c11463418a3-lib-modules\") pod \"perf-node-gather-daemonset-f2xwm\" (UID: \"df420945-3c9d-41c5-bf0a-0c11463418a3\") " pod="openshift-must-gather-r4v6g/perf-node-gather-daemonset-f2xwm"
May 26 17:09:27.945840 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:27.945814    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8k89\" (UniqueName: \"kubernetes.io/projected/df420945-3c9d-41c5-bf0a-0c11463418a3-kube-api-access-x8k89\") pod \"perf-node-gather-daemonset-f2xwm\" (UID: \"df420945-3c9d-41c5-bf0a-0c11463418a3\") " pod="openshift-must-gather-r4v6g/perf-node-gather-daemonset-f2xwm"
May 26 17:09:28.049598 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:28.049499    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r4v6g/perf-node-gather-daemonset-f2xwm"
May 26 17:09:28.050565 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:28.050482    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fvwnc_7b87e383-816e-4130-80ba-843dd20a187d/dns/0.log"
May 26 17:09:28.067788 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:28.067764    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fvwnc_7b87e383-816e-4130-80ba-843dd20a187d/kube-rbac-proxy/0.log"
May 26 17:09:28.135956 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:28.135915    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lvfvd_02c134e5-0f87-48c2-80ed-427d242f1435/dns-node-resolver/0.log"
May 26 17:09:28.174352 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:28.174320    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r4v6g/perf-node-gather-daemonset-f2xwm"]
May 26 17:09:28.177413 ip-10-0-130-83 kubenswrapper[2572]: W0526 17:09:28.177386    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poddf420945_3c9d_41c5_bf0a_0c11463418a3.slice/crio-c52de29ca5c7cf6c407d85b1a6ffcbe67749f3f21b8c2fc61ae493da480de320 WatchSource:0}: Error finding container c52de29ca5c7cf6c407d85b1a6ffcbe67749f3f21b8c2fc61ae493da480de320: Status 404 returned error can't find the container with id c52de29ca5c7cf6c407d85b1a6ffcbe67749f3f21b8c2fc61ae493da480de320
May 26 17:09:28.587751 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:28.587720    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qdk28_10ea1129-78a4-4d29-ab50-50b69b6d30c1/node-ca/0.log"
May 26 17:09:28.972377 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:28.972341    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r4v6g/perf-node-gather-daemonset-f2xwm" event={"ID":"df420945-3c9d-41c5-bf0a-0c11463418a3","Type":"ContainerStarted","Data":"5308a16839af752a0698dc48ccd08f31041f8113b6bd637947519b222e7f343e"}
May 26 17:09:28.972377 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:28.972375    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r4v6g/perf-node-gather-daemonset-f2xwm" event={"ID":"df420945-3c9d-41c5-bf0a-0c11463418a3","Type":"ContainerStarted","Data":"c52de29ca5c7cf6c407d85b1a6ffcbe67749f3f21b8c2fc61ae493da480de320"}
May 26 17:09:28.972932 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:28.972468    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-r4v6g/perf-node-gather-daemonset-f2xwm"
May 26 17:09:28.989725 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:28.989658    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-r4v6g/perf-node-gather-daemonset-f2xwm" podStartSLOduration=1.989638222 podStartE2EDuration="1.989638222s" podCreationTimestamp="2026-05-26 17:09:27 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 17:09:28.988099379 +0000 UTC m=+4085.463826740" watchObservedRunningTime="2026-05-26 17:09:28.989638222 +0000 UTC m=+4085.465365576"
May 26 17:09:29.596116 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:29.596083    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hqnkg_5e434be1-fcca-488a-aeff-38268e150853/serve-healthcheck-canary/0.log"
May 26 17:09:30.062109 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:30.062081    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xsqdx_556b4c71-e309-4091-88b9-c1073bd4ede3/kube-rbac-proxy/0.log"
May 26 17:09:30.082379 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:30.082355    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xsqdx_556b4c71-e309-4091-88b9-c1073bd4ede3/exporter/0.log"
May 26 17:09:30.099584 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:30.099559    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xsqdx_556b4c71-e309-4091-88b9-c1073bd4ede3/extractor/0.log"
May 26 17:09:32.043233 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:32.043196    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-6f7545b5b9-98fbz_3d17d333-ec9a-4a4c-8fc0-08eddf7ab0f5/manager/0.log"
May 26 17:09:32.284048 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:32.284015    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-696fc77849-b7h68_b08cd028-080d-4719-842c-93bc6dfa6cce/manager/0.log"
May 26 17:09:32.303699 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:32.303620    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-lqlhh_a40c3e2d-ef26-45c2-b3e8-d6bd68cc3813/s3-init/0.log"
May 26 17:09:32.396804 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:32.396777    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-custom-5c88b85bb7-sgf6n_56b94dfa-4f52-4112-bd8c-ddda1a1916fb/seaweedfs-tls-custom/0.log"
May 26 17:09:32.416792 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:32.416763    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-serving-7fd5766db9-8zs7n_75f178e5-c6fa-4767-b2d4-35d1f685ab34/seaweedfs-tls-serving/0.log"
May 26 17:09:34.984857 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:34.984825    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-r4v6g/perf-node-gather-daemonset-f2xwm"
May 26 17:09:37.413229 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:37.413200    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mjznd_23cb5db1-ddc3-46a0-bc9d-3133631ffc58/kube-multus-additional-cni-plugins/0.log"
May 26 17:09:37.435770 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:37.435738    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mjznd_23cb5db1-ddc3-46a0-bc9d-3133631ffc58/egress-router-binary-copy/0.log"
May 26 17:09:37.458450 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:37.458424    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mjznd_23cb5db1-ddc3-46a0-bc9d-3133631ffc58/cni-plugins/0.log"
May 26 17:09:37.483267 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:37.483240    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mjznd_23cb5db1-ddc3-46a0-bc9d-3133631ffc58/bond-cni-plugin/0.log"
May 26 17:09:37.503721 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:37.503695    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mjznd_23cb5db1-ddc3-46a0-bc9d-3133631ffc58/routeoverride-cni/0.log"
May 26 17:09:37.527709 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:37.527683    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mjznd_23cb5db1-ddc3-46a0-bc9d-3133631ffc58/whereabouts-cni-bincopy/0.log"
May 26 17:09:37.548113 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:37.548033    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mjznd_23cb5db1-ddc3-46a0-bc9d-3133631ffc58/whereabouts-cni/0.log"
May 26 17:09:37.739111 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:37.739084    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-wk2zv_5ad6cab9-b922-456e-ad28-95f40ce6692c/kube-multus/0.log"
May 26 17:09:37.835958 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:37.835880    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-qpzqg_8828138d-e7ed-4d5c-ba76-26429679c7b1/network-metrics-daemon/0.log"
May 26 17:09:37.850809 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:37.850779    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-qpzqg_8828138d-e7ed-4d5c-ba76-26429679c7b1/kube-rbac-proxy/0.log"
May 26 17:09:38.909051 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:38.909022    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l2lg5_f9823062-c35f-4aca-b363-912ec43ecb40/ovn-controller/0.log"
May 26 17:09:38.942539 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:38.942514    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l2lg5_f9823062-c35f-4aca-b363-912ec43ecb40/ovn-acl-logging/0.log"
May 26 17:09:38.961481 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:38.961445    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l2lg5_f9823062-c35f-4aca-b363-912ec43ecb40/kube-rbac-proxy-node/0.log"
May 26 17:09:38.981535 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:38.981505    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l2lg5_f9823062-c35f-4aca-b363-912ec43ecb40/kube-rbac-proxy-ovn-metrics/0.log"
May 26 17:09:38.999414 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:38.999380    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l2lg5_f9823062-c35f-4aca-b363-912ec43ecb40/northd/0.log"
May 26 17:09:39.019245 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:39.019219    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l2lg5_f9823062-c35f-4aca-b363-912ec43ecb40/nbdb/0.log"
May 26 17:09:39.038759 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:39.038719    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l2lg5_f9823062-c35f-4aca-b363-912ec43ecb40/sbdb/0.log"
May 26 17:09:39.136728 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:39.136699    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l2lg5_f9823062-c35f-4aca-b363-912ec43ecb40/ovnkube-controller/0.log"
May 26 17:09:40.451394 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:40.451364    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-rvlkc_8f54a9e9-f1b3-4073-bf4e-fef6f31dc622/network-check-target-container/0.log"
May 26 17:09:41.373562 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:41.373539    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-jlndg_c54d2f93-a58c-4f73-966d-2584dc7015e8/iptables-alerter/0.log"
May 26 17:09:42.035545 ip-10-0-130-83 kubenswrapper[2572]: I0526 17:09:42.035515    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-8hhhx_744c275a-8d87-4882-8e2e-73254829ca4a/tuned/0.log"